Event recap: What Works in Federal Agencies (Urban Institute)

By Ben Holston, Urban Institute, Research Assistant

As Moneyball for Government starts to take hold, evidence-based policymaking now has its own scorecard.

This week, Results for America launched the fifth iteration of their Federal Invest in What Works Index highlighting how federal agencies are making progress towards more evidence-based policymaking. The report measures the extent to which federal agencies are currently building the infrastructure necessary to be able to use data, evidence, and evaluation in budget, policy, and management decisions.

Coinciding with the release, Results For America invited representatives from participating agencies to present on their use of evidence and discuss evidence-based policymaking. Office of Management & Budget (OMB) Director Shaun Donovan concluded the event with a keynote highlighting what evidence-based policymaking has already accomplished and its potential to grow in the future.

The Invest in What Works Index 2016 

The 2016 Index evaluated seven federal agencies on the quality of their evidence-based policymaking. The Index ranks agencies on a 10-point scale across 10 criteria. These criteria are equally weighted and include evaluation and research, data, use of evidence in grant programs, and repurposing funds away from programs that do not work. 

The Millennium Challenge Corporation (MCC) received the highest score (85) in the index, followed closely by the U.S. Agency for International Development (USAID). All seven agencies in the Index received high scores on data, leadership, and evaluation. However, scores were far more varied in the “repurpose for results” category, which looks at whether an agency shifted funds away from programs that do not work. The number of criteria has expanded each year, and overall this year’s agencies performed the highest of any year RFA has created this index.

Event panelists

The launch event had a few prominent themes, including the importance of building a culture of evidence, transparency, and rigorous evaluation.

  • Culture of evidence: One of the most prominent themes of the event was building a “culture of evidence” at the agency level. The White House Office of Social Innovation and Civic Participation laid out a clear theory of change to promote evidence within an agency: find what works, fund what works, and help get the sector ready. Several agencies have also launched organization-wide evaluation policies to make evidence a cornerstone of their organization’s culture. Many speakers noted that this culture could become pervasive across agencies, making clear efforts to learn from the best programs implemented in other organizations.
  • Transparency: Another major theme the agencies highlighted was the need for consistent transparency and accountability when pursuing evidence-based policymaking. Publicizing all results, including those that demonstrate an agency program may be ineffective, is critical to making evidence work. The presenters and panelists frequently noted times when they would have preferred not to see the negative results of an evaluation, but understood the importance of learning from that feedback to improve their programs.
  • Rigor: Money talks, and one of the most direct applications of rigorous evidence used by the agencies is intentionally giving more funding to programs that use more rigorous evidence. In this case, rigorous evidence is tied directly to the strength of the evaluation used in the program. Several agencies also noted that evaluations in general have been getting stronger, including many that use existing administrative data to maintain the strength of the evaluation while driving down costs.

Keynote address 

OMB Director Shaun Donovan concluded the event with a keynote address providing a broad look at evidence-based policymaking. He set forth three key principles: act on strong evidence, build what we lack, and invest in what works. He noted tiered evidence grant programs and pay for success (PFS) as important mechanisms to pursue these principles. Donovan also used the event to announce 11 new PFS feasibility studies funded through the Social Innovation Fund (SIF), joining 14 announced last week. This announcement brought the total number of state and local governments pursuing PFS using SIF funding to 67.

Donovan praised PFS, pointing to the power of money to incentivize social service providers to get the evidence they need to prove success. Donovan ended his remarks by capturing the essence of what makes evidence-based policymaking so promising: “We can show the American people that the work that we’re doing works.”

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research.