6 ways to strengthen PFS evaluations (Urban Institute)

By Sarah Gillespie

Pay for success (PFS) is often defined by its financial risks: if the program funded through the model achieves the target outcomes agreed upon by the project’s partners, investors see a return on the upfront capital they provided. If the program falls short, government doesn’t pay and the project’s investors take a loss.

The target outcomes that drive repayment are measured by a rigorous evaluation – making the independent evaluator central to the project as well as the only partner without a financial stake. Their agenda is focused on high-quality data collection and analysis to ground the project in evidence and determine project outcomes. 

The Urban Institute serves as an evaluator for two of the twelve active pay for success projects in the United States: the Massachusetts project on juvenile justice and reentry and the Denver project scaling permanent supportive housing to tackle chronic homelessness. Through this work we’ve developed a deeper understanding of the ways the PFS model requires unique evaluation considerations, including the distinctive and critical role played by evaluators.

Our recommendations are discussed in two new papers we’ve written on PFS evaluation. One paper takes an in-depth look at the role of the evaluator in each PFS stage, while the other is written specifically for evaluators navigating the details of research design within a PFS model.

As explored in our papers, PFS partners can strengthen the evaluation and the entire project by adopting the following best practices:

  • Bring the evaluation partner on board as early as possible to help ensure that important decisions in the early development phases, such as the target population and program model, are grounded in evidence.
  • Provide time and support for an evaluation planning period so that the most rigorous evaluation design possible can be created with full buy-in from all key PFS partners. 
  • Provide time and support for an implementation pilot period with the service provider to minimize implementation failure and to test key implementation assumptions, such as how the referral process will work and whether it will refer as many participants as expected. 
  • Refer to other PFS contracts to adapt language that builds in evaluation flexibility, such as data checkpoints and alternate analysis plans, and to learn from other evaluation structures. 
  • Consult with the independent validator, if one has been identified to validate the evaluator’s findings, as early as possible to address any potential analytical or institutional review board concerns before implementation begins.
  • Create a process for constant communication between the evaluator and other PFS partners, such as an operations committee, to coordinate on implementation or evaluation challenges as soon as they arise, especially during early implementation.

Perhaps the most important takeaway from these papers is that evaluation is an ongoing process within a PFS project and not a stand-alone stage. Partners can strengthen PFS evaluations and projects by understanding these roles and addressing these considerations as early (and often!) as possible. 

Have a Pay for Success question? Ask our experts at PFSsupport@urban.org!

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.