Research Designs for Evaluating Pay for Success Programs (Virginia Pay for Success Lab)

Pay for Success financing is grounded in evidence-based interventions. Research is an essential component of developing and sustaining a viable Pay for Success program. During the initial, feasibility stage of Pay for Success financing, retrospective research lays the foundation for effective interventions and programming. Evaluators also use prospective research to measure the success of a program.

Evaluative research is only as strong as its research design. It is important to consider the type of research that will be conducted, how to set up a control group, the outcome of interest, and the follow-up timeframe when developing and implementing a research design.

RANDOMIZED CONTROL TRIALS VERSUS QUASI-EXPERIMENTAL DESIGNS

The “gold standard” in research is a randomized control trial (RCT). This design randomly assigns individuals to either a control or a treatment group. A large sample for both control and treatment group yields two groups that, in essence, should be statistically identical in every regard except the treatment condition. For instance, several evaluations of workforce development programs use RCTs to measure the impact of these programs. Individuals in the treatment group participate in the intervention, while control group individuals do not.

Not all program evaluations can leverage random assignment. In particular, research designs built around post-program observational data do not have the ability to do so. Quasi-experimental designs are useful in the absence of randomization. For a workforce development program, researchers might utilize pre- and post-program data to measure the effect of the treatment. Other evaluations may compare outcomes from individuals barely eligible and ineligible for participation in a workforce development program. These quasi-experimental designs use a variety of techniques to craft a control group to measure the true effect of a treatment.  Regardless of the technique, it is important to create a control group that share similar demographic characteristics as the treatment group.

DEFINING AN OUTCOME

Research measures the effect of some treatment on an outcome. It is important to define an outcome (or a group of outcomes) clearly at the start of a research project. An ideal outcome is an observable trait that is observed for both the control and treatment group and can be measured several years after an individual receives treatment (goes through a program). Evaluations of workforce development programs often select employment rates, income, and unemployment insurance as outcomes.  

FOLLOW-UP

The follow-up stage is one of the final, but critical components of a research design. Many of the effects from workforce development programs are not realized until years after the intervention. Most evaluations of workforce development programs have a follow-up timeline ranging from six months to four years. Shorter follow-up timeframes better isolate the immediate effect of a program, like gainful employment rates, but some of the meaningful outcomes cannot be measured until several years after the program, such as annual earnings. 

It is important to note that longer follow-up timelines decrease the strength of an effect estimate because, as length of time increases, it becomes more difficult to control for confounding, differential effects of time on members of the treatment group. Essentially, a shorter follow-up time frame allows for a stronger causal inference claim, but can overlook longer-term outcomes. A longer follow-up time period evaluates outcomes from a program that appear later, but has a weaker cause and effect link between an intervention/ program and an observed outcome. 

There are numerous sources online that provide great detail on developing different research designs.

Authored by Grady Brown; edited by Katherine Bailey and Daniela Eppler. 

Virginia Pay for Success Lab

The University of Virginia Pay for Success Lab launched in 2015 with financial support from Social Entrepreneurship @ UVA. The mission of the Lab is to reduce barriers for exploring PFS finance and inform the PFS field . The Lab undertakes research, project management, and program evaluations for PFS feasibility studies and projects. For more information, visit http://seatuva.org/pfslab