How can we know if something is working?
We conduct evaluations to answer questions about the impacts of District programs, policies, and practices on important outcomes for residents. We could be asking questions about something as simple as an email to residents, something as complex as a housing subsidy, or something as resource-intensive as a job training program. Evaluation helps us shed light on questions like:
- Did this program help Washingtonians?
- Did the program save the District time, money, or reduce administrative burden?
- Which version of this program is more effective?
- For which group is this program most effective?
Sometimes, we get to plan an evaluation from the beginning—as a program is being designed, well before it is launched. In other cases, an agency is already doing something, and we’re asked to evaluate a program after it has been rolled out. Occasionally, we plan around one set of conditions, then a program changes midway through. In all of these cases, we want to help our agency partners and the District understand what happened using the best data available, and on a realistic timeline to inform key decisions.
The hardest part of evaluating a program is often identifying the counterfactual. We want to compare what really happened (e.g. with the program or intervention) to what would have happened (e.g. without the program or intervention) for the same population during the same time period. Social scientists use different methods to estimate this:
- The “gold standard” for measuring impact is a randomized evaluation.
- When a randomized evaluation is not possible or feasible, we use quasi-experimental evaluation methods to construct the best counterfactual we can.
Wherever possible, we also pair quantitative evaluations with qualitative research to help interpret and explain our findings, or design solutions informed by them.