Randomized Evaluation — The Lab @ DC

What is a randomized evaluation?
Our best tool for figuring out if something works is a randomized evaluation. Also known as a randomized controlled trial or RCT, this method is the most rigorous and unbiased way to see if a program is causing the changes we observe. It has been central to modern science, from agriculture to medicine, and we think that, wherever feasible, it should be the primary tool governments use to gain clear evidence of the impact of their programs.

In a randomized evaluation, participants are assigned to one of two groups. One group receives the program, and the other doesn’t. The key is that the assignment is completely random, like flipping a coin: heads, you get the program; tails, nothing changes. This process leaves us with two groups—sometimes called treatment and control—that should be identical, on average. We can then compare their outcomes, such as school attendance or housing stability. Randomized evaluations can also have multiple treatment arms, to test different versions of a program.

Why is randomized evaluation important?
When we randomly assign participants in this way, any difference between outcomes for the two groups can be linked to the program. This matters because it helps us rule out other possible causes of the outcomes we observe, such as the types of people who applied or other changes happening over time.

Here’s an example:
About a quarter of the District’s 911 medical responses are to events that don’t require emergency services and can instead be treated in primary and urgent care clinics. An unnecessary 911 response could mean that fewer responders will be available for a real emergency. The District decided to try out redirecting these calls to in-house 911 nurses who can arrange for appropriate care. Now imagine that at the end of the year, officials see that unnecessary emergency room visits decreased by 10% from the previous year.

First, this would be great! It would mean that residents are using the emergency room more appropriately. Unfortunately, it's impossible to say if this change was caused by sending some 911 calls to the in-house nurses. Maybe the weather was better this year, and it was easier for residents to get to a clinic. Maybe individual doctors started calling patients when they were due for a checkup. Or maybe the mayor launched a DC-wide marketing campaign highlighting the importance of only using 911 for medical emergencies. Maybe more factors than just the nurses are at work.

Instead of facing all of these “maybes,” we ran a randomized evaluation to see whether nurses could reduce unnecessary emergency room visits. When done properly, randomization removes those maybes from the calculations. After the evaluation, if we’re sure that the nurses are the reason for reduced emergency room visits, we can plan the budget and supports needed to send all non-emergency calls to them. If the nurses are not the reason, then we should try something else.

We randomly assigned callers to either speak to the nurses or receive a standard 911 response. Because the assignment was random, we could expect the same proportion of callers in each group, such as male and female or Medicaid beneficiaries. The only difference between the groups is whether they spoke to the nurse or not.

In this example, we found that ambulance dispatches fell a lot when calls were sent to nurses, compared with the standard 911 response. Because the groups were randomly assigned, we can be confident that this difference is due to the nurses and not some other difference between the two groups.

*Randomized assignment means that the treatment and control groups should be very similar.*

Is it fair that some people, selected randomly, are not offered the program we are testing?
When considering the ethics of a randomized evaluation, we consider four key questions:

1. Do we know the program is beneficial for DC residents?
If we don’t know whether the program will have the intended benefits, a randomized evaluation is ethical and can help ensure that it does not cause harm. If it has no measurable effect, then our limited public resources are not spent on something that doesn’t work for residents.

2. Are there enough resources to serve everyone?
We consider whether there are enough resources for everyone to participate. If resources are sufficient and we have a strong reason to believe it is better than other options, then it would be unethical to not offer it to everyone we can. If resources are limited and we are unsure of the program’s benefits, a randomized evaluation can help determine if we should fund it for all eligible residents. Random selection is also typically a fairer way to choose participants for a limited program, rather than first-come, first-served or relying on referrals, which can unintentionally exclude certain people or groups.

3. Can we improve the program?
If we know the program is beneficial (Question 1) and we have enough resources to serve everyone (Question 2), then we wouldn’t use a randomized evaluation to test whether the program works. We could, however, randomly assign different versions of the program to test possible improvements. In this case, the control group receives one version of the program and the treatment group receives another version that tests an idea we think might improve the program’s impact. This approach is often called “A/B Testing”.

4. Would participating in the evaluation burden or harm anyone?
In addition to the first three questions, we also ask whether participating in the evaluation could burden or harm anyone. These harms could come from different sources:

One harm could be psychological burden of knowing you’re part of a randomized evaluation that may not benefit you directly. To reduce this burden, we work to be transparent about our projects and to engage residents—using civic design methods to get feedback, posting our plans and findings publicly, and holding informational community events.
Another harm could come from placing large burdens on participants, such as excessive surveys or interviews. We try to avoid this by designing evaluations that fit into regular government operations and rely on existing data rather than special sessions or extra surveys.
A third harm could occur if an evaluation fails to protect residents’ privacy. All of our evaluations follow strict data security rules. Our agreements outline responsibilities like using secure networks, encrypting laptops, storing data anonymously, and destroying data we no longer need.

In each evaluation, we weigh these potential harms against the possible benefits of improving a program and advancing District services. When needed, we work with external Institutional Review Boards (IRBs) to get additional guidance.

How does The Lab use randomized evaluations?
Now that you’ve read about randomized evaluations, check out a few examples of how we have used the technique to answer these questions for Washingtonians: