Blog: Test

August 22, 2019

Last week we discussed the third step in The Lab’s project cycle: Do Something. This week we explore step four: Test.

On the go? Listen to this blog as a podcast.

What’s the best money you’ve ever spent? Maybe on a product, a class, a trip, or even just a nice dinner? What was the purchase that you would make again 1,000 times over? Personally, I may have found true happiness in buying a simple cookbook.

What about the other side of that coin? What’s the purchase that you wish more than anything you could undo? My wife and I just bought a clothes dryer—our second in 8 months. If only we'd known in December what we know now. 

When faced with a decision, big or small, we can’t know what will happen in the long run. We’re all human, so we tend to focus on the potential positives. We imagine more purchases like the cookbook and less like the dryer. It would be great if we had more chances to try before we buy.

In any sector—public, private, or non-profit—it can be tempting to only see the upside of every innovative new idea. In government, it might be the health care program that worked so well in Scandinavia; the educational campaign that hits all the trendiest social media platforms; or the technology with the potential to make government “friction-less.” But do these innovations live up to their greatest promises? 

This is why DC government is testing as many programs and services as we can before deciding whether to invest in them long-term.  The Lab @ DC is rarely the source of those innovative ideas, but instead, we provide DC agencies with the methods to try before we buy:

Don’t Agonize, Randomize!
When we build new policies or programs the process is full of what-ifs. Should we make the benefit $700 or $500 per month? Should we have participants meet with a case manager twice a month or once? 

As policymakers, we’re trying to create the best possible service for residents and it can be agonizing to think about what might happen if we get even one small detail wrong. That’s why we encourage our agency partners to start out small and use randomized evaluations to test everything we work on together. 

For example, the Department of Human Services believes it has an innovative coaching model to better connect job-seeking low-income residents with education and employment, but they don’t want to roll it out to thousands of residents without being sure. We’re helping the agency to randomly assign a few hundred job-seekers to this new coaching model, while the rest receive their usual services. We’ll see if that model leads to better education and employment outcomes before expanding it.

On a smaller scale, when the Metropolitan Police Department was mailing postcards to possible recruits, they didn’t need to decide up front which slogan to use and whether a photo of a male or female officer worked better. Instead, we helped them test multiple options.

Predict, but Verify
We can’t be everywhere at once. That simple fact means that every day public servants need to decide where to send precious resources, like police officers or snow plows. The Lab’s work in predictive modeling can help make that decision. Here’s one example:

Rodents, unfortunately, are a problem throughout the city, but there’s no way DC Health’s Rodent Control Team can inspect every block every day. We worked with Rodent Control to create a predictive model that uses data about things we know about—like where restaurants and alleys are located—to make predictions about things we don’t —like where we’re most likely to find rats in a specific week. Our hope was that they could find rats before they multiplied.

Based on cutting-edge analysis of historical data, the model told us very precisely which DC blocks were most likely to have rodent burrows. But did we buy that? Not without testing it in the real world. We took the city blocks that our model predicted would have rats and used a randomized evaluation to see if our model could work in real-time. It turns out that in real time the model could not meaningfully improve operations for the Rodent Control Team. That’s exactly why we try before we buy.

No Randomized Test? Use the Next Best
In each of the examples so far, our agency partners have been able to build in randomized tests where we might have previously just guessed about what worked. But, what if it’s not possible to test things using a randomized evaluation? That’s when we have to get a little more creative.

Take DC’s Crime Gun Intelligence Center (CGIC), for example. The CGIC links evidence across criminal cases where the same gun is used. In 2017, the Metropolitan Police Department received a federal grant to test improvements to CGIC services for the 7th Police District (7D). Because the improvements were only being done in 7D, randomization was not possible. So, we began evaluating the CGIC improvements, by comparing crime rates, gunshot alerts, and gun-related arrests in 7D to a combination of areas in DC that are similar to 7D. The results will not be as clear as if we could have randomized, but it’s far better than not testing at all.

Sometimes, a new service is being rolled out everywhere and for everyone, making it hard to test, but that doesn’t mean we have to walk away. When the District Department of Transportation piloted dockless bike-share across DC, we used as much data as possible to measure whether overall ridership increased during the pilot and whether the bikes were used across all 8 wards.

Next Up
Over the last 2+ years, we’ve created dozens of opportunities for DC Government to try before we buy. So, what are we buying? Next week, you’ll see that the decision to invest isn’t always that simple, but at the very least, we’re making sure we know what we’re getting for our money. 

Next Up: Decide.

Sam Quinney is The Director of The Lab @ DC. He lives in Ward 6 with his wife, infant son, and dog. He is currently testing any and all approaches to get his 10-month old to say “Dada” before “Mama.” So far, none of his innovative ideas are living up to their greatest promise.