USAID is often criticized for not rigorously evaluating its development programs. Like many other aid agencies, USAID has historically prioritized performance evaluations that measure the extent to which its programs generate their expected outputs, rather than impact evaluations that provide evidence of their net, attributable impacts. Senior USAID administrators have for the better part of the last decade expressed rhetorical support that more impact evaluations should be undertaken. However, high-level interest has not quickly or easily translated into practical changes in the way that that USAID does evaluation.
But things are finally beginning to change inside the Ronald Reagan Building and across the agency’s global network of field missions. A growing number of USAID-funded programs are being subjected to impact evaluations. These evaluations establish a credible counterfactual, addressing the question of what would have happened in the absence of a given USAID program in order to isolate the change in a development outcome—or a set of development outcomes—that is attributable to that particular program as opposed to other factors. One of the principal reasons why the agency is moving in this new direction is the availability of new data, methods, and tools.
A case in point is USAID West Bank/Gaza’s recent $900 million investment in rural infrastructure. Under the Infrastructure Needs Program II (INP II), USAID funded the construction or rehabilitation of 59 rural road segments in the West Bank. It was interested in subjecting the program to a rigorous impact evaluation. However, a randomized controlled trial (RCT)—the “gold standard” in impact evaluation—was not a viable option. Randomly assigning the placement of rural roads is neither feasible nor advisable. USAID decided to instead commission a geospatial impact evaluation (GIE). GIEs use precisely georeferenced intervention data and outcome data to establish a counterfactual retroactively, eliminating the need to randomly assign individuals, firms, or communities into treatment and control groups at the outset of a program. Also, many GIEs leverage readily available data like satellite observations, so they can often be implemented at a fraction of the time and cost of traditional RCTs (which usually require custom baseline and endline surveys).
We are members of a research team at AidData that conducted the evaluation of INP II (available here). We first identified the exact routes of the road segments supported by USAID and the precise dates when these investments were completed. In order to identify the geographical areas that could plausibly benefit from the improved conditions of these roads, we created 5-kilometer buffers around each of these 59 road segments. We then subdivided these “catchment areas” into 750 x 750-meter grid cells corresponding to our outcome measures. In order to measure a key outcome that the INP II sought to improve, we used a proxy for local economic development that is particularly useful in countries where GDP cannot be consistently measured over time and geographic space: high-resolution, high-frequency satellite imagery on nighttime light output. More specifically, we used NOAA’s Visible Infrared Imaging Radiometer Suite to develop a monthly measure of luminosity that could be consistently tracked between April 2012 (11 months prior to the first road improvements) and December 2016 (five months after the last road improvements) in each and every 750 meter square grid cell within 5 kilometers of an improved road segment. The maps provided in the figure below illustrate each step of this process.
With the resulting dataset of nearly 400,000 grid cell-month observations, we used a quasi-experimental panel framework to rigorously estimate program impacts due to completion of road improvements (“treatment”). We compared post-treatment nighttime light in each grid cell to counterfactual outcomes obtained from that same grid cell’s own preceding nighttime light levels and trends, as well as the outcomes of grid cells near not-yet-improved road segments. The variation in the timing of road improvements across grid cells and the inclusion of month- and grid cell-level fixed effects at fine geographic levels address concerns about confounding and omitted variables (e.g., the implementation of a nearby employment program, falling electricity costs in a particular area).
The results of the evaluation are encouraging. We find strong evidence that local economic output, as measured by remotely sensed nighttime light output, increased due to the implementation of INP II. In communities that benefited from improvements to multiple road segments, we find that INP II had even larger economic impacts. These findings suggest that USAID and other donors should consider prioritizing rural road improvements in areas with several potential access points to a larger road network.