The Counterfactual Challenge:How Machine Learning Can Enhance Policy Evaluation

Abstract

Understanding the impact of public policies and programmed actions is a complex and challenging task that deserves attention and resources. Even if the economic literature on counterfactual methods and identification is well established, conventional methodologies still struggles with holding up the unconfoundedness assumption in complex socio-economic and policy contexts. This paper sets up a ‘counterfactual challenge’, testing the ability of conventional matching versus a novel application of Supervised Machine Learning classification process to identify suitable counterfactuals. Working with high-dimensional data in complex socio-economic and policy contexts, results show that Machine Learning algorithms are better equipped to effectively balance treatment and control groups across a wide range of covariates compared to conventional matching methods. In the context of decision making and policy planning, we show the potential of Machine Learning to drastically improve the reliability and precision of information supporting policymakers in their choices. This is argued to have a positive impact on the effective use of public resources especially in complex and underdeveloped areas and contexts. Improvements in the precision of impact evaluation could result in significant gains in resource efficiency, both by generating realistic expectations of policy outputs, and improving scarce resource allocation we show that the use of Machine Learning algorithms for counterfactual identification consistently provides more precise results and supports policymakers in navigating complex contexts.