User:Econpragma/Credibility Revolution (Economics)

The credibility revolution is an expression used for the first time by Joshua Angrist, et Jörn-Steffen Pischke in their seminal paoer to describe the increase in the reliability of the empirical tests in economics. Such an evolution is sometimes labelled the "empirical turn of economics", tough this does not imply that economists did not test their theories before.

The concept of 'credibility revolution' encompasses three key changes in the way economists test their theories in the scientific literature. Firstly, economic research papers relies now increasingly on experimental and quasi experimental research designs like differences in differences, instrumental values, regression discontinuity design, natural experiments, or even randomized experiments. Such techniques are deemed to have a higher internal validity than mere multiple linear regressions or VAR that were massively used before. Secondly, better statistical practices are now adopted. For instance, papers now present several statistical models to reduce the risk of data dredging or p-hacking. Thirdly, such an improvement in empirical work is also due to the availability of more and better data.

This narrows the gap between the empirical methods of economics and those prevailing in other fields - in experimental psychology for instance, or even in medical disciplines like epidemiology.

Such an 'empirical revolution' is however not fully consensual. Both the reliability of the empirical techniques above-mentioned and the impact of this evolution on the activity of the economists have been put into question.

Empirical data assessing the reality of the phenomenon
A study of D Hamermesh (2013) is often quoted to quantity the phenomenon. In 1963, 51% of economics paper were purely theoretical. In 2011, this share has declined to less than 20%, "with most of the decline taken up by empirical studies for which the author(s) created the data set. The rest of the decline is accounted for by growth in theory with simulation (mostly macroeconomic calibrations) and experimental work (either in a laboratory or in the field). " (p.168, Harmermesh, 2013). This paper however is not precise enough about the methods used in these empirical papers.

An empirical study of Henrik Jacobsen Kleven about papers in public economics of the National Bureau of Economic Research allows a more detailed analysis of the currrent trend in economic scientific literature. While the above-mentioned study designs were barely absent in the literature in the 1980s, lab and controlled experiments appeared in 10% of the total amount of the papers published, quasi experiments and natural experiments in 20% of it, and differences in difference in about 25% of this total in 2011. Such methods are more credible in the sense that they allow to reduce the risk of biases like the omitted variable bias, reverse causality bias one or even to circumvent them in the case of randomized experiments.

Shortcomings and critiques
Results of empirical studies are not always reproducible as in others experimental disciplines. Studies should always be replicated several times. Such replications can be integrated in a meta-analysis and treated to circumvent certain biases like the publication bias. This is here less a critique of the methods than a call for cautiousness when reading the result of one study.

Another frequent critique is presented by Angrist et al (2010) : "Critics of design-driven studies argue that in pursuit of clean and credible research designs, researchers seek good answers instead of good questions". Indeed while the above-mentioned methods are more reliable, they are also easier to use to answer highly precise and local research questions, for instance : "what was the impact in the minimum wage in a federal State ?" . However, crucial questions in economists do not share this feature : "Is capitalism compatible with the preservation of the environment ?" "What will be the economic impact of a breakdown of the euro zone ?". Such a change in research practices would thus be the death of economic theory. The risk is that in search of easy-to-address research questions, economists would tend to circumvent the issues that matter.

To this critique, Angrist et al answer that "small ball sometimes wins big games". They show experimental studies were used to measure important structural parameters like labor supply elasticities, or the elasticity of intertemporal substitution. One could quote the empirical estimate of the value of the budgetary multiplier using such methods,. More generally, Angrist et al claim such studies allow to contrast competing hypothesis, thus ending long-standing academic debates and fostering consensus.

Although sometimes being deemed the 'golden standard' -for instance by Esther Duflo, randomized experiments also brought a flow of criticism. For instance, in his nuanced paper, Dani Rodrik (2008) questions the external validity of such experiments : "The typical evaluation will have been carried out in a specific locale on a specific group and under specific experimental conditions. Its generalizability to other settings is never assured" (Rodrik, 2008). Is a policy tested in a country A going to yield the same results in a country B ? There might also be so called general equilibrium effects when a policy that was tested locally is then generalized. If a policy was effective only because a part of the population was favored over the rest, then its effects mechanically disappear when the policy is generalized (see the notion of zero-sum game). Finally, others pointed out the fact that in economics it can be difficult to constitute placebo groups just like in medical studies.