DID OBAMA’S STIMULUS increase employment, or would employment have risen even without it? Will the Affordable Care Act increase health insurance coverage without causing medical costs to skyrocket? Would national testing of children improve education or worsen it? Most urgent questions of public policy turn on empirical imponderables, and so policymakers fall back on ideological predispositions or muddle through. Is there a better way?
Maybe government could do a better job at finding the facts. Jim Manzi proposes that policymakers rely on better empirical research. The gold standard for empirical research is the randomized field trial (RFT). Suppose that a pharmaceutical company claims that a new drug will cure baldness. The best way to test this claim is to randomly select two groups of bald men, and give the drug to members of one group and a placebo to members of another group. If the men in the first group sprout hair, while those in the second do not, we obtain strong evidence that the drug functions as advertised.
The RFT works better than most other types of empirical investigation. Most of us use anecdotes or common sense empiricism to make inferences about the future, but psychological biases interfere with the reliability of these methods. Serious empiricists frequently use regression analysis. Suppose, for example, that the analyst believes that drinking orange juice causes baldness in men. He could gather data by asking men whether they drink orange juice and then obtaining from them other information that could conceivably be related to baldness—for example, whether they wear excessively tight hats. The analyst crunches the data to see whether there is a correlation between drinking orange juice and baldness, controlling for the other factors like hat-tightness. If there is, he will more likely than not claim that drinking orange juice causes baldness and earn his fifteen minutes of fame in the New York Times science section.
Regression analysis is inferior to RFT because of the difficulty of ruling out confounding factors (for example, that a gene jointly causes baldness and a preference for tight hats) and of establishing causation. But RFT has its limitations as well. It is enormously expensive because you must (usually) pay a large number of people to participate in an experiment, though one can obtain a discount if one uses prisoners, especially those in a developing country. In addition, one cannot always generalize from RFTs. Clinical studies for new pharmaceuticals use ordinary adults, and one can only hope that the results apply to children, pregnant women, and elderly people as well. Thus, academic research proceeds in fits and starts, using RFT when it can, but otherwise relying on regression analysis and similar tools, including qualitative case studies, which can be useful for stimulating hypotheses.
The superiority of RFT as well as its limits are well established in the academic literature. According to Manzi, businesses also use RFT whenever they can. A business such as Wal-Mart, with thousands of stores, might try out some innovation like a new display in a random selection of stores, using the remaining stores as a control group. If sales rise in the stores with the new display relative to sales in the other stores, one can infer that the display increased business.
In the first two thirds of his book, Manzi describes the historical development of the RFT and its philosophical basis, and includes a digression on the philosophy of science. The argument will be familiar to empiricists and philosophers, but it may interest a popular audience, and is well done and readable. Manzi argues that RFT should be used in universities, businesses, and government more often than it currently is, but in making this argument he sails with the wind at his back. Few people would disagree with him.
A more ambitious argument emerges in the last part of the book. Manzi argues that the RFT—or more precisely, the overall approach to empirical investigation that the RFT exemplifies—provides a way of thinking about public policy. This is the most imaginative and interesting part of Manzi’s book, but it is less successful than the earlier chapters.
It becomes clear only in the later sections why Manzi included the chapters on the philosophy of science. The book is less interested in the RFT than in the limits of empirical knowledge. Given these limits, what attitude should we take toward government? As Manzi explains, our grasp of the universe is shaky even where, as in the case of physics, “hard science” plays the dominant role. The scientific method cannot establish truths; it can only falsify hypotheses. The hypotheses come from our daily experience, so even when science prunes away intuitions that fail the experimental method, we can never be sure that the theories that remain standing reflect the truth or just haven’t been subject to the right experiment. And even within its domain, the experimental method is not foolproof. When an experiment contradicts received wisdom, it is an open question whether the wisdom is wrong or the experiment was improperly performed. Thus, when a recent experiment appeared to show that neutrinos could travel faster than the speed of light, scientists did not chuck the theory of relativity, but looked for ways in which the experiment may have gone awry. Much of scientific knowledge turns out to depend on norms of scientific behavior, good faith, convention, and other phenomena that in other contexts tend to provide an unreliable basis for knowledge.
Under this view of the world, one might be attracted to the cautious conservatism associated with Edmund Burke, the view that we should seek knowledge in traditional norms and customs, which have stood the test of time and presumably some sort of Darwinian competition—a human being is foolish, the species is wise. There are hints of this worldview in Manzi’s book, though he does not explicitly endorse it. He argues, for example, that we should approach social problems with a bias for the status quo; those who seek to change it carry the burden of persuasion. Once a problem is identified, we should try out our ideas on a small scale before implementing them across society—except in those instances where an immediate all-or-nothing decision is called for, as occurs during wars and economic crises. Meanwhile we should encourage variation across society, so that ideas can be tried out in different conditions, and proposals will bubble up from below. Manzi thus embraces federalism, and argues that we should permit states (and smaller-scale localities) to experiment with different policies.
Manzi concludes his book with a few policy proposals. Pursuing the theme of federalism, Manzi argues that the federal government should institutionalize policy waivers, so states can opt out from national programs and pursue their own initiatives. A state should be allowed to opt out of federal penalties for drug crimes, for example. Manzi also thinks that a new federal agency should be created with the mission to conduct, manage, and keep tabs on RFTs that test policy initiatives. And Manzi wants to improve the stock of human capital by deregulating schools and giving green-cards to foreigners with science and engineering degrees.
But Manzi does not develop his proposals very carefully. It is one thing to say, as he does, that federalism is useful because we can learn as states experiment with different policies. But Manzi takes away much of the force of this observation when he observes, as he must, that the scale of many of our most urgent problems—security, the economy—is at the national level, so policymaking in response to these problems cannot be left to the states. He also worries about social cohesion, which must be maintained at a national level even while states busily experiment. Presumably, this implies national policy of some sort, although Manzi is vague about what it would be.
Manzi’s commitment to federalism and his technocratic approach to policy, which relies so heavily on RFT, sit uneasily together. The RFT is a form of planning: the experimenter must design the RFT and then execute it by recruiting subjects, paying them, and measuring and controlling their behavior. By contrast, experimentation by states is not controlled: the critical element of the RFT—randomization—is absent. Typically, one or two states experiment with a policy, and then other states imitate the first movers if the experiment appears successful, or if they are neighbors or share a similar population—a phenomenon known in the political science literature as policy diffusion. The right way to go would be for the national government to conduct experiments by implementing policies in different states (or counties or other local units) by randomizing—that is, by ordering some states to be “treatment” states and other states to be “control” states, while closing the borders to prevent people from migrating to whichever states implement the policy they prefer. Federalism serves various purposes, but the “laboratory” function has never worked as well as advertised.
Another problem emerges in a brief discussion of school reform. Manzi advocates charter schooling—indeed, for-profit schooling—because schools can experiment with educational policy only if they enjoy independence. But his proposal turns out to require a huge intervention by the national government. In order to ensure that charter or for-profit schools teach students something of value, Manzi advocates a system of national testing. He does not seem to realize that, if this approach were taken, everything that children learn would be determined by bureaucrats in Washington. Competition among schools would take place, but only subject to this constraint. Thus, communities would no longer experiment with different curricula—and so educational innovation would be curtailed. Federalism is more complicated than it may appear.
Or consider Manzi’s argument that government should use immigration law to “recruit” scientists and engineers. How does he know that we need scientists and engineers rather than masseuses and chiropractors? And which scientists and engineers do we need? The current system, under which employers do the recruiting, is more decentralized, more bottom-up; Manzi rejects it because he has convinced himself that the United States must build up its human capital. And the other element of the immigration system, which gives preference to close family relations, may explain America’s unique ability to maintain social cohesion despite a continuous influx of foreigners. Manzi’s reasoning reflects the top-down approach to social policy that he is otherwise skeptical of—although, to be sure, he is willing to subject his proposals to RFTs.
Eric A. Posner is a professor at the University of Chicago Law School.