The Model Must Be Right First

I ultimately found this article at Scientific American from David Freedman (hat tip: Glenn Reynolds) about the shortcomings of models a bit unsatisfying. The article is about the experience of, presumably, geology grad student Jonathan Carter:

Carter wanted to observe what happens to models when they’re slightly flawed–that is, when they don’t get the physics just right. But doing so required having a perfect model to establish a baseline. So Carter set up a model that described the conditions of a hypothetical oil field, and simply declared the model to perfectly represent what would happen in that field–since the field was hypothetical, he could take the physics to be whatever the model said it was. Then he had his perfect model generate three years of data of what would happen. This data then represented perfect data. So far so good.

The next step was “calibrating” the model. Almost all models have parameters that have to be adjusted to make a model applicable to the specific conditions to which it’s being applied–the spring constant in Hooke’s law, for example, or the resistance in an electrical circuit. Calibrating a complex model for which parameters can’t be directly measured usually involves taking historical data, and, enlisting various computational techniques, adjusting the parameters so that the model would have “predicted” that historical data. At that point the model is considered calibrated, and should predict in theory what will happen going forward.

Carter had initially used arbitrary parameters in his perfect model to generate perfect data, but now, in order to assess his model in a realistic way, he threw those parameters out and used standard calibration techniques to match his perfect model to his perfect data. It was supposed to be a formality–he assumed, reasonably, that the process would simply produce the same parameters that had been used to produce the data in the first place. But it didn’t. It turned out that there were many different sets of parameters that seemed to fit the historical data. And that made sense, he realized–given a mathematical expression with many terms and parameters in it, and thus many different ways to add up to the same single result, you’d expect there to be different ways to tweak the parameters so that they can produce similar sets of data over some limited time period.

The reason I found the article unsatisfying is that, far from explaining why economic models in particular fall short, the article actually asserted that regardless of what you’re modeling incorrect models can’t be corrected by recalibration. However, since solid models do, in fact, exist and are employed successfully every day something else must be going on in economic models.

I think there are several key takeaways from this. First and foremost the model must be right and you determine that a model is right by its predictive power without recalibration over time. Repeated recalibration doesn’t make a flawed model better.

However, accurate, reliable models exist and are used every day. How can this be? The answer is that the models were solid to begin with and we know that by their predictive power. Producing sound models requires judgment, experience, and the elusive thing called insight.

The perihelion precession of the planet Mercury does not follow the Newtonian principles. The Newtonian model does not predict its orbit correctly. It could not be accounted for by recalibration (they tried). Accounting for the deviations from the Newtonian model required the then-new insight of Einstein’s general relativity.

Why aren’t economic models, the models attacked in the title of the article, better? The early political economic philosophers like Adam Smith and David Ricardo had something that mathematical notation and mountains of data can’t provide: they had insight, in Ricardo’s case earned by working practically since infancy at the London stock exchange. But, importantly, they weren’t trying to produce quantitative results but qualitative explanations for behaviors.

Another problem is that the monetary and political stakes are very high and preferred outcomes may overwhelm analytical rigor.

In the final analysis I think we’re getting ahead of ourselves in demanding quantitative results from economics and there are good reasons to believe, since economics deals with purposeful actors rather than mechanical phenomena, that determinative, accurate, quantitative predictions may remain elusive forever. At the very least we’ll need considerably more insight than we have right now and, as I noted above, insight is elusive, too.