There’s a nicely thought-provoking article at Ars Technica by Cathleen O’Grady that dips its toe into the waters of one of the thorniest issues in science. Just how confident should you be about any particular scientific finding?
And it should come as no surprise that the same people have funny ideas about which sciences are precise.
Uncertainty in science can creep in at different places. Perhaps the most obvious one is measurement error: how well-tuned are the instruments you’re using? But there are more complex areas of uncertainty, too: science generalizes from an observable sample of the data to entire populations; it draws on existing data to make projections; and there is often disagreement over how results should be interpreted.
Broomell and Kane surveyed 217 people online to find out how they perceived uncertainty in a range of scientific fields. People were given a brief description of each field and then rated each field for 14 different dimensions of uncertainty. The questions covered factors like expert disagreement, measurement accuracy, over-generalization of results, and amount of abstraction. People were also asked to rate how valuable they thought each field is using a seven-point rating of the field’s quality, social benefit, influence over their personal decisions, and how much funding it should be given.
The results suggested that the dimensions of uncertainty that had something to do with precision (like randomness and measurement accuracy) played the biggest role in people’s perceptions. The fields that were labelled as “least precise†were psychology and evolution. Also on this end of the scale were economics, climate science, and—wait for it—astrophysics. On the other end, forensics was perceived as the field with the highest level of precision, followed by aerospace engineering.
There are all sorts of other issues. For example, “science” (from the Latin word for “knowledge”) falls into two broad categories: the descriptive sciences and the predictive ones. Physics and chemistry are, broadly, predictive. Anthropology is descriptive. I’d generalize and say that all of the social sciences including economics are mostly descriptive.
Economics is an interesting case. It does provide some guidance on general propensities or tendencies but it can’t tell you whether the stock market will go up or down. Or what the unemployment or inflation rate will be next year.
One of the reasons for that is problems with observation. We don’t really know what the unemployment rate is now in the way we know, say, what the temperature of the liquid in a flask is now.
Or meteorology. Using the biggest, fastest supercomputers in the world our present models can tell you what the weather will be like in a particular location in a few hours or, with less confidence, in a few days. There’s much, much less confidence in predicting the weather a year or even a month in advance other than in the broadest terms. Or in predicting catastrophic events like tornadoes.
I definitely believe in evolution just as I believe in gravity but while I’m pretty confident about the effects of gravity if I step off the roof of a three story building I don’t have nearly the confidence level in the effects of evolution. A lot of that is because of the role of random variation in evolution.
Engineering on the other hand is heuristic. But that’s a subject for a different post.
Otto Neurath on human knowledge:
“We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. In this way, by using the old beams and driftwood the ship can be shaped entirely anew, but only by gradual reconstruction.”
Not surprising, cognitive studies have shown such differing perceptions for decades, even among scientists.
It is kind of ironic that the author relies on one study for this article though. What level of uncertainty should we assign to her conclusions?
Note, too, that the author of the article’s background is in two descriptive sciences. No apparent formal background in anything predictive.
What do you differentiate a predictive from descriptive science?
Read “How” for “What” in the preceding.
A descriptive science seeks to provide post hoc descriptions of phenomena along with possible analogies. A predictive science attempts to provide models that can be used to predict future outcomes. Rutherford summarized it as “all science is either physics or stamp-collecting.”
Economists have confused putting their observations into the form of mathematical formulae with models that can be used to make reliable predictions. I think among the reasons for that are they’re not empirical enough and attribute too high a confidence level for their findings.
I don’t find that distinction very clean. Moreover, Rutherford’s characterization (or was is Luis Alverez’s) is demeaning, as if there is no rigor outside of physics [mathematics] (the unfortunate influence of Plato all these years on). A lot of what goes on in what Kuhn called “normal science” is what he called “puzzle-solving”: Why is this like that? And in fact, judging from my conversations with researchers, what does go on is simply the amassing of data in the furtherance of a finer, more accurate description of the phenomena. Not much prediction there.
More basic than a model in a particular science is the paradigm within which the model is developed. And a paradigm is, we might say, extra-scientific — the scaffolding of beliefs and assumptions about the world that dictate to scientists what to look for and where and how to look for it, what counts as good or poor research. Physics has its paradigm, anthropology has its paradigm. To suppose there is no one standard of rigor across paradigms is to make a category mistake.
I try to always remember that we humans have five senses, some sharp, some not so. Things that are obvious to a canine, we cannot detect. Visuals to birds, we see as only brown, what they see in other frequencies, we cannot know. Be a bit humble, is all I say. Just because these five weak senses cannot detect God, does not mean He cannot detect you.
Another way to look at it is along a continuum of ambiguity. At one end, you have science that can be tested and retested and retested and the result is always the same – science as wholly verifiable, without ambiguity. At the other end is science where empirical testing is not possible or the available evidence is scant and highly ambiguous. At this end it may never be possible to move beyond theory.
Damn, I’m having a bad typing day. That should have been:
“To suppose there is one standard of rigor across paradigms is to make a category mistake.”
My kind of science.
Andy
I’ll admit right up front, as someone with two physics heavy engineering degrees, to being biased. But I find the one end of your spectrum to be real science. At the other end I find it to be, variously, heuristic and at generally useful, especially in production and at times policy settings, to mere speculation. Soft to downright mushy science, even at the risk of demeaning my original profession.
Dave Schuler: A descriptive science seeks to provide post hoc descriptions of phenomena along with possible analogies.
A prediction may concern present observations of the evidence of past events. So a paleontologist may conclude that there were once fishapods, then predict the placement of fishapod fossils in the geological strata.
There is certainly knowledge we would consider scientific in pure description, such as Linnean taxonomy. However, the vast majority of sciences make every attempt to be predictive, even if tentatively so.
Gray Shambler: I try to always remember that we humans have five senses, some sharp, some not so.
“The fact that we live at the bottom of a deep gravity well, on the surface of a gas covered planet going around a nuclear fireball 90 million miles away and think this to be normal is obviously some indication of how skewed our perspective tends to be.†— Douglas Adams