How Do You Measure Progress?

I found this piece at the Wall Street Journal by Christopher Mims on innovation very thought-provoking. Here’s a sample:

Should the U.S. invest in a generation of new intercontinental ballistic missiles? What has really propelled decades of consistently rising computer performance? Is research into new forms of nuclear power a dead end? And should we credit Elon Musk with revolutionizing the automobile industry, or is he just riding the coattails of history?

These are the sorts of questions that researchers who study the history of innovation and what it says about the future say we can now answer—thanks, of course, to innovation.

Much of it is devoted to the findings of this study by Anuraag Singh, Giorgio Triulzi, and Christopher L. Magee, “Technological improvement rate predictions for all technologies: Use of patent data and an extended domain description”, a graph from which is at the top of this post. Here’s Mr. Mims’s explanation of the study:

Using both previously untapped pools of data and new analytical methods, along with the usual tools of modern-day forecasting—namely, the predictive algorithms often described as “artificial intelligence”—they are taking a quantitative approach to examining how quickly technologies improve.

The result isn’t a crystal ball for what’s next. Indeed, one of the conclusions of this group of academics is that attempts to predict the exact nature of the next technological advance are doomed to fail. But their research could help us understand how quickly existing technologies are getting better.

As you might expect I have some reservations. What are they actually measuring? I’m not convinced that the number of patents filed is actually a reasonable surrogate measure for technological progress. I have read hundreds maybe thousands of patent applications. I have written one and received a patent. I think they are more reasonably considered a gauge of what people are interested enough in to invest in than in progress.

I also believe that they are looking at what is effectively a snapshot rather than the movie that represents actual technological progress. Said another way the progress in robotics may be 18.5% this year and 28.5% next year. Or .5%. Furthermore as the graph at the top of the page progress in different areas rather obviously proceeds as a “long tail” phenomenon—a few are developing very quickly indeed while most are barely developing at all. Under the circumstances what does “average rate” mean? I don’t think it has any meaning.

If the findings of this study are true, it’s relevant to the real world in a number of ways, in particular in terms of intellectual property and capital investment. In the past I have expressed my dissatisfaction with our present system of intellectual property and the findings of this study fully support my views. At the very least our system of patents and copyrights needs to be much, much more constrained both in time and in scope. Our present system is actually impeding creativity and progress rather than encouraging them.

But, if the rates of technological improvement in different areas vary based on something other that the degree of capital investment, shouldn’t there be differences in the deductibility of money spent on research and development to provide more encouragement to progress in areas in which we want greater progress but progress is lagging and less to areas in which progress is not lagging?

I would hasten to note that doesn’t even touch on a pet peeve of mine—that too often what is claimed to be R&D is actually marketing. That is especially egregious in the pharmaceutical industry.