AI can’t predict the movements of the stock market. That’s the message of Gregory Zuckman’s Wall Street Journal story:
Wall Street has long used automated algorithms for tasks such as placing trades and managing risk. But investors haven’t made much progress relying on AI to tackle their biggest challenge: beating the market. While some see ChatGPT as a way to boost sales and research efforts, the investing results using AI haven’t been especially impressive.
“Progress in applying AI to investing has been limited, though innovations in language modeling could change that in the years ahead,†says Jonathan Larkin, a managing director with Columbia Investment Management Co., which manages the $13 billion endowment for Columbia University and invests in various funds.
Wall Street had a head start in AI. Four decades ago, mathematicians-turned-quants including Jim Simons, founder of Renaissance Technologies, developed algorithms to turn investing decisions over to their computers.
He and other quants have spent years using machine learning, a type of AI. They have built trading models that can extrapolate from past data to identify patterns and develop profitable trades, with limited human intervention.
But few firms have found success turning all of their operations over to machines, quants say. And they haven’t enjoyed dramatic advances with self-learning or reinforcement learning, which entail training computers to learn and develop strategies on their own. Indeed, Renaissance and others rely on advanced statistics rather than cutting-edge AI methods, say people at the firms.
“Most quants still take a “theory-first†approach where they first establish a hypothesis of why a certain anomaly might exist, and they form a model around that,†says Mr. Larkin.
One big problem: Investors rely on more limited data sets than those used to develop the ChatGPT chatbot and similar language-based AI efforts. ChatGPT, for example, is a model with 175 billion parameters that uses decades—and sometimes centuries—of text and other data from books, journals, the internet and more. By contrast, hedge funds and other investors generally train their own trading systems using pricing and other market data, which is limited by nature.
Personally, I doubt that the problem is data. I suspect it’s more that the market isn’t rational as well as being susceptible to manipulation.
It really shouldn’t matter that the market is irrational or can be manipulated. The essence of the efficient market hypothesis (almost always misunderstood or mischaracterized) is that all prior information is imbedded in price, and that the next datapoint is best characterized as the product of a stochastic process. It doesn’t matter how smart you or your computer algorithm is.
There are two things that cannot be true simultaneously: the statement in the quoted passage, “Investors rely on more limited data sets than those used to develop the ChatGPT chatbot and similar language-based AI efforts. ChatGPT, for example, is a model with 175 billion parameters that uses decades—and sometimes centuries—of text and other data from books, journals, the internet and more. By contrast, hedge funds and other investors generally train their own trading systems using pricing and other market data, which is limited by nature” and the efficient market hypothesis. If the efficient market hypothesis is true, it is also true that an algorithm could be designed that predicted the movements of the market perfectly. Or vice versa. If no such algorithm can be designed, the efficient market hypothesis is false.
“If the efficient market hypothesis is true, it is also true that an algorithm could be designed that predicted the movements of the market perfectly.”
No it can’t. In the EMH the next price move is random. Period. Full stop.
Most programmatic trading is designed (for options, mostly) to capture tiny and very short lived deviations from options pricing theory. In a way, its the very mechanism that brings actual prices back in line with theoretical prices. That it requires such exotic and rapid computing is testament to how efficient the market is.
There is no theory of stock prices that involves time except for the primitive concept of discounted cash flows. Derivatives are more complicated, involving time and storage cost etc. If you are really interested, I’d say go look up the original work on EMH and stock prices by Gene Fama and his protege Ken French. Later, they did the definitive work on derivatives. I took both their classes and it was amazing to see the difference between what they were really saying and public commentary, even in sophisticated financial press.