The Physics of Wall Street

In August 2007, a cascade of quant hedge fund losses — later called the quant crisis — announced to the world that something unusual was happening inside the black boxes of Wall Street. The funds running these strategies were staffed by physicists and mathematicians, and the press was quick to cast them as villains. But the story I want to tell is more complicated, and in many ways more interesting, than that indictment.

Isaac Newton despaired that he could calculate the motions of heavenly bodies but not the madness of men — referring to his losses in the South Sea bubble. Newton’s frustration captures a puzzle that has haunted thinkers for centuries: markets seem to defy the orderly mathematics that governs everything else in nature. Yet on average, professional fund managers underperform simple index funds. There has to be a better way.

Jim Simons’s Medallion Fund earned 2,478.6% over a decade, blowing every competitor out of the water. Renaissance Technologies employs about two hundred people, a third of them with PhDs — not in finance, but in physics, mathematics, and statistics. The story of how physicists learned to do science on markets is the story this book tells, from its origins in a Parisian dissertation in 1900 to the hedge funds of the twenty-first century.

In 1955 or thereabouts, Paul Samuelson was handed a half-century-old dissertation by a Frenchman he had never heard of. Louis Bachelier. The document was astounding. Here, in 1900, was a complete mathematical treatment of how market prices change with time — what we would now call the random walk model. No economist had done anything like it.

Bachelier’s insight was that at the moment a trade is executed, neither buyer nor seller has an advantage: the buyer thinks the price will go up, the seller thinks it will not. Every piece of available information is already incorporated into the price. This means that future price movements must be unpredictable — random, in the technical sense. The mathematics of such a process was already known from physics, where Einstein would independently describe it five years later in his treatment of Brownian motion.

Poincaré supervised the thesis but was forced to conclude it fell too far from the mainstream of French mathematics to earn the highest distinction. Bachelier received the grade of honorable, not très honorable. The committee’s reluctance to reward work that used intuitions from finance to drive new mathematics set the tone for fifty years of neglect. The random walk model wasn’t even the punch line of Bachelier’s thesis — his real goal was a formula for pricing options, a problem that would not be solved again until Black and Scholes in 1973.

The random walk model is not the same as saying markets are perfectly efficient or that prediction is pointless. Bachelier’s options pricing formula shows exactly how to use the mathematics of randomness to extract value — the assumption that underlying prices are random is the key to the formula’s effectiveness, not an obstacle to it. Statistical predictions can still help investors, as long as you have a way to translate the formula into an investment strategy.

M. F. M. Osborne was a naval astrophysicist who began thinking about stock markets around 1956, while working at a government lab in Virginia. He imagined handing a page of the Wall Street Journal to a statistician trained in astronomy and totally unfamiliar with finance. Such a person, approaching the data with fresh eyes, would see something economists had missed: stock prices behave exactly like particles undergoing Brownian motion.

Osborne’s key improvement over Bachelier was recognizing that it is not the price itself but the logarithm of the price that follows a random walk. This log-normal formulation matched the empirical data and made sense from a behavioral standpoint — a $1 move in a $10 stock represents a much bigger psychological step than the same move in a $100 stock. Osborne published in Operations Research, not an economics journal, which meant his work reached theoretically minded practitioners who weren’t yet reading Samuelson.

The hallmark of Osborne’s approach was a willingness to challenge his own assumptions when the data stopped fitting. When he discovered that prices were not equally likely to move up or down at all times — that the infrastructure of the trading floor created systematic non-randomness — he didn’t defend the model. He modified it, producing his ‘extended Brownian motion’ framework. This iterative discipline, pulling yourself up by the bootstraps as you progressively understand what you’re studying, is the proper scientific way to work with models in finance.

If you continue to trade based on a model whose assumptions have ceased to be met by the market, and you lose money, it is hardly a failure of the model. It is like attaching a car engine to a bicycle and then complaining that it doesn’t fly. Not everyone who has worked with mathematical models in finance has been as sensitive to this methodological point as Osborne was — which is one of the principal reasons why mathematical models have sometimes been associated with financial ruin.

Benoit Mandelbrot walked into Hendrik Houthakker’s office at Harvard and saw a picture on the blackboard. ‘Isn’t that a wealth distribution plot?’ he asked. Houthakker explained it was actually a graph of daily returns from cotton markets — and that the data simply refused to fit the normal distribution that Osborne’s model predicted. This chance encounter in 1961 redirected Mandelbrot’s career toward finance.

What Mandelbrot saw in the cotton data was a Lévy-stable distribution: a family of distributions with fatter tails than the normal, first studied by the French mathematician Paul Lévy, whose late work Mandelbrot had absorbed as a disciple. The pattern of extreme moves in cotton prices was self-similar — the same disproportionate concentration of large swings that characterized the full data set showed up in any subsample. This was the same mathematical structure Mandelbrot was beginning to recognize everywhere, from wealth distributions to coastlines to fault systems.

The implication for finance was alarming. In Mandelbrot’s framework, markets are ‘wildly random’ rather than ‘mildly random.’ The normal distribution assigns negligible probability to moves of ten or twenty standard deviations; Lévy-stable distributions assign them genuine weight. The rare catastrophic event is not an anomaly to be set aside — it is a structural feature of the market’s behavior. Mandelbrot called this property ‘wild randomness,’ and he spent decades arguing that standard financial models were systematically underpricing tail risk.

Pushing forward with simpler available tools while Mandelbrot and his early converts worked out the consequences of fractals and self-similarity was the only sensible choice. You need to start with the simplest theory that works, get as far as you can, and then ask where the theory has gone wrong. But by the end of the 1990s, the dedicated core of mathematicians and statisticians testing Mandelbrot’s proposals with ever more detailed data reached a clear verdict: the tails of financial return distributions are far fatter than the normal distribution predicts, and this matters enormously for risk.

Edward Thorp arrived in Las Vegas in the early 1960s with a system for beating blackjack that he had worked out as a physics PhD. Card counting — keeping track of what has already been played and adjusting your strategy accordingly — gives the player a genuine edge by exploiting the fact that a shoe already rich in tens and aces is more likely to produce a blackjack on the next hand. With his backer Mr. X, Thorp turned $10,000 into $21,000 in thirty man-hours of play. The casino is just a special case of a broader principle: when you can access even partial relevant information, you can compare market odds to true odds.

John Kelly Jr. at Bell Labs had established the essential connection between information theory and gambling. The Kelly criterion specifies exactly what fraction of your wealth to bet when you have an edge: advantage divided by payout. If you always follow this rule, you are guaranteed to outperform any other betting strategy in the long run. Thorp recognized that the stock market was not so different from a casino game — you make bets based on partial information about the future, and if things go your way, you get a payout. The Kelly criterion applied to finance as naturally as it did to blackjack.

Thorp read the collection of essays featuring Bachelier, Osborne, and Mandelbrot in the summer of 1964 and was quickly convinced that stock prices behave randomly in the precise sense those authors described. He then derived an equation telling him what a warrant should really be worth. What Thorp had that Bachelier and Osborne never imagined was five years of gambling experience: calculating a ‘true’ price for a warrant is a lot like calculating the ‘true’ odds on a horserace. If the market price differs from your theoretical price, you can construct a hedge.

Thorp’s warrant-pricing model was, in retrospect, equivalent to what Black, Scholes, and Merton would derive nearly a decade later. Thorp used a computer program to calculate prices rather than derive an explicit closed-form equation — and that difference in presentation cost him credit for one of the most important results in twentieth-century finance. The underlying argument was different from Black and Scholes’s, but the output was the same: a way to price and hedge derivative securities that would make arbitrage-free markets possible.

Fischer Black came to finance from physics and mathematics with an advisor who worried about his ‘dilettantism.’ After struggling for months on a differential equation describing the instantaneous rate of change of a stock price, Black gave up — he didn’t know enough advanced mathematics to solve it. When he and Scholes pooled their approaches in 1969, something clicked. Robert Merton, an engineer by training, independently rederived the same equation from an entirely different starting point. With two routes to the same answer, all three were convinced they were on to something big.

The options pricing formula Black, Scholes, and Merton discovered went through rejection after rejection from academic journals before finally appearing in 1973 — the same year the Chicago Board Options Exchange opened and the International Monetary Market launched. Between these two new markets, Black and Scholes found a world perfectly poised to take advantage of their new ideas. Within months, traders on the CBOE floor were using handheld calculators running the Black-Scholes formula.

When the market crashed in 1987, everyone with portfolio insurance tried to sell their stocks at the same time. There were no buyers — everyone was selling. Computers executing the trades ended up selling at far lower prices than the designers of the portfolio insurance strategies had expected, because the strategies assumed there would always be a market to trade into. This was a case where the model worked fine under normal conditions, but its assumptions — that there would always be a liquid market — broke down catastrophically precisely when they were most needed.

O’Connor and Associates survived 1987 by being a little more sophisticated in how it used its models. Michael Greenbaum had built a risk management team in the late 1970s specifically because he recognized that Black-Scholes was failing to properly account for extreme events. The volatility smile that appeared suddenly after the crash — implying that options with different strike prices required different volatility assumptions — was not evidence that models are unreliable in principle. It was evidence that the specific assumptions of Black-Scholes had ceased to hold. The model hadn’t failed; its assumptions had.

Doyne Farmer and Norman Packard met as graduate students in Santa Cruz and built a hidden computer — worn under clothing, signals transmitted via vibrating magnets strapped to the torso — to predict the motion of a roulette ball. Their group, calling themselves the Eudaemons after Aristotle’s concept of ideal human flourishing, proved you could beat a casino with physics. Farmer once had to excuse himself every ten minutes claiming stomach trouble while the wires of his hidden computer burned his skin. The practical difficulties of casino roulette eventually convinced them that the real opportunity was elsewhere.

In 1991, Farmer and Packard left their positions at Los Alamos and the Santa Fe Institute to found what would become the Prediction Company. Their goal was to do the impossible: to predict the behavior of financial markets. The key insight from their years at the Santa Fe Institute’s economics conferences was that financial markets might exhibit the same kinds of patterns their chaos-theory work had identified in other complex systems — not deterministic prediction, but statistical regularities with temporary predictive power.

It is tempting to say that the Prediction Company ‘used chaos theory to predict the markets.’ In fact, this isn’t quite right. Farmer and Packard didn’t apply chaos theory as a meteorologist applies fluid dynamics — they used methods developed for studying chaotic systems to search the historical data continuously for patterns that had some temporary predictive power, then developed tools to evaluate how long those patterns remained valid. Their approach was modest and conservative: find a pattern, trade it while it lasts, recognize when it has passed its prime.

Over the firm’s first fifteen years, one knowledgeable source told me that its risk-adjusted return was almost one hundred times larger than the S&P 500 return over the same period. The Prediction Company survived as an active subsidiary of UBS long after most of its quant peers had failed. What distinguished it was not a single grand theory of markets, but a rigorous, self-critical methodology for finding and validating short-lived statistical regularities — the same empirical discipline that had served Osborne so well forty years earlier.

Didier Sornette was a materials scientist studying how pressure tanks fail when he realized that the pattern of small cracks preceding a catastrophic rupture looked exactly like the log-periodic oscillations he was observing in financial data before market crashes. The moment of inspiration came in 1991. A market bubble, in Sornette’s framework, is not just a period of inflated prices — it is a period of collective herding behavior that builds toward a critical point at which the system must collapse, much as a material builds internal stress toward a fracture.

According to Sornette, the standard economic reasoning that bubbles can end only with dramatic external news is wrong. The crashes he studied — including the 1987 crash and the 1997 Hong Kong crash — appear to result from internal instabilities in the market itself. The log-periodic signature of an approaching critical point is in principle visible to anyone who knows what to look for. Since first predicting the October 1997 crash, Sornette has had a remarkable track record: he identified the log-periodic pattern in advance of the September 2008 crash.

Sornette distinguishes ‘dragon kings’ from ordinary black swans. Black swans, in Nassim Taleb’s sense, are extreme events drawn from the tail of the usual distribution — they are simply very rare instances of the same underlying process. Dragon kings are a different category: outliers that arise from fundamentally different mechanisms than the rest of the distribution. They are monstrous, but they are not random. Because they emerge from a distinct causal process, they can in principle be anticipated — which is precisely what Sornette’s methods attempt to do.

Sornette’s anti-bubble work shows the same logic in reverse. In January 1999 he posted a paper claiming that the Japanese Nikkei was in an anti-bubble — a period of artificially suppressed prices with a log-periodic signature predicting recovery. Many economists dismissed the claim as a publicity stunt. By the end of the year, the Nikkei had recovered by precisely the 50% Sornette had predicted. If Mandelbrot showed that markets are more wildly random than Bachelier or Osborne imagined, Sornette showed that at least some extreme events can still be anticipated if you know what pattern precedes them.

Eric Weinstein and Pia Malaney attacked the index number problem — how to compress complex information about the cost and quality of goods into a single number — using gauge theory, the branch of mathematical physics that Jim Simons himself had helped develop. Gauge theories use geometry to compare apparently incomparable physical quantities: lengths of rulers at different locations, or baskets of goods consumed by people with different tastes. Malaney argued her dissertation at Harvard; Jorgenson’s response was to throw her out of his office.

Weinstein had arrived at his collaboration with Malaney after a period of deep skepticism about whether mathematics could be productive in economics at all. What changed his mind was confronting Arrow’s impossibility theorem — the proof that no voting system can turn the ranked preferences of all individuals in a community into a fair community-wide ranking. This was not hand-waving; it was a rigorous mathematical result with direct economic consequences. If mathematics could produce that, it could produce more.

The Weinstein-Malaney proposal is different from every other chapter in this book. Every other physicist I discussed was looking at a bunch of statistics — stock prices, market moves, annual returns — and trying to make predictions about how the numbers would change in the future. Weinstein and Malaney were trying to understand what the numbers mean in the first place. The index number problem is not about prediction; it is about the foundations of how we measure economic welfare. That, too, is a problem where physics has something to offer.

Exporting gauge theories to economics remains a hard sell. The biggest danger facing mathematical modelers is the belief that today’s models are the last word on markets. Weinstein was right that late 2008 presented a unique opportunity for someone inclined to change the way economists thought about the world. But the opportunity was largely squandered — the crisis produced regulatory reform, not intellectual reform. The institutions that might have supported genuinely new thinking chose instead to patch the existing framework.

The excesses of the 2000s that led to the recent crash were enabled by physicists and mathematicians who didn’t understand the real-world consequences of what they were doing, and by profit-hungry banks that let these quants run wild. There is much that is right in this criticism. But it trades on a misunderstanding of what mathematical modeling in finance actually is, and what its failures reveal.

Using physics as a springboard for new ideas in finance does not involve describing people as though they were quarks or pendulums. Mandelbrot and Osborne made progress by drawing on their familiarity with statistics to identify patterns in data. Farmer and Packard brought tools from chaos theory. Sornette adapted methods from seismology. None of them assumed markets were physical systems. They assumed that the methodology of physics — start with the simplest tractable model, push it until it fails, then revise — was applicable to financial data. That assumption has been vindicated repeatedly.

Just as O’Connor survived 1987 by being more sophisticated than anyone else in how it used its models, Jim Simons’s Renaissance Technologies returned 80% in 2008 — again by being smarter than the competition. The difference between Renaissance and other hedge funds is that Renaissance has figured out a way to do what most people believe is impossible: do science on financial data, with the same rigor and iterative discipline that scientists apply to physical data. The crisis resulted from failures to apply this discipline, not from applying it.

The biggest danger is not bad models — it is the belief that today’s models are the last word. The stories in this book show the right methodology in action: use simplifying assumptions to make a problem tractable and solve it, then double back and ask what happens when you play with those assumptions. Sometimes you realize the original solution was exactly right; more often, you realize the original solution was a good approximation that becomes less good under extreme conditions. The lesson is not to stop modeling. It is to model better, always treating current models as the beginning of an inquiry, not its end.