Advances in Financial Machine Learning

The rate of failure in quantitative finance is high, particularly so in financial ML. The few who succeed amass a large amount of assets and deliver consistently exceptional performance. In my experience, there is one critical mistake that underlies most failures: discretionary PMs work in silos, and when that model is applied to quants — hire 50 PhDs and demand that each produce an investment strategy within six months — it always backfires.

Wherever I have seen that formula applied, it has led to disaster. Each PhD frantically searches for investment opportunities and eventually settles on some spurious pattern that backtests well on the data at hand. The result is a collection of overfit strategies that fail immediately in live trading, and the firm shuts down, blaming ML rather than the process.

Every successful quantitative firm I am aware of applies the meta-strategy paradigm. This book was written as a research manual for teams, not individuals. Through its chapters you will learn how to set up a research factory, where the role of each quant is to specialize in a particular task while having a holistic view of the entire assembly line.

The flexibility and power of ML techniques have a dark side. When misused, ML algorithms will confuse statistical flukes with patterns. Combined with the low signal-to-noise ratio that characterizes finance, this all but ensures that careless users will produce false discoveries at an ever-greater speed. This book exposes the most pervasive errors — and their solutions.

Time bars are perhaps the most popular among practitioners, but they should be avoided. Markets do not process information at a constant time interval, and time-sampled returns violate the IID assumption more severely than bars formed by information flow. Dollar bars — sampling an observation every time a pre-defined market value is exchanged — are my preferred starting point.

Dollar bars are more stable than tick or volume bars because the number of outstanding shares changes over a security’s life due to corporate actions, splits, and buybacks. Even after adjusting for splits, tick and volume counts are distorted; dollar value exchanged normalizes across these events and produces near-constant bar counts week-to-week.

Beyond standard bars, information-driven bars sample whenever tick imbalances exceed an expected threshold. Tick imbalance bars (TIBs) are produced more frequently under the presence of informed trading — asymmetric information that triggers one-sided order flow — making them natural detectors of information arrival rather than arbitrary time slices.

Volume imbalance bars and dollar imbalance bars extend the same logic to signed volume and dollar flow. The idea in all cases is to produce bars that contain equal amounts of information regardless of volumes or prices traded, yielding series with better statistical properties for ML input features.

The standard academic approach labels observations by fixed-time horizon: if the return over the next N days exceeds a threshold, label it +1; otherwise -1. This is wrong for investment applications. An investor cares about profit and loss, not whether a return crossed an arbitrary threshold at an arbitrary future date.

I introduce the triple-barrier method, which labels an observation according to the first of three barriers touched: a profit-taking barrier above, a stop-loss barrier below, and a vertical time barrier at the right. This is the labeling scheme that investment professionals actually use, and it makes more sense than any fixed-horizon alternative I have found in the literature.

Meta-labeling is a second innovation: given a model that sets the side of the bet (long or short), train a secondary ML algorithm only to determine the size, including the possibility of zero size. The primary model provides the signal; the meta-model controls the bet. This separation cleanly maps onto how discretionary PMs actually work and is exactly what the quantamental movement has been looking for.

You can add a meta-labeling layer to any primary model — an ML algorithm, an econometric equation, a technical trading rule, or even a human’s discretionary call. When the primary model is a discretionary PM, meta-labeling will tell us when to pursue or dismiss that call. The features used by such an ML algorithm could range from market information to psychological assessments.

Most non-financial ML researchers can assume observations are IID: one blood sample per patient, independent. In finance, outcomes overlap in time. If I hold a position from t=0 to t=5, and another position overlaps from t=3 to t=8, the two labels share information. Standard bootstrapping treats them as independent, which it demonstrably does not.

Sequential bootstrapping addresses the overlap problem. Rather than sampling uniformly, it reduces the draw probability of any observation whose outcome overlaps with an already-selected observation. The goal is to produce samples as close to IID as possible, which in turn reduces the correlation among individual estimators in an ensemble and makes bagging more effective.

Average uniqueness quantifies how much independent information each labeled observation contains: the fraction of its lifespan not shared with concurrent labels. This scalar can be used both to weight observations for bootstrap sampling and to weight them during training — highly overlapping outcomes should receive less weight than non-overlapping ones.

Return attribution provides a second weighting dimension: observations with larger absolute returns carry more information and should be upweighted. Combining return attribution with average uniqueness gives a two-dimensional weighting scheme that corrects for both temporal overlap and the relative magnitude of realized outcomes.

Financial ML faces a fundamental tension I call the stationarity-versus-memory dilemma. Price series have memory: every value depends on a long history of previous levels. Integer differentiation — computing returns on log-prices — achieves stationarity but at the cost of discarding all memory. The resulting series is near-noise, which is one reason the literature has been so biased toward the efficient markets hypothesis.

Fractional differentiation resolves this dilemma. By applying a differencing operator of non-integer order d, we can produce a series that is stationary yet retains as much memory as desired. The minimum d needed to pass an ADF stationarity test is the sweet spot: it removes just enough dependence to satisfy the inferential requirement while preserving the predictive signal.

The fixed-width window fracdiff (FFD) method implements fractional differentiation efficiently by truncating the infinite-memory expansion to a fixed window. This makes the computation tractable on long financial series and avoids the look-ahead problem that arises when using an expanding window on early observations.

On E-mini S&P 500 futures log-prices, the minimum d that achieves stationarity is well below 1.0. A full integer differentiation over-differences this series, destroying the memory that carries predictive information. FFD with d = d is a strictly superior input feature for any ML algorithm applied to financial prices.

Bagging reduces the variance of forecasts by averaging N estimators trained on independent bootstrapped subsets of the data. The ensemble forecast’s variance is a function of the covariance among individual estimators: bagging is only effective to the extent that those estimators disagree. Sequential bootstrapping (Chapter 4) directly lowers that covariance by making bootstrap samples as independent as possible.

Random forests improve on bagging by introducing a second level of randomness: when optimizing each split in a decision tree, only a random subset of features is considered. This decorrelates the individual trees further, reducing both variance and the risk of any single dominant feature monopolizing every split.

Boosting takes an entirely different approach: classifiers are fit sequentially, with each iteration re-weighting observations that were misclassified before. This corrects bias as well as variance. However, boosting’s sequential nature makes it prone to overfitting in finance, where the signal-to-noise ratio is low enough that sequential bias correction can latch onto noise.

In finance I generally prefer bagging over boosting. Boosting’s corrections may be appropriate in domains with high signal and clear patterns, but financial returns are so close to noise that the additional variance-reduction from bagging’s decorrelated ensembles is usually worth more than the bias correction from boosting.

Standard k-fold cross-validation leaks information in finance because labels overlap in time. When a testing observation at time t uses a label that spans [t-5, t+5], any training observation whose label overlaps with [t-5, t+5] has seen some of the same information. Leakage inflates cross-validation scores to the point where the evaluation is nearly useless.

The fix is purging: before each training fold, remove all observations whose labels overlap in time with any label in the testing fold. This is not optional; it is a correctness requirement. Without purging, hyper-parameter search and model selection are both corrupted.

Purging alone is insufficient when features incorporate serially correlated signals like ARMA processes. We additionally impose an embargo: a buffer of observations immediately following the test set are removed from training as well. The embargo width should be proportional to the serial correlation horizon of the features being used.

Purged k-fold CV is implemented as a drop-in replacement for scikit-learn’s KFold class. It should be used everywhere a train/test split appears: hyper-parameter tuning, backtesting, and performance evaluation. Using any other cross-validation scheme on financial data is a methodological error.

One of the most pervasive mistakes in financial research is to take data, run it through an ML algorithm, backtest the predictions, and repeat until a nice-looking result appears. Academic journals are filled with such pseudo-discoveries. Repeating a test over and over on the same data will likely lead to a false discovery. This methodological error is so notorious among statisticians that they consider it scientific fraud.

Feature importance — understanding which inputs actually drive a model’s predictions — is the antidote. I present three methods ordered by their sensitivity to substitution effects. Mean decrease impurity (MDI) is fast and computed on-the-fly inside random forests, but it is biased toward high-cardinality features and inflated by correlated substitutes.

Mean decrease accuracy (MDA), also called permutation importance, randomly shuffles one feature at a time and measures the resulting loss of predictive power. It is model-agnostic and more robust to substitution effects than MDI. When a feature’s permuted score drops sharply, that feature is carrying unique, non-redundant information.

Single feature importance (SFI) trains an entirely separate model using only one feature at a time, avoiding substitution effects entirely. It is the most conservative estimate. Comparing MDI, MDA, and SFI across the same feature set reveals which features are genuinely informative versus which appear important only because they share information with other features in the model.

I recommend developing models for entire asset classes or investment universes rather than for specific securities. If you find a pattern only on security Y, no matter how apparently profitable, it is likely a false discovery. Investors diversify; mistakes worth exploiting occur across instruments.

Hyper-parameter tuning done improperly is among the most reliable routes to a strategy that backtests beautifully and fails immediately in live trading. The ML literature correctly demands cross-validation of any tuned hyper-parameter, but standard CV schemes fail in finance. The purged k-fold CV from Chapter 7 must be used here as well.

I recommend scoring with neg_log_loss rather than accuracy when tuning for investment strategies. Accuracy is coarse: it treats a 51% confident correct prediction the same as a 99% confident correct prediction. Log-loss scores penalize miscalibrated probabilities and reward the full probability distribution, which matters directly for bet sizing.

Randomized search over hyper-parameters is preferable to grid search when the parameter space is large. Grid search evaluates every combination; randomized search samples combinations from specified distributions, finding good regions with far fewer evaluations. In finance, where CV evaluations are expensive due to purging requirements, this difference is material.

The goal of hyper-parameter tuning is not the best in-sample fit but the most stable generalization. Parameters that produce sharp cross-validation peaks — small regions of hyperparameter space with very high scores surrounded by degradation — are red flags: they suggest the model is fitting noise rather than signal.

There are fascinating parallels between strategy games and investing. Your ML algorithm can achieve high accuracy, but if you do not size your bets properly, your investment strategy will inevitably lose money. Bet sizing is the bridge between a classifier’s probabilistic output and a position size in dollars.

Given a predicted probability p[x] that the outcome is +1, I derive the bet size as m = 2Z[z] - 1, where z is the test statistic of the null hypothesis that p = 0.5 and Z is the normal CDF. This maps predicted probabilities to bet sizes in [-1, 1] with a natural zero at p = 0.5, scaling position size continuously with the ML model’s confidence.

A second approach uses the current signal’s rank within the historical distribution of signal strengths. The bet size is the probability that the current signal will not be exceeded by future signals: 0.9 when only 10% of historical signals were stronger. This strategy-independent approach does not require knowing the true probability of the outcome.

Discrete bet sizes can be derived by averaging multiple concurrent signals into a single position, with each signal contributing ±1. This avoids the need for re-balancing on every small probability shift and allows the position to evolve continuously as signals are added or removed from the portfolio.

Backtest overfitting is arguably the most important open problem in all of mathematical finance. It is the quantitative equivalent of p-hacking in statistics: the only backtests that most people share are those that worked, creating an acute selection bias. If there was a precise method to prevent backtest overfitting, a backtest would be almost as good as cash rather than a sales pitch.

Backtest overfitting can be defined as selection bias on multiple backtests. It takes place when a strategy is developed to perform well on historical data by monetizing random patterns. Because those random patterns are unlikely to recur, the strategy so developed fails almost immediately. Every backtested strategy is overfit to some degree.

The standard litany of backtest errors — survivorship bias, look-ahead bias, storytelling, data mining — is well known but regularly ignored. What is less appreciated is that even a rigorous walk-forward backtest can be overfit if the researcher has tried enough configurations before publishing. The backtest is out-of-sample in time but in-sample across the research process.

Practical defenses include: building models for asset classes rather than individual instruments; using bagging to reduce overfitting; applying the deflated Sharpe ratio, which corrects for the number of trials; and treating any strategy where you cannot articulate a structural economic reason for the edge with extreme skepticism.

Walk-forward (WF) and conventional CV backtesting schemes test a single path through history. A researcher can unwittingly select the configuration that happened to perform best on that single path, without any measure of how likely that performance is to recur. Single-path testing is insufficient evidence.

The combinatorial purged cross-validation (CPCV) method addresses this by generating the precise number of train/test combinations needed to produce a target number of distinct backtest paths, φ, while applying purging and embargos throughout. Testing on multiple paths dramatically reduces the probability that a researcher selects a strategy that over-performed on one lucky draw.

CPCV provides a direct estimate of the probability of backtest overfitting (PBO): the fraction of paths on which the in-sample best strategy fails to be the best out-of-sample. A high PBO signals that the apparent performance is due to selection bias rather than genuine skill.

The Sharpe ratio computed from CPCV paths also yields a distribution rather than a single estimate, allowing researchers to compute the probability that the strategy’s true Sharpe ratio exceeds any target. This probabilistic view of strategy evaluation is far more honest than reporting the single observed Sharpe from a walk-forward.

Backtesting on a single historical path means that, even if no overfitting occurs, the researcher’s assessment is conditioned on one realization of history. Synthetic backtesting augments this with many unseen paths drawn from a model fitted to the observed data, reducing the likelihood that the strategy has been fit to statistical flukes in the one path we happened to observe.

I focus on the optimal trading rule (OTR) framework, which searches for the profit-taking and stop-loss levels that maximize performance for a given stochastic process. The underlying price process is modeled as a discrete Ornstein-Uhlenbeck process, capturing mean-reversion that is common in spread and pairs strategies.

Calibrating a trading rule on a random walk through historical simulations leads to backtest overfitting, because one random combination of profit-taking and stop-loss that happened to maximize the Sharpe ratio will be selected. Our procedure prevents this by recognizing that performance exhibits no consistent pattern across synthetic paths when the process is a random walk.

When the OU process has a non-trivial mean-reversion coefficient, the OTR framework identifies trading rules that are robust across synthetic paths, not merely those that exploited noise on the observed path. This is the correct criterion for strategy selection: stability across the statistical ensemble, not performance on a single history.

The Sharpe ratio is the canonical measure of strategy quality, but its standard computation assumes IID Gaussian returns. Financial returns are non-Gaussian, autocorrelated, and fat-tailed. The probabilistic Sharpe ratio (PSR) corrects for this by computing the probability that the true SR exceeds a given threshold, accounting for skewness and kurtosis in the observed return distribution.

The deflated Sharpe ratio (DSR) goes further by correcting for the number of trials a researcher conducted before reporting the strategy. When 50 configurations are tried and the best is reported, the expected maximum SR under the null of no skill is substantially above zero. DSR deducts this selection bias from the observed SR, yielding a more honest estimate of true skill.

Implementation shortfall — the gap between paper portfolio performance and live trading performance — is one of the most common reasons investment strategies fail. Broker fees, average slippage per turnover, and market impact must be incorporated into the backtest statistics. A strategy that ignores these costs will almost always disappoint in production.

Time under water (TuW) measures how long a strategy goes without recovering to a new high-water mark. It is the investor’s experience of the strategy, complementing the Sharpe ratio’s summary of average risk-adjusted return. A strategy with a high SR but extreme TuW intervals may be uninvestable in practice due to redemption pressure.

Investment strategies are often implemented in terms of positions held until one of two conditions is met: profit-taking or stop-loss. Even when no explicit stop-loss is declared, there is always an implicit one at the point where the investor can no longer finance the position or bear the pain caused by increasing unrealized losses.

I derive the probability of strategy failure as a function of four parameters under direct control of the portfolio manager: precision rate, betting frequency, profit-taking threshold, and stop-loss threshold. This analytical decomposition lets the PM understand exactly which parameter needs adjustment to make a strategy viable before committing capital.

Unlike the PSR, which does not distinguish between parameters under or outside the PM’s control, this framework isolates the actionable levers. A strategy with insufficient precision can potentially be saved by narrowing the profit target, increasing betting frequency, or loosening the stop-loss — each of which has a quantifiable effect on the failure probability.

The method shares the binary outcome structure of the triple-barrier labeling scheme introduced in Chapter 3, creating a coherent end-to-end framework: label with triple barriers, train a classifier to predict which barrier is touched, size bets from predicted probabilities, and evaluate risk using the analytical failure-probability formula.

This chapter introduces the Hierarchical Risk Parity (HRP) approach, which addresses three major concerns of quadratic optimizers like Markowitz’s Critical Line Algorithm: instability, concentration, and underperformance. HRP applies graph theory and ML techniques to build a diversified portfolio based on the information in the covariance matrix without requiring its inversion.

Classical mean-variance optimization fails because it amplifies estimation errors in the covariance matrix. Small perturbations in inputs produce wildly different allocations. HRP is stable by design: it uses hierarchical clustering to identify natural groups of assets, then allocates by recursive bisection along the cluster tree rather than by matrix inversion.

The three-stage HRP algorithm is: (1) compute a distance matrix from correlations and build a hierarchical cluster tree; (2) quasi-diagonalize the covariance matrix by reordering assets so that similar assets are adjacent; (3) allocate by recursive bisection, distributing weight inversely proportional to cluster variance. No matrix inversion occurs at any step.

Monte Carlo simulations show that HRP delivers superior out-of-sample performance compared to CLA and inverse-variance portfolios. More importantly, HRP can compute an allocation on a singular or nearly singular covariance matrix — a practical necessity when the number of assets exceeds the number of observations, which is routine in large investment universes.

The HRP methodology extends beyond portfolio optimization. It applies wherever decisions must be made under uncertainty with a nearly singular covariance matrix: capital allocation across portfolio managers, allocation across algorithmic strategies, and bagging/boosting of ML signals from a large random forest.

Financial markets operate in regimes. A model fit on one regime may be catastrophically wrong in another. Detecting when the underlying data-generating process has shifted — a structural break — is therefore a prerequisite for any robust ML deployment. I organize structural break tests into CUSUM tests (which detect deviations from white noise) and explosiveness tests (which detect exponential growth or collapse).

CUSUM tests were introduced in Chapter 2 as event-based samplers; here they are extended to formal hypothesis tests. The Brown-Durbin-Evans method monitors cumulative recursive residuals against confidence bands. The Chu-Stinchcombe-White method applies the same logic to cumulative sums of standardized returns. Both detect sustained drift from a baseline model.

Explosiveness tests are designed to detect bubbles: price processes that grow or collapse exponentially faster than a random walk, a dynamic that is unsustainable and eventually reverses. The supremum augmented Dickey-Fuller (SADF) test fits a right-tail unit-root test on every sub-sample ending at time t and reports the supremum test statistic across all starting points.

SADF applied to E-mini S&P 500 futures correctly identifies explosive episodes around the dot-com bubble and the 2007-2008 crisis. These structural break indicators can be used directly as features in an ML model, providing the algorithm with a signal of regime change that simple price or return features do not carry.

Entropy is a measure of the complexity — and therefore the predictability — of a financial time series. A highly predictable series has low entropy; a series indistinguishable from noise has high entropy close to its theoretical maximum. Tracking how entropy evolves over time gives an ML model a direct measure of market efficiency that is not available from returns alone.

The Lempel-Ziv algorithm estimates entropy by measuring the compression rate of a message: a complex, high-entropy message requires a large dictionary of non-redundant substrings. Applied to encoded financial returns, the LZ compressibility estimate is a non-parametric, model-free entropy estimator that works on short sequences — a practical advantage over plug-in estimators.

Encoding financial series for entropy estimation requires discretizing continuous returns into a finite alphabet. Binary encoding (up/down), quantile encoding (assigning rank buckets), and sigma encoding (multiples of volatility) each make different tradeoffs between information density and robustness. The choice of encoding should be validated empirically; fractionally differentiated series (Chapter 5) are better inputs than raw returns because they retain memory.

Shannon entropy connects to volatility through the Gaussian case: for normally distributed returns, entropy is proportional to log-volatility. This entropy-implied volatility estimate is complementary to GARCH-based estimates and can detect changes in information content that volatility measures miss — particularly when the distribution of returns shifts without changing its variance.

Market microstructure theory provides a rich vocabulary of features that are directly observable from tick data and that capture the informational content of order flow. The first generation of models — trade classification rules like the tick rule, and Roll’s spread model — used only price information. Subsequent generations incorporated volume, then sequential arrival of orders, yielding progressively richer feature sets.

Kyle’s lambda measures price impact: the coefficient linking signed order flow to price change. It captures illiquidity as a risk factor with an associated premium. Hasbrouck’s lambda is a related measure derived from the variance of the pricing error in a structural model. Both are direct inputs to ML feature matrices as measures of market depth and informed trading activity.

The probability of informed trading (PIN) models the arrival rate of informed versus uninformed traders as a structural parameter. The volume-synchronized PIN (VPIN) is a high-frequency, non-parametric estimate of PIN computed on volume bars: it classifies tick volume as buy- or sell-initiated and computes the imbalance relative to total volume. VPIN has been proposed as a real-time early warning indicator of liquidity crises.

Microstructural information is properly defined using signal-processing concepts. Given a features matrix containing VPIN, Kyle’s lambda, cancellation rates, and related quantities, the entropy of the information content can be estimated to determine how much new information a market maker would gain from observing a new tick. This connects Chapters 18 and 19 into a unified information-theoretic view of order flow.