Training · QF-1 · Lesson 03 of 05
Backtesting honestly
The rule
A backtest is an argument, not a proof. The failure mode is a strategy that fits history and never repeats. The work is making the test harder than the strategy, so the surprise arrives on paper rather than in the book.
What a backtest is
A backtest simulates how a strategy would have performed on historical data. It is necessary but dangerous: it is easy to build a strategy that fits the past perfectly and fails immediately in live trading.
The main biases
Most bad backtests fail for one of a small set of reasons.
- ·Look-ahead bias: using information not available at the time.
- ·Survivorship bias: excluding companies that failed.
- ·Overfitting: tuning parameters until noise looks like signal.
- ·Ignoring costs: assuming trades execute at mid-price with no impact.
- ·Data snooping: testing many ideas and reporting only the winner.
Multiple testing
If you test hundreds of variations, some will look great by chance. Adjust significance thresholds for the number of trials, or deflate the reported Sharpe ratio. Researchers increasingly demand a t-statistic above 3 rather than 2 for new factors.
- Strategies tested
- 200
- Significance level
- 5%
- Expected false positives
- 10
- Required t-stat (stricter)
- ≥ 3.0
Out-of-sample testing
Hold back part of the data, ideally a recent period, and test only once after finalising the strategy. Walk-forward analysis repeatedly fits on a window and tests on the next. Paper trading before committing capital is the final out-of-sample test.
Reporting honestly
A credible backtest report includes returns after costs, volatility, Sharpe, maximum drawdown, turnover, capacity estimates, factor exposures and the number of variations tried. If live results diverge from the backtest, investigate before adding capital.
Hold onto this
- ·Split the sample, and never tune on the part you report as evidence.
- ·Model transaction costs, borrow, and the market impact your own order creates.
- ·Report the drawdown and the number of trials run; a strategy found in 500 tries is not significant.
Progress saved on this device
Education, not financial advice. Figures may be simplified for illustration.