Almost every investment strategy you see advertised comes with a chart: a line that rises steadily for years and leaves the market behind. That line is usually a backtest. It can be an honest tool or a very convincing illusion, and from the outside the two look exactly the same. This guide explains what a backtest is and the five traps that most often turn one into fiction.
What a backtest is
A backtest applies the rules of a strategy to past prices, as if they had been followed at the time, and records what would have happened: which trades would have opened, when they would have closed and with what result.
It is useful for several things:
- Understanding a method: how often it trades, how long it holds, what its typical losses look like.
- Discarding bad ideas cheaply: if a rule did not work in the past, there is little reason to expect it to work now.
- Knowing what to expect: how deep the bad periods were and how long they lasted.
But a backtest describes what the rules would have done, not what anyone did. Between the two there is room for many mistakes, almost all of them in the same direction: making the past look better.
Trap 1: look-ahead bias, using information you did not have
A backtest must only use the information that was available at the moment of each decision. When it uses something that was only known later, the result is inflated, sometimes enormously.
Some ways it slips in:
- Results published later. A rule that buys "companies with good annual results" on 31 December, when those results were only published in February.
- Revised data. Using today's version of a figure that was corrected months after it was first published.
- Prices in the wrong order. Assuming that, on the day of purchase, you sold at a high that was reached before you bought. On a daily chart you only see the high, the low, the open and the close, not the order in which they happened.
- Index membership. Choosing stocks from today's list of an index to test what you would have done ten years ago.
The last two are especially sneaky because they do not look like "knowing the future"; they look like ordinary data. A good backtest decides at the close of a day only with what was known at that close, and executes at prices that could really have been obtained afterwards.
Trap 2: survivorship bias, forgetting the losers
If you test a strategy only on the companies that exist today, you are leaving out all those that went bankrupt, were taken over cheaply or were removed from the index for doing badly. Those are precisely the stocks a strategy would have suffered with.
The effect is that the past looks friendlier than it was: the sample has been filtered, after the fact, in favour of the survivors.
The clean solution is to test on the companies that were really in each index on each date, including the ones that later disappeared. That data is harder to obtain, and many backtests, including our own historical simulations, are built on the current members of each index. That is why we read them as a ceiling, a description of how the method behaves, never as a promise of what it will give.
Trap 3: overfitting, a rule tailored to the past
Imagine trying a hundred variations of a rule: an RSI threshold of 25, 26, 27... a stop at 7%, 8%, 9%... Out of a hundred, one will have done brilliantly by pure chance. If you keep that one and present its result, you are not presenting a strategy; you are presenting the luckiest attempt.
This is called overfitting: adjusting the rule so closely to past data that it learns the noise as well as the pattern. Its typical symptom is a spectacular backtest that disappoints as soon as it meets new data.
How to defend against it:
- Few parameters, decided for a reason. Each additional adjustable number is another opportunity to fit the noise.
- Out-of-sample testing. Build the rule on one period and measure it on a later period it has never seen. A result measured that way is worth much more than one on the data used to build it.
- Robustness. If the result collapses when a parameter is moved slightly, it was probably chance.
- Register the rule before the test. Writing down the rule and the date it will be judged, before seeing the data, removes the temptation to keep adjusting.
Trap 4: forgetting costs
A backtest that buys and sells at the closing price, with no commissions, is describing a world that does not exist. In reality every trade pays:
- Commissions to the broker.
- The spread, the difference between the buying and selling price.
- Slippage: the price at which an order is filled is often slightly worse than the one on the screen, especially in a fast market or a small stock.
- Currency exchange, if you buy shares in another currency.
- Taxes on gains and dividends, which depend on each investor.
Costs weigh more the more often you trade and the smaller each target is. A strategy that aims for a few per cent per trade can see a large part of its edge disappear in costs. That is why it is worth checking what your broker charges before following any short-term method; we discuss it in Mean Reversion vs Trend Following.
Trap 5: drawing conclusions from too few cases
Ten trades are an anecdote. With small samples, chance dominates, and a result that looks excellent can appear without any real advantage behind it.
Three questions help:
- How many trades is the result based on?
- How many independent periods? A thousand trades all made in the same good year are not a thousand independent tests: they share the same market.
- Does it hold in different conditions? Rising markets, falling markets, sideways markets.
We go deeper into this in How to Read a Success Rate Without Fooling Yourself.
The best test: the published record
All of the above is about making simulations more honest. But the strongest evidence is not a simulation: it is a record of what was actually published, on the day, with the levels that were given, and what happened afterwards, with nothing deleted.
A record cannot use information from the future, cannot forget its losers and cannot be re-tuned after the fact, as long as nothing is edited once it is written.
What we do about it
We try to apply these same rules to ourselves:
- The public scoreboard lists every published signal, losers and signals whose order was never filled included, not a selection of winners.
- The quality of our Conviction Score is measured out of sample: on a later period that the model did not see while it was being built.
- The rule our public test measures was registered before its data, with a fixed decision date, so it cannot be adjusted to fit the results.
- Our historical simulations decide at the close and execute afterwards, at prices that could really have been obtained, and every return we publish is before tax.
- Each signal publishes its exit rules before it opens, so the plan cannot be rewritten afterwards.
None of this makes a result good; it makes it honest, which is what lets you judge it. The details are on the methodology page.
A checklist for any backtest
- Could the rule have known something it did not know at the time?
- Does it include the companies that disappeared?
- How many variations were tried before choosing this one?
- Was it measured on data it had never seen?
- Does it include commissions, spread, slippage and taxes?
- How many trades, and across how many different market periods?
- Is there a published record to compare it with?
If a backtest cannot answer these questions, treat it as advertising, not evidence.
In short
- A backtest describes what rules would have done, not what anyone did.
- Look-ahead bias and survivorship make the past look better than it was.
- Trying many variations and keeping the best one produces an illusion: measure out of sample.
- Costs and taxes can eat a large part of an edge, especially with small targets.
- The strongest evidence is a published record with nothing deleted.
This content is for educational purposes only and does not constitute financial advice. Past performance does not guarantee future results.