How backtesting actually works

The one-sentence version

A backtest converts a vague idea — “buy stocks that have been going up” — into an unambiguous sequence of simulated decisions, then reports what that sequence produced.

The conversion is the whole point. “Going up” has to become which stocks, measured over what window, rebalanced how often, holding how many positions, at what cost. Most of the value of running a backtest comes from being forced to answer those questions before you ever see a result.

The loop, step by step

Every backtest engine, from a spreadsheet to an institutional platform, runs some version of this loop:

Step	What happens	Where it goes wrong
1. Define the universe	Pick the set of securities the strategy is allowed to hold.	If the list is today's survivors, the test is already biased. See survivorship bias.
2. Compute a signal	At each decision date, score every candidate on some rule — trailing return, a moving-average cross, a valuation ratio.	Using any data published after that date is look-ahead bias.
3. Build the portfolio	Turn scores into weights: top N names, equal weight, rank weight, whatever the rule says.	Weighting choices quietly drive most of the result, and are easy to tune after the fact.
4. Rebalance on a schedule	Move from yesterday's portfolio to today's target on the chosen cadence.	Daily rebalancing looks great on paper and is often uninvestable after costs.
5. Charge costs	Subtract commissions, spread, and slippage on every share traded.	A zero-cost assumption flatters high-turnover strategies enormously.
6. Compound forward	Apply the period's returns to the portfolio and record the new equity value.	Ignoring delisted positions silently deletes your worst outcomes.

Run that loop across every date in the sample and you get an equity curve: the simulated value of the account over time. Every metric below is just a different summary of that one curve.

What comes out the other end

A single equity curve gets compressed into a handful of numbers. Each answers a different question, and reading only one of them is how people talk themselves into bad strategies.

Metric	Question it answers	What it hides
CAGR	How fast did it compound?	Says nothing about the ride. A 20% CAGR through an 80% drawdown is unholdable in practice.
Sharpe ratio	How much return per unit of volatility? (Sharpe, 1966)	Treats upside and downside volatility identically, and assumes returns are reasonably well-behaved.
Max drawdown	What was the worst peak-to-trough loss?	A single historical worst case. The next one can always be deeper.
Consistency	What share of periods were positive?	Can look excellent right up until one catastrophic period.
Volatility	How much did returns bounce around?	Low historical volatility is not a promise of low future volatility.

StratPick's composite score exists precisely because no single one of these is trustworthy alone.

Six ways a backtest misleads you

In rough order of how often they ruin an otherwise sensible test:

• 1. Overfitting. Tune enough knobs and you will eventually fit the noise in your sample rather than any durable pattern. This is the big one. Work by Bailey and López de Prado on backtest overfitting, and by Harvey, Liu and Zhu on multiple testing in asset pricing, both make the same uncomfortable point: the more strategies you search, the higher the bar a result must clear to mean anything — and almost nobody raises the bar.
• 2. Survivorship bias. Testing on the names that exist today quietly removes every company that failed. The dead ones are exactly the outcomes your strategy needed to survive.
• 3. Look-ahead bias. Using anything in a decision that was not knowable at the time: a restated earnings figure, a later index membership list, a full-sample average. It inflates results and leaves no visible trace in the output.
• 4. Unrealistic costs. Spread, commission, market impact, and borrow costs. A strategy turning over its book weekly needs to clear a far higher hurdle than the same idea rebalanced annually.
• 5. Taxes. Most backtests, StratPick's included, report pre-tax results. High-turnover strategies realize mostly short-term gains, taxed as ordinary income in a US taxable account. The after-tax curve can look very different.
• 6. Regime dependence. Fifteen years of data can contain only two or three genuinely distinct market regimes. A strategy that only ever saw falling rates has not been tested against rising ones.

In-sample, out-of-sample, and the honesty problem

In-sample is the data you looked at while building the strategy. Out-of-sample is data the strategy never saw during design. Only the second kind is evidence.

The trap is subtle: out-of-sample data stops being out-of-sample the moment you use it to make another decision. Check your held-out period, dislike the result, adjust one parameter, and check again — you have just folded that period into your in-sample set. Do it a few times and you have no clean test left, only a more elaborate way of fitting the same noise.

The only fully honest out-of-sample test is forward time: data that did not exist when the strategy was fixed. It is slow, unglamorous, and the only version that cannot be quietly re-run.

Reading someone else's backtest

A short checklist worth applying to any impressive equity curve, including the ones on this site:

• What was the universe, and did it include companies that later failed?
• How many variants were tried before this one was shown to you?
• What costs were charged, and what is the annual turnover?
• Is the result pre-tax, and how much of the gain is short-term?
• How many independent market regimes does the sample actually contain?
• Is there any genuinely forward, post-publication track record?
• How different is the result from simply holding a broad index over the same window?

That last one matters more than it sounds. A great many strategies that look clever are just leveraged beta wearing a costume.

Frequently asked questions

What is backtesting?

Replaying a fixed set of trading or allocation rules over historical market data to see what those rules would have produced. It turns a vague idea into a specific sequence of simulated trades and a measurable equity curve.

Is a good backtest a reliable predictor of future returns?

No. A backtest describes one path history actually took, not the distribution of paths it could have taken. Strong results can come from genuine edge, from luck, or from having tried many variations until one fit. Backtests are far better at rejecting bad ideas than confirming good ones.

What is look-ahead bias?

Using information in a simulated decision that would not have been available at the time. Restated financials, later-published index membership, and full-period averages are the usual culprits. It inflates results invisibly.

How many strategies can I test before the results stop meaning anything?

Sooner than most people expect. Every extra variation raises the odds that your best result is noise. The multiple-testing literature in finance argues that standard significance thresholds are much too lenient once the number of strategies searched is taken into account.

What is the difference between in-sample and out-of-sample testing?

In-sample is the data used while designing the strategy; out-of-sample is data it never saw. Only out-of-sample results carry evidential weight, and they stop counting as out-of-sample the moment you use them to make another tuning decision.