Bootstrapping Backtest Confidence Bands

A backtest confidence interval answers one question: how much of an equity curve is the strategy, and how much is the particular slice of history it ran on. Bootstrapping a backtest builds that interval by resampling the return series many times over, recomputing the statistic on each resample, and reading the percentiles of what comes back. A Sharpe of 1.4 measured over a single year of daily observations carries a 95 percent interval wide enough to contain zero, and the panels below measure why.

Why one equity curve is one sample

A backtest hands you one number per statistic: one Sharpe ratio, one annualized return, one worst drawdown, one hit rate. None of those is the strategy's true value. Each is an estimate built from a finite run of trading days, and a different stretch of the same length would have printed something else.

The panel below removes the strategy from the question entirely. It measures the simplest possible position, holding SPY, one calendar year at a time, on close-to-close price returns.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2011-12-01'
      AND date <  '2026-01-01'
)
SELECT
    toString(toYear(date))                            AS year,
    count()                                           AS obs_count,
    round(avg(ret) * 252 * 100, 2)                    AS ann_return_pct,
    round(stddevSamp(ret) * sqrt(252) * 100, 2)       AS ann_vol_pct,
    round(avg(ret) / stddevSamp(ret) * sqrt(252), 2)  AS sharpe_ratio
FROM daily
WHERE date >= '2012-01-01'
  AND ret IS NOT NULL
GROUP BY year
ORDER BY year

Each row covers about 250 sessions, close to the standard trading year. Nothing about the position changed across any of them. In 2012 the annualized Sharpe measured 1.06; in 2025 it measured 0.88, with 14 years drawn on the chart. One year of holding a broad index moves the headline number by more than most readers would file under noise, and a strategy backtest of the same length inherits at least that much. The annualization convention behind the column sits in our Sharpe ratio guide, and the period-return mechanics in how monthly returns are measured.

How wide is the band around a backtest Sharpe?

The standard error of a Sharpe estimate falls roughly with the square root of the number of observations. Rather than lean on that formula, measure the spread directly. Cut twenty years of sessions into non-overlapping windows of a fixed length, compute the annualized Sharpe inside each window, and look at how far apart those numbers land.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2005-12-01'
      AND date <  '2026-01-01'
),
indexed AS
(
    SELECT
        ret,
        row_number() OVER (ORDER BY date) AS i
    FROM daily
    WHERE date >= '2006-01-01'
      AND ret IS NOT NULL
),
blocks AS
(
    SELECT
        w,
        intDiv(i, w)                                            AS blk,
        count()                                                 AS n,
        avg(ret) / nullIf(stddevSamp(ret), 0) * sqrt(252)        AS sharpe
    FROM indexed
    CROSS JOIN (SELECT arrayJoin([21, 63, 126, 252, 504]) AS w) AS ws
    GROUP BY w, blk
    HAVING n = w
)
SELECT
    concat(toString(w), ' sessions')                          AS horizon,
    count()                                                   AS block_count,
    round(quantileDeterministic(0.05)(sharpe, blk), 2)        AS sharpe_p05,
    round(quantileDeterministic(0.50)(sharpe, blk), 2)        AS sharpe_p50,
    round(quantileDeterministic(0.95)(sharpe, blk), 2)        AS sharpe_p95,
    round(quantileDeterministic(0.95)(sharpe, blk)
        - quantileDeterministic(0.05)(sharpe, blk), 2)        AS sharpe_band_width
FROM blocks
WHERE sharpe IS NOT NULL
GROUP BY w
ORDER BY w

At 21 sessions per window, the 5th to 95th percentile of measured Sharpe runs from -3.37 to 6.97, a spread of 10.34 Sharpe points across 238 windows. Stretch each window to 504 sessions and the spread falls to 1.67 points, now measured over only 8 windows. Two things move at once. The estimate gets more precise with sample size, and the count of independent samples history can supply collapses. That tension is the entire subject.

How to bootstrap a backtest confidence interval

The procedure is short enough to write out in full.

1. Start with the strategy's realized return series, one figure per period, N of them.
2. Draw N returns from that list at random, with replacement. Some appear twice, some not at all.
3. Recompute the statistic on the resample.
4. Repeat several thousand times, keeping every value.
5. Sort the stored values and read the 2.5th and 97.5th percentiles for a 95 percent interval.

Seed the random number generator before step 2, and record the seed next to the published interval. A bootstrap is a Monte Carlo estimate: two unseeded runs disagree in the last digits, and a reviewer who cannot rerun your interval has no way to check it. That is the same discipline described in a reproducible backtest setup.

Mean return and Sharpe pass through step 2 cleanly, since both read the return list as a set. Maximum drawdown does not. Drawdown reads the path in order. A resample that reorders returns yields a worst drawdown that no ordering of the real trade sequence produced. Bootstrapping it is still worth doing, provided the output is labeled for what it is: the distribution of drawdown across reshuffled histories, not a forecast of the next one. The definition is in maximum drawdown.

Why the IID bootstrap is wrong for market returns

Step 2 assumes every return is independent and identically distributed, the IID assumption. Daily returns break it in a way that changes interval width. Signed returns carry only faint one-day memory. Their magnitudes cluster: large moves sit next to large moves, and quiet days arrive in runs.

WITH daily AS
(
    SELECT
        ticker,
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (PARTITION BY ticker ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker IN ('SPY', 'MSFT', 'KO', 'XOM', 'JNJ', 'PG')
      AND date >= '2015-11-01'
      AND date <  '2026-01-01'
),
lagged AS
(
    SELECT
        ticker,
        date,
        ret,
        lagInFrame(ret, 1) OVER (PARTITION BY ticker ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS ret_prev
    FROM daily
    WHERE ret IS NOT NULL
      AND abs(ret) < 0.35
)
SELECT
    ticker,
    count()                                AS obs_count,
    round(corr(ret, ret_prev), 3)          AS return_autocorr,
    round(corr(abs(ret), abs(ret_prev)), 3) AS abs_return_autocorr
FROM lagged
WHERE date >= '2016-01-01'
  AND ret_prev IS NOT NULL
GROUP BY ticker
ORDER BY abs_return_autocorr DESC

For SPY, the lag-one autocorrelation of absolute daily returns measured 0.366 across 2514 sessions, against -0.133 for the signed returns on the same days. Compare the two bars on any name in the chart. Shuffling a return series destroys that clustering, and an IID bootstrap run on this data reports a narrower interval than the sample supports. The error runs in the least helpful direction: it flatters the strategy.

The moving block bootstrap, and picking a block length

The repair is to resample contiguous blocks instead of single returns. Pick a block length L, draw blocks of L consecutive returns at random with replacement, and glue them end to end until the synthetic series is N long. Dependence inside a block survives intact: those returns stay in their original order. Only the joins between blocks are artificial.

Block length trades two errors against each other, and no setting escapes both. Short blocks behave like the IID bootstrap and understate the interval, which is bias. Long blocks preserve more dependence but leave fewer distinct blocks to draw from, so each resample repeats large chunks of the same history and the interval itself gets noisy, which is variance. Published rules of thumb scale the length with N to the one-third power. Treat them as starting points.

A cheaper diagnostic is to check how variance scales with horizon. Under independence, the variance of a k-day return sum equals k times the one-day variance, and the ratio of the two sits at 1.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2005-12-01'
      AND date <  '2026-01-01'
),
indexed AS
(
    SELECT
        ret,
        row_number() OVER (ORDER BY date) AS i
    FROM daily
    WHERE date >= '2006-01-01'
      AND ret IS NOT NULL
),
base AS
(
    SELECT varSamp(ret) AS var_1d FROM indexed
),
blocks AS
(
    SELECT
        k,
        intDiv(i, k)  AS blk,
        count()       AS n,
        sum(ret)      AS block_ret
    FROM indexed
    CROSS JOIN (SELECT arrayJoin([2, 3, 5, 10, 21, 42, 63]) AS k) AS ks
    GROUP BY k, blk
    HAVING n = k
)
SELECT
    concat(toString(k), ' sessions')                          AS block_length,
    count()                                                   AS block_count,
    round(varSamp(block_ret) / (k * any(var_1d)), 3)          AS variance_ratio
FROM blocks
CROSS JOIN base
GROUP BY k
ORDER BY k

At 2 sessions the ratio measured 0.844; at 63 sessions it measured 0.584, computed over 78 non-overlapping blocks. Values near 1 mean the sum scales the way independent draws would at that horizon. Values away from 1 mark the horizons where dependence is still doing work, and that is the range a block length has to cover. Pick the shortest block that clears it, then look at what the block count has fallen to.

Bucketed resampling and the count beside every interval

Resampling within buckets, by volatility regime or by calendar month, keeps the conditioning that made the question interesting. If the claim is that a strategy earns its Sharpe in high-volatility regimes, an interval built by drawing only from high-volatility days is the one that tests the claim. The cost is arithmetic. Splitting 250 observations into four buckets leaves roughly 60 per bucket, and a 60-observation bootstrap gives an interval about twice as wide as the full-sample version.

So report the bucket count beside every interval, every time. The block_count column in the panels above is that habit made visible: a 5th percentile read off nine windows is a different object from one read off two hundred, even when both print to two decimals.

What a bootstrap cannot fix

A bootstrap quantifies one thing, the sampling noise in a statistic computed from the data you have. It is silent on whether that data was ever achievable.

A backtest contaminated with look-ahead bias produces a return series that was never tradable, and a bootstrap of it hands back a tight, confident band around a fiction. A universe assembled from today's index members carries survivorship bias, and every resample of that universe inherits it. A third gap is the selection effect: run 200 variants, keep the best, and its bootstrap interval describes the sampling noise of that one variant while ignoring the 199 draws that went into picking it. Honest bands are useful. They are no substitute for out-of-sample data.

FAQ

What does a bootstrap confidence interval tell you about a backtest?

It gives the range of values the statistic would plausibly take if the same process were sampled again over the same number of periods. A 95 percent interval that contains zero means the sample is too short to separate the strategy from no edge at all.

How many bootstrap resamples are enough?

For a 95 percent interval, a few thousand resamples is usually stable, and 10,000 is a common default that costs little. Deeper tail quantiles need more: the 1st percentile of 1,000 draws is read off about ten values.

What block length should a moving block bootstrap use?

There is no universally correct length. Rules of thumb scale it with sample size to the one-third power, and a practical check is the horizon at which variance stops scaling linearly, which the variance ratio panel above measures. Publish the length you used alongside the interval.

Can you bootstrap a maximum drawdown?

Yes, with an ordering caveat. Drawdown depends on the sequence of returns, so a resample built by shuffling reports the drawdown of a reordered history. A block bootstrap keeps short runs intact and is the better tool for path statistics.

Does a bootstrap correct for overfitting?

No. It measures sampling noise inside one return series. Look-ahead bias and the selection effect from testing many variants both sit outside what it can see.

Every panel here carries the SQL that produced it. Change the ticker or the window length and run it yourself on the Strasmore terminal.