How to Bootstrap Backtest Confidence Bands

Backtest confidence interval dey answer one question: how much of equity curve come from the strategy, and how much come from the particular period of history wey e run on. Bootstrapping a backtest dey build that interval by resampling the return series many times, recalculating the statistic for each resample, then checking the percentiles of the results. Sharpe of 1.4 wey dem measure from one year of daily observations fit get 95 percent interval wide enough to include zero, and the panels below dey show why.

Why one equity curve na one sample

Backtest dey give you one number for each statistic: one Sharpe ratio, one annualized return, one worst drawdown, one hit rate. None of dem na the strategy’s true value. Each one na estimate wey come from finite run of trading days, and another period of the same length for the same strategy for show another result.

The panel below remove the strategy from the question completely. E measure the simplest position possible: holding SPY, one calendar year at a time, based on close-to-close price returns.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2011-12-01'
      AND date <  '2026-01-01'
)
SELECT
    toString(toYear(date))                            AS year,
    count()                                           AS obs_count,
    round(avg(ret) * 252 * 100, 2)                    AS ann_return_pct,
    round(stddevSamp(ret) * sqrt(252) * 100, 2)       AS ann_vol_pct,
    round(avg(ret) / stddevSamp(ret) * sqrt(252), 2)  AS sharpe_ratio
FROM daily
WHERE date >= '2012-01-01'
  AND ret IS NOT NULL
GROUP BY year
ORDER BY year

Each row cover about 250 sessions, close to the standard trading year. The position no change across any of dem. For 2012, the annualized Sharpe measure 1.06; for 2025, e measure 0.88, with 14 years drawn on the chart. Holding a broad index for one year fit change the headline number by more than wetin most readers go classify as noise. Strategy backtest wey get the same length go inherit at least that level of variation. The annualization convention behind the column dey inside our Sharpe ratio guide, while the mechanics for period return dey inside how monthly returns dey measured.

Sharpe backtest estimate dey get how wide band around am?

Standard error of Sharpe estimate dey reduce roughly according to square root of number of observations. Instead make we depend on that formula, measure the spread directly. Divide twenty years of trading sessions into non-overlapping windows wey get fixed length. Calculate annualized Sharpe inside each window, then check how far apart the numbers dey.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2005-12-01'
      AND date <  '2026-01-01'
),
indexed AS
(
    SELECT
        ret,
        row_number() OVER (ORDER BY date) AS i
    FROM daily
    WHERE date >= '2006-01-01'
      AND ret IS NOT NULL
),
blocks AS
(
    SELECT
        w,
        intDiv(i, w)                                            AS blk,
        count()                                                 AS n,
        avg(ret) / nullIf(stddevSamp(ret), 0) * sqrt(252)        AS sharpe
    FROM indexed
    CROSS JOIN (SELECT arrayJoin([21, 63, 126, 252, 504]) AS w) AS ws
    GROUP BY w, blk
    HAVING n = w
)
SELECT
    concat(toString(w), ' sessions')                          AS horizon,
    count()                                                   AS block_count,
    round(quantileDeterministic(0.05)(sharpe, blk), 2)        AS sharpe_p05,
    round(quantileDeterministic(0.50)(sharpe, blk), 2)        AS sharpe_p50,
    round(quantileDeterministic(0.95)(sharpe, blk), 2)        AS sharpe_p95,
    round(quantileDeterministic(0.95)(sharpe, blk)
        - quantileDeterministic(0.05)(sharpe, blk), 2)        AS sharpe_band_width
FROM blocks
WHERE sharpe IS NOT NULL
GROUP BY w
ORDER BY w

For 21 sessions per window, the 5th to 95th percentile of measured Sharpe dey run from -3.37 reach 6.97. Na spread of 10.34 Sharpe points across 238 windows. If each window stretch reach 504 sessions, the spread drop to 1.67 points, but this time na only 8 windows dem measure. Two things dey change together. Estimate dey become more precise as sample size increase, but number of independent samples wey historical data fit provide dey reduce sharply. Na this tension be the whole subject.

How to bootstrap confidence interval for backtest

This procedure short enough to write out complete.

1. Start with the strategy’s realized return series, one figure for each period, N of dem.
2. Pick N returns from that list at random, with replacement. Some fit show two times, while some no go show at all.
3. Recalculate the statistic on the resample.
4. Repeat am several thousand times, and keep every value.
5. Arrange the values wey you store from low to high, then read the 2.5th and 97.5th percentiles for a 95 percent interval.

Set the random number generator seed before step 2, and write the seed beside the published interval. Bootstrap na Monte Carlo estimate. Two runs wey no get seed fit disagree for the last digits. If reviewer no fit run your interval again, dem no get way to check am. Na the same discipline wey reproducible backtest setup describe.

Mean return and Sharpe pass through step 2 cleanly because both dey treat the return list as a set. Maximum drawdown no be like that. Drawdown dey read the path in order. Resample wey rearrange returns fit produce worst drawdown wey no ordering of the real trade sequence ever produce. E still make sense to bootstrap am, as long as you label the output correctly: na the distribution of drawdown across reshuffled histories, no be forecast of the next one. Definition dey for maximum drawdown.

Why IID bootstrap wrong for market returns

Step 2 assume say every return independent and identically distributed, wey be the IID assumption. Daily returns dey break this assumption in a way wey dey change interval width. Signed returns get only small one-day memory. But their magnitudes dey cluster: big moves dey come near big moves, while quiet days dey happen in runs.

WITH daily AS
(
    SELECT
        ticker,
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (PARTITION BY ticker ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker IN ('SPY', 'MSFT', 'KO', 'XOM', 'JNJ', 'PG')
      AND date >= '2015-11-01'
      AND date <  '2026-01-01'
),
lagged AS
(
    SELECT
        ticker,
        date,
        ret,
        lagInFrame(ret, 1) OVER (PARTITION BY ticker ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS ret_prev
    FROM daily
    WHERE ret IS NOT NULL
      AND abs(ret) < 0.35
)
SELECT
    ticker,
    count()                                AS obs_count,
    round(corr(ret, ret_prev), 3)          AS return_autocorr,
    round(corr(abs(ret), abs(ret_prev)), 3) AS abs_return_autocorr
FROM lagged
WHERE date >= '2016-01-01'
  AND ret_prev IS NOT NULL
GROUP BY ticker
ORDER BY abs_return_autocorr DESC

For SPY, lag-one autocorrelation of absolute daily returns measure 0.366 across 2514 sessions, compared with -0.133 for signed returns on the same days. Compare the two bars for any name for the chart. If you shuffle a return series, you destroy that clustering. Then IID bootstrap run on this data go report an interval wey narrow pass wetin the sample support. The error dey go for the worst direction: e make the strategy look better than e really be.

The moving block bootstrap, and how to choose block length

The fix na to resample returns wey dey follow each other, instead of single returns. Choose block length L. Randomly pick blocks of L consecutive returns with replacement. Join dem together until the synthetic series reach N length. Dependence inside each block remain intact because the returns still dey their original order. Na only the points wey join two blocks be artificial.

Block length dey balance two errors, and no setting fit avoid both. Short blocks behave like IID bootstrap and make the interval look too narrow. Na bias be this. Long blocks preserve more dependence, but dem leave fewer different blocks to pick from. Each resample then repeat big parts of the same history, and the interval itself become noisy. Na variance be this. Published rules of thumb dey scale block length with N to the one-third power. Use dem as starting points.

One cheaper diagnostic na to check how variance dey scale with horizon. If returns independent, variance of a k-day return sum equal k times the one-day variance. The ratio of both stay at 1.

WITH daily AS
(
    SELECT
        date,
        toFloat64(close) / nullIf(lagInFrame(toFloat64(close), 1)
            OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 0) - 1 AS ret
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2005-12-01'
      AND date <  '2026-01-01'
),
indexed AS
(
    SELECT
        ret,
        row_number() OVER (ORDER BY date) AS i
    FROM daily
    WHERE date >= '2006-01-01'
      AND ret IS NOT NULL
),
base AS
(
    SELECT varSamp(ret) AS var_1d FROM indexed
),
blocks AS
(
    SELECT
        k,
        intDiv(i, k)  AS blk,
        count()       AS n,
        sum(ret)      AS block_ret
    FROM indexed
    CROSS JOIN (SELECT arrayJoin([2, 3, 5, 10, 21, 42, 63]) AS k) AS ks
    GROUP BY k, blk
    HAVING n = k
)
SELECT
    concat(toString(k), ' sessions')                          AS block_length,
    count()                                                   AS block_count,
    round(varSamp(block_ret) / (k * any(var_1d)), 3)          AS variance_ratio
FROM blocks
CROSS JOIN base
GROUP BY k
ORDER BY k

At 2 sessions the ratio measure 0.844. At 63 sessions e measure 0.584. The calculation use 78 non-overlapping blocks. Values near 1 mean the sum dey scale the way independent draws for that horizon go scale am. Values away from 1 show the horizons where dependence still dey matter. Na that range block length need cover. Choose the shortest block wey clear am, then check how low the block count don fall.

Bucket resampling and count wey dey beside each interval

Resampling inside buckets, based on volatility regime or calendar month, dey preserve the condition wey make the question interesting. If claim be say strategy dey earn its Sharpe during high-volatility regimes, interval wey use only high-volatility days na the one wey fit test that claim. The cost na arithmetic. If you split 250 observations into four buckets, each bucket go get roughly 60 observations. Bootstrap wey use 60 observations go give interval wey about two times wider than the full-sample version.

So, report the bucket count beside every interval every time. The block_count column for the panels above dey make this habit clear: 5th percentile wey come from nine windows na different thing from one wey come from two hundred windows, even when both print to two decimal places.

Wetin bootstrap no fit fix

Bootstrap dey quantify one thing: sampling noise for statistic wey you calculate from the data wey you get. E no talk whether that data ever dey achievable.

Backtest wey look-ahead bias don contaminate go produce return series wey nobody fit trade, and bootstrap of that series go return tight, confident band around fiction. Universe wey you build from today index members get survivorship bias, and every resample of that universe go inherit am. Another gap na selection effect: run two hundred variants, keep the best one, and its bootstrap interval go describe sampling noise for that one variant, while e ignore the one hundred and ninety-nine draws wey enter the selection. Honest bands dey useful. But dem no replace out-of-sample data.

FAQ

Bootstrap confidence interval dey tell you wetin about backtest?

E dey show the range of values wey the statistic fit reasonably get if dem sample the same process again across the same number of periods. If 95 percent interval contain zero, e mean say the sample too short to separate the strategy from no edge at all.

How many bootstrap resamples dey enough?

For 95 percent interval, few thousand resamples usually dey stable, and 10,000 na common default wey no cost much. Deeper tail quantiles need more: the 1st percentile from 1,000 draws dey come from around ten values.

Which block length moving block bootstrap suppose use?

No single length dey correct for every case. Rules of thumb dey scale am with sample size to the one-third power. One practical check na the horizon where variance stop scaling linearly. The variance ratio panel above dey measure this. Publish the length wey you use together with the interval.

You fit bootstrap maximum drawdown?

Yes, but ordering get caveat. Drawdown depend on the sequence of returns, so resample wey shuffle returns go report drawdown for reordered history. Block bootstrap dey keep short runs together, and e better for path statistics.

Bootstrap dey correct overfitting?

No. E dey measure sampling noise inside one return series. Look-ahead bias and the selection effect from testing many variants dey outside wetin e fit see.

Every panel for here carry the SQL wey produce am. Change the ticker or the window length, then run am yourself for the Strasmore terminal.