When Equal Weight Beats Optimization

Equal weight dey beat optimized position sizing anytime sample short too much to pin down expected returns. Short samples na the normal case, no be exception.

Optimizer na correct arithmetic wey dem apply to input wey nobody dey measure well. 1/N portfolio, wey use equal weight, no dey estimate anything at all. Na why e still hold up when the estimates bad.

Equal weight really optimization beat am?

Under one particular condition, yes. Make we take the terms one after another. Equal weight, wey dem write as 1/N, dey divide capital equally across N positions and e no need any forecast. Mean variance optimization dey find the weights wey get the best expected return for each unit of estimated variance. Kelly criterion dey find the weights wey maximize long-run compound growth. For uncorrelated assets, the answer reduce to weight wey dey proportional to each asset expected excess return divided by its variance. Our Kelly criterion position sizing guide explain this formula well.

Both optimizers need the same input: expected return for each asset. Na this quantity sample dey estimate worst. Volatility dey come from short window with reasonable precision. Mean no dey behave like that. The gap between both estimates big enough to show for any price history. The panel below use six widely held names. E measure each calendar year from 2015 through 2024 separately. Then e show the gap between the highest and lowest yearly estimate. E do this twice for each name: first for annualized average daily return, then for annualized volatility.

WITH prices AS
(
    SELECT
        ticker,
        date,
        toFloat64(max(close)) AS c
    FROM global_markets.stocks_daily_aggs
    WHERE ticker IN ('AAPL', 'MSFT', 'NVDA', 'SPY', 'KO', 'JNJ')
      AND date >= '2015-01-01'
      AND date <  '2025-01-01'
    GROUP BY ticker, date
),
rets AS
(
    SELECT
        ticker,
        date,
        c / lagInFrame(c, 1) OVER (PARTITION BY ticker ORDER BY date ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) - 1 AS ret
    FROM prices
),
yearly AS
(
    SELECT
        ticker,
        toYear(date)                     AS yr,
        avg(ret) * 252 * 100             AS mean_pct,
        stddevPop(ret) * sqrt(252) * 100 AS vol_pct
    FROM rets
    WHERE isFinite(ret)
    GROUP BY ticker, yr
    HAVING count() >= 200
)
SELECT
    ticker,
    round(max(mean_pct) - min(mean_pct), 1) AS mean_estimate_range_pct,
    round(max(vol_pct) - min(vol_pct), 1)   AS vol_estimate_range_pct
FROM yearly
GROUP BY ticker
ORDER BY mean_estimate_range_pct DESC

For NVDA the yearly mean estimate cover 184.7 percentage points across those ten years. For the same name and same years, volatility estimate get range of 28.6 points. The steadiest name for the panel, KO, still see its yearly mean estimate move through 23.9 points. Any method wey give last year's average return to an optimizer as this year's expected return dey feed am a number wey fit move that much.

How long sample you need to estimate expected return?

Standard error of estimated mean return na the asset volatility divided by square root of sample length for years. Na years matter, no be number of observations. If you sample the same twelve months every hour instead of every day, e no add anything. Put asset wey get 20 percent volatility inside that formula, and one year history gives standard error of 20 percentage points per year. Four years go cut am to 10. To bring am down to one point — the precision wey you go need before ranking two assets whose true means differ by two points — you need four centuries of data.

Volatility dey different. Its precision improves with number of observations, no be calendar span. So, few months of daily data already fit put am within a few points. The panel below divide twenty years of daily returns from one index fund into non-overlapping blocks of fixed length. E calculate both estimates inside every block, then show how far the estimates scatter from one block to another.

WITH prices AS
(
    SELECT
        date,
        toFloat64(max(close)) AS c
    FROM global_markets.stocks_daily_aggs
    WHERE ticker = 'SPY'
      AND date >= '2005-01-01'
      AND date <  '2025-01-01'
    GROUP BY date
),
rets AS
(
    SELECT
        date,
        c / lagInFrame(c, 1) OVER (ORDER BY date ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) - 1 AS ret
    FROM prices
),
numbered AS
(
    SELECT
        ret,
        row_number() OVER (ORDER BY date ASC) AS i
    FROM rets
    WHERE isFinite(ret)
),
sweep AS
(
    SELECT
        arrayJoin([21, 63, 126, 252, 504]) AS n,
        i,
        ret
    FROM numbered
),
blocks AS
(
    SELECT
        n,
        intDiv(i - 1, n)                 AS blk,
        avg(ret) * 252 * 100             AS mean_pct,
        stddevPop(ret) * sqrt(252) * 100 AS vol_pct
    FROM sweep
    GROUP BY n, blk
    HAVING count() = n
)
SELECT
    concat(toString(n), ' sessions') AS sample_length,
    round(stddevPop(mean_pct), 1)    AS mean_estimate_spread_pct,
    round(stddevPop(vol_pct), 1)     AS vol_estimate_spread_pct,
    count()                          AS window_count
FROM blocks
GROUP BY n
ORDER BY n ASC

For blocks of 21 sessions, the mean estimate scatter across 53.7 percentage points, compared with 10.9 points for the volatility estimate from those same blocks. If you extend the block to 504 sessions, the mean scatter narrow to 10.6 points. E dey narrow according to a square root: every time you halve the error, you need four times the data.

Seeded simulation wey you fit run by yourself

None of this prove say estimation error big enough to give the win to 1/N. This one prove am. The recipe need Python install only: no data files, no downloads and no third-party packages. E draw its own returns from parameters wey you choose, so true answer dey known and you fit score both allocations against am.

1. Import two modules, import random and import statistics, plus from math import log. Set the seed for the next line with random.seed(20260816).
2. Define five assets with true annual expected returns of 5, 6, 7, 8 and 9 percent. Each one get true annual volatility of 20 percent, and dem no correlate with one another. Convert am to daily step with mu_d = mu_a / 252 and sd_d = sd_a / (252 ** 0.5).
3. Choose sample length T from the sweep [60, 125, 250, 500, 1000, 2500, 5000].
4. Draw T daily returns for each asset with random.gauss(mu_d, sd_d). That draw na the complete history wey the manager get to see.
5. Estimate each asset mean with statistics.mean and volatility with statistics.stdev. Na only these two estimates the optimizer know.
6. Build the equal weight portfolio by giving each of the five assets 0.2. E no use any of the estimates.
7. Build the optimized portfolio by setting each raw weight to mu_hat / (sd_hat ** 2). Then divide every weight by the sum of the absolute raw weights. This put both portfolios for the same gross exposure.
8. Draw fresh out-of-sample path of 2520 daily returns for each asset from the true parameters. Neither allocation don see am.
9. Score both portfolios on that path with the average of log(1 + r), where r na the weighted sum of the five asset returns for the day.
10. Repeat steps 4 through 9 for 2000 independent trials at each T. Record the share of trials where equal weight beats the optimized weights.

Read the sweep from short T to long T. For the short end, equal weight win clear majority of the trials. The share dey fall steadily as T grow, and for the long end optimized weights win most of dem. Somewhere for the middle, both cross. The crossover point dey change with the seed and with the spread between the true means. But the direction no change: more data favor the optimizer, while less data favor 1/N. To replicate am, change the integer inside random.seed(20260816) and run the sweep again.

Why optimizer dey amplify bad mean estimate

Look where the estimates dey enter the weight. Estimated mean dey for numerator, while estimated variance dey for denominator. If you double mean estimate, weight go double. If you cut variance estimate by one-quarter, weight go rise by one-third. For lucky sample, both errors fit point the same way: one run of good returns dey push measured mean up, and e often dey reduce measured variance for that same period. Optimizer go read that asset as the standout among the group and size the position like that. Outside the sample, e go return to being ordinary member of the group. Equal weight no ever see that lucky sample.

The simulation na the softer version of this problem. Five uncorrelated assets leave diagonal covariance matrix with nothing to invert. Real portfolios dey correlated, and sample covariance matrix wey come from only slightly more observations than assets dey close to singular. When you invert am, small errors for the inputs turn into very large weights. Na so optimizer fit arrive at position worth several times the capital for one name, against offsetting short for close substitute. Our note on concentration risk for portfolio explains wetin that kind weight fit do to drawdown.

The input no dey stay fixed too. The panel below dey track rolling one-year average daily return, annualized and sampled monthly, for broad index fund alongside one large consumer staples name.

WITH prices AS
(
    SELECT
        ticker,
        date,
        toFloat64(max(close)) AS c
    FROM global_markets.stocks_daily_aggs
    WHERE ticker IN ('SPY', 'KO')
      AND date >= '2015-10-01'
      AND date <  '2025-01-01'
    GROUP BY ticker, date
),
rets AS
(
    SELECT
        ticker,
        date,
        c / lagInFrame(c, 1) OVER (PARTITION BY ticker ORDER BY date ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) - 1 AS ret
    FROM prices
),
trailing AS
(
    SELECT
        ticker,
        date,
        avg(ret) OVER (PARTITION BY ticker ORDER BY date ASC ROWS BETWEEN 251 PRECEDING AND CURRENT ROW) * 252 * 100 AS trailing_mean_pct,
        row_number() OVER (PARTITION BY ticker ORDER BY date ASC) AS i
    FROM rets
    WHERE isFinite(ret)
)
SELECT
    toString(toStartOfMonth(date))                     AS month,
    round(avgIf(trailing_mean_pct, ticker = 'SPY'), 1) AS spy_trailing_mean_pct,
    round(avgIf(trailing_mean_pct, ticker = 'KO'), 1)  AS ko_trailing_mean_pct
FROM trailing
WHERE i >= 252
  AND date >= '2017-01-01'
GROUP BY month
ORDER BY month ASC

Every point on those two lines na number wey some optimizer fit accept as expected return for that date. The first monthly reading for SPY na 17.7% and the last na 25.8%, with 96 readings between dem. KO dey open the same window at -0.4%. Any weighting scheme wey dey use this series month by month go inherit every wobble inside am.

Half Kelly and shrinkage fit solve am?

Dem soften the problem. Four fixes dey show for real-world use, and each one gives up part of the theoretical optimum to reduce how sensitive the result be to the estimate.

• Half Kelly dey cut every weight by half. Kelly growth curve flat near im peak, but e steep below am. So half stake dey give up small part of the theoretical growth rate, while e dey reduce the damage from overstated edge by much more. Volatility targeting dey apply the same idea to the denominator. E sizes from the one input wey short sample fit provide.
• Shrinkage dey pull every estimated mean toward the cross-sectional average of the estimates. The amount dey increase as the sample become smaller. If you push am to the limit, every asset get the same expected return. With equal volatilities, optimizer go then return equal weight by itself. Equal weight na the fully shrunk portfolio.
• Constraints, meaning no shorting and a cap for each position, dey limit how far one lucky estimate fit push one weight. Dem na damage control, no be improvement to the estimate.
• If you remove expected returns completely, you remain with minimum variance and risk parity. Dem use covariance only. Na the input wey the sample fit actually provide.

None of the four dey create information wey sample no contain. Wetin dem change na how much of the portfolio depend on information wey you fit no get.

Wetin optimization dey bring

The crossover fit happen for both directions. Once T big reach relative to the spread between the true means, the optimized allocation go overtake am and remain ahead. The wider that spread, the earlier e go happen. Optimization still be the correct tool whenever the input na wetin data dey provide. Covariance and hedge ratios dey estimate far better than expected returns.

Equal weight get im own cost. E dey ignore information wey you truly get. For small universe, e fit concentrate as much as any optimizer. Treat am as the baseline wey any candidate weighting must beat, using data wey the weights never see. To fit weights and score dem on the same sample na look-ahead bias for backtesting wey just wear another cap, and e dey make the optimizer look better every time. If you want an honest range around measured edge instead of one point estimate, bootstrapped confidence intervals na the standard tool.

FAQ

Equal weight really dey beat mean variance optimization?

For short samples, e often dey. Optimizer need estimate of expected return for each asset. That estimate get standard error wey roughly equal the asset volatility divided by the square root of the sample length in years. As long as that error remain bigger than the real differences between the assets, optimizer dey sort noise. Equal weight no need estimate wey fit wrong.

How many observations you need to estimate expected return?

You need pass wetin almost anybody get. For asset wey get 20 percent volatility, one year of daily data leave standard error near 20 percentage points per year. To cut that error by half, you need four times the calendar period. Adding intraday observations inside the same year no go help. But you fit use just some months of data for volatility.

Half Kelly na simply smaller bet?

Na smaller bet with better terms. Kelly growth curve flat near the peak. So, if you halve the stake, you give up only small part of the theoretical growth rate, while you roughly halve the path volatility and reduce the cost if you overstate your edge.

Wetin be the 1/N portfolio?

Na portfolio wey put equal fraction of capital inside each of N positions and rebalance according to schedule. E no need return forecast or covariance estimate. Na this make am the standard benchmark for every weighting scheme wey need those estimates.

Every panel here get the exact SQL under the chart, and the simulation carry its seed. If you wan run the same estimate stability check across your own list of names, ask for am in plain English on the Strasmore terminal.