Brier score: how to grade forecast

Brier score dey grade probability forecast against wetin really happen. Take the probability wey you talk, subtract the outcome (1 if the event happen, 0 if e no happen), square the difference, then calculate the average of all the squares across every forecast wey you make. Lower score better: 0 mean perfect record, while 0.25 na the score wey you go get if you talk “50%” for everything.

Wetin be Brier score?

Forecast na claim about probability, and one probability by itself no fit be right or wrong. You talk say something get 30% chance and e happen. You dey wrong? Not yet. Glenn Brier, wey write for weather forecasters in 1950, solve the matter by refusing to grade any single forecast. The score dey grade the whole log at once.

See one made-up log of ten forecasts. None of these numbers come from market. Dem na figures wey dem invent to show the arithmetic.

• 90% stated, the event happened. Squared miss 0.01.
• 80% stated, e happened. 0.04.
• 70% stated, e no happen. 0.49.
• 60% stated, e happened. 0.16.
• 50% stated, e no happen. 0.25.
• 40% stated, e no happen. 0.16.
• 30% stated, e happened. 0.49.
• 20% stated, e no happen. 0.04.
• 10% stated, e no happen. 0.01.
• 95% stated, e happened. 0.0025.

The ten squared misses add up to 1.6525. Divide am by ten, and the Brier score na 0.165. Na the complete calculation be that, and e dey run inside the python3 wey already come with any Mac or Linux machine. You no need install anything, and no libraries dey needed.

• rows = [(0.90, 1), (0.80, 1), (0.70, 0), (0.60, 1), (0.50, 0), (0.40, 0), (0.30, 1), (0.20, 0), (0.10, 0), (0.95, 1)]
• brier = sum((p - o) * (p - o) for p, o in rows) / len(rows)
• flat = sum((0.5 - o) * (0.5 - o) for _, o in rows) / len(rows)
• print(round(brier, 3), round(flat, 3)) prints 0.165 0.25

Why 0.25 na the number wey person need beat

If you talk say 50% for everything, every squared miss go be 0.25, no matter wetin happen. That fixed 0.25 na the no-information benchmark for any set of yes-or-no questions. The skill score turn am into one figure: 1 minus your score divided by the benchmark score. The made-up log wey score 0.165 get skill score of 0.34.

One warning fit spoil plenty bad scorekeeping. If the events wey you dey grade only happen 10% of the time, saying flat 10% every time fit score 0.09, even though you know nothing about the individual questions. Score below 0.25 no be evidence of skill when the questions no get balanced outcomes. The honest benchmark na the base rate of the questions wey you actually answer.

Calibration and confidence dey get grade together

Calibration mean say, out of everything wey you call 70%, nearly 70% of am go happen. Confidence, wey statisticians dey call resolution, na how far you willing move away from the base rate when you know something. Forecaster wey dey answer 50% for every question fit get perfect calibration but e no useful at all. Brier score dey measure both qualities together: miscalibration dey increase am, while confidence wey prove correct dey reduce am.

If you group the made-up log by the probability wey dem state, you fit see how calibration check dey work. For python, na one line for each bucket, hits = [o for p, o in rows if p >= 0.8], then calculate the average of hits.

• Dem state 10% to 30%, average na 20%: 1 out of 3 happen, 33%.
• Dem state 40% to 50%, average na 45%: 0 out of 2 happen, 0%.
• Dem state 60% to 70%, average na 65%: 1 out of 2 happen, 50%.
• Dem state 80% to 95%, average na 88%: 3 out of 3 happen, 100%.

Ten forecasts no near enough to understand anything from those buckets. Calibration need hundreds of resolved questions for each bucket. The remaining part of this page dey use forecast log wey get millions of forecasts.

Market own forecast dey get grade

Every listed option get one number wey dey behave like stated probability. Delta dey measure how much option price dey move when underlying stock move one dollar. For contract wey go finish in the money (get value for expiry) or worthless, absolute value of delta dey close to market implied probability say e go finish in the money. E dey come from the same surface wey set AAPL implied volatility. Every contract dey expire, so dem dey grade every one of those forecasts.

The panel below dey take every SPY option wey dem observe roughly one month before expiry, from January 2025 reach May 2026. E group dem by stated delta, then count how often each contract finish in the money.

WITH settle AS
(
    SELECT
        date                             AS settle_date,
        any(toFloat64(underlying_close)) AS settle_px
    FROM global_markets.options_greeks
    WHERE underlying_symbol = 'SPY'
      AND date >= '2025-01-01'
      AND date <  '2026-08-01'
    GROUP BY date
),
scored AS
(
    SELECT
        toUInt8(floor(abs(toFloat64(g.delta)) * 10))    AS bucket,
        abs(toFloat64(g.delta))                         AS stated,
        startsWith(lower(toString(g.option_type)), 'c') AS is_call,
        if(is_call,
           s.settle_px > toFloat64(g.strike_price),
           s.settle_px < toFloat64(g.strike_price))     AS finished_itm
    FROM global_markets.options_greeks AS g
    INNER JOIN settle AS s ON s.settle_date = g.expiration_date
    WHERE g.underlying_symbol = 'SPY'
      AND g.date >= '2025-01-01'
      AND g.date <  '2026-06-01'
      AND g.expiration_date <= '2026-07-31'
      AND g.days_to_expiry BETWEEN 28 AND 35
      AND g.iv_converged = 1
      AND g.volume > 0
      AND abs(g.delta) > 0.02
      AND abs(g.delta) < 0.98
)
SELECT
    concat(toString(bucket * 10), ' to ', toString(bucket * 10 + 10), '%') AS stated_bucket,
    round(avg(stated) * 100, 1)       AS stated_pct,
    round(avg(finished_itm) * 100, 1) AS finished_itm_pct,
    count()                           AS sample_size
FROM scored
GROUP BY bucket
ORDER BY bucket

Compare the two series with each other. For the lowest bucket, market state average of 5.2%, and those contracts finish in the money 3.7% of the time. For the highest bucket, stated 94% come with 96.5% finishing in the money. Across all 10 buckets, the realized frequency dey around the same level as the stated one. Na that calibration table dey do: e show the gap between wetin forecaster talk and wetin happen, bucket by bucket, instead of hiding the gap inside one average.

Market dey beat coin flip?

Now na the score itself. We use the same setup for several well-known names. Each name’s Brier score dey beside the flat 50% baseline, wey we calculate from exactly the same contracts.

WITH settle AS
(
    SELECT
        underlying_symbol                AS sym,
        date                             AS settle_date,
        any(toFloat64(underlying_close)) AS settle_px
    FROM global_markets.options_greeks
    WHERE underlying_symbol IN ('SPY', 'AAPL', 'MSFT', 'NVDA', 'KO', 'TSLA')
      AND date >= '2025-01-01'
      AND date <  '2026-08-01'
    GROUP BY sym, settle_date
),
scored AS
(
    SELECT
        g.underlying_symbol                             AS symbol,
        abs(toFloat64(g.delta))                         AS stated,
        startsWith(lower(toString(g.option_type)), 'c') AS is_call,
        if(is_call,
           s.settle_px > toFloat64(g.strike_price),
           s.settle_px < toFloat64(g.strike_price))     AS finished_itm
    FROM global_markets.options_greeks AS g
    INNER JOIN settle AS s
        ON s.sym = g.underlying_symbol AND s.settle_date = g.expiration_date
    WHERE g.underlying_symbol IN ('SPY', 'AAPL', 'MSFT', 'NVDA', 'KO', 'TSLA')
      AND g.date >= '2025-01-01'
      AND g.date <  '2026-06-01'
      AND g.expiration_date <= '2026-07-31'
      AND g.days_to_expiry BETWEEN 28 AND 35
      AND g.iv_converged = 1
      AND g.volume > 0
      AND abs(g.delta) > 0.02
      AND abs(g.delta) < 0.98
)
SELECT
    symbol,
    round(avg((stated - finished_itm) * (stated - finished_itm)), 4) AS market_brier,
    round(avg((0.5 - finished_itm) * (0.5 - finished_itm)), 4)       AS coin_flip_brier,
    formatReadableQuantity(count())                                  AS graded_contracts
FROM scored
GROUP BY symbol
ORDER BY market_brier

NVDA scores 0.1122 across 28.54 thousand graded contracts, against the flat baseline’s 0.25 on the same set. The weakest among the 6 names, SPY, still get 0.1375. Options no be magic here. Deep out-of-the-money contracts get deltas near 0.02, and most of dem expire worthless. That one na easy forecast to get right, and the score already reflect the advantage. Na the main reason why Brier score wey dem quote alone hardly tell you anything.

Same forecaster fit score differently for different questions

If na the same method, the score fit still move depending on how far the question dey from resolution.

WITH settle AS
(
    SELECT
        date                             AS settle_date,
        any(toFloat64(underlying_close)) AS settle_px
    FROM global_markets.options_greeks
    WHERE underlying_symbol = 'SPY'
      AND date >= '2025-01-01'
      AND date <  '2026-08-01'
    GROUP BY date
),
scored AS
(
    SELECT
        multiIf(g.days_to_expiry <=   7, 1,
                g.days_to_expiry <=  14, 2,
                g.days_to_expiry <=  30, 3,
                g.days_to_expiry <=  60, 4,
                g.days_to_expiry <= 120, 5,
                6)                                      AS horizon_rank,
        abs(toFloat64(g.delta))                         AS stated,
        startsWith(lower(toString(g.option_type)), 'c') AS is_call,
        if(is_call,
           s.settle_px > toFloat64(g.strike_price),
           s.settle_px < toFloat64(g.strike_price))     AS finished_itm
    FROM global_markets.options_greeks AS g
    INNER JOIN settle AS s ON s.settle_date = g.expiration_date
    WHERE g.underlying_symbol = 'SPY'
      AND g.date >= '2025-01-01'
      AND g.date <  '2026-06-01'
      AND g.expiration_date <= '2026-07-31'
      AND g.days_to_expiry BETWEEN 1 AND 250
      AND g.iv_converged = 1
      AND g.volume > 0
      AND abs(g.delta) > 0.02
      AND abs(g.delta) < 0.98
)
SELECT
    multiIf(horizon_rank = 1, '1 to 7 days',
            horizon_rank = 2, '8 to 14 days',
            horizon_rank = 3, '15 to 30 days',
            horizon_rank = 4, '31 to 60 days',
            horizon_rank = 5, '61 to 120 days',
            '121 to 250 days')                                       AS horizon,
    round(avg((stated - finished_itm) * (stated - finished_itm)), 4) AS market_brier,
    round(avg((0.5 - finished_itm) * (0.5 - finished_itm)), 4)       AS coin_flip_brier,
    count()                                                          AS sample_size
FROM scored
GROUP BY horizon_rank
ORDER BY horizon_rank

The market’s delta score 0.1084 for contracts wey get 1 to 7 days left to run, and 0.1218 for contracts wey get 121 to 250 days left to run. Na the same forecaster and same method, but two different sets of questions. The flat 50% baseline dey at 0.25 for the first row and 0.25 for the last row. Na the one fixed reference across every horizon. Any comparison between two forecasters must use the same questions for the same window. Otherwise, na question difficulty you dey compare, no be skill.

Why proper scoring rules dey matter

Mean absolute error, wey be the average plain distance between your probability and the outcome, sound like reasonable alternative but e dey grade badly. Suppose you genuinely believe say event get 70% chance. If you state 70%, your expected absolute error na 0.7 x 0.3 + 0.3 x 0.7 = 0.42. If you state 100% instead, e fall to 0.7 x 0 + 0.3 x 1 = 0.30. Under absolute error, honest number dey cost you. Metric wey dey reward you for overstating confidence no dey measure forecasting properly.

Brier score no get this problem. If you believe 70% and state 70%, your expected score na 0.7 x 0.09 + 0.3 x 0.49 = 0.21. If you state 100%, e rise to 0.30. If you state 60%, e rise to 0.22. The minimum dey exactly on the number wey you believe. Rule wey get this property dem dey call proper. Na why Brier become the default for forecasting tournaments.

Log score na the other common proper rule. You take natural log of the probability wey you assign to the outcome wey happen, then you turn the sign. If you state 1% for something wey happen, e cost 4.6. If you state 0%, e cost infinity. For the made-up ten forecast log, e come to 0.483 against 0.693 for the flat baseline. Brier dey between 0 and 1, so e read more naturally as scorecard. Log score no get upper limit. This one suit grading where the tails carry the risk.

Wetin this mean for event contracts

Event contract wey go pay $1 if something happen and $0 if e no happen dey trade for price wey don already show probability. Sixty two cents na 62% forecast, before dem remove spread and fees. Our guide on event contract prices as probabilities explain this conversion, while how event contracts settle explain wetin “e happen” mean according to the contract own wording.

That price na forecast wey get public track record and settlement date. So na benchmark wey your own log need beat. Score your forecasts on the same questions and for the same period. If trader Brier score dey above the price own, e get no measured edge for that question set, no matter how convincing the trade story sound. The same test work for sports odds after you remove the vig from betting odds, and for rate markets, where Fed rate odds give you dated probability for question wey get known resolution day.

FAQ

Lower Brier score better?

Yes. Brier score na error measurement, so 0 mean perfect record and 1 na the worst possible result. You fit get 1 if you talk say everything get 100% chance, but none of them happen. Any score below 0.25 better pass the score wey you go get if you answer 50% for every question.

Wetin be good Brier score?

No single number dey universally good, because the score depend on how hard the questions be. Weather forecaster wey get 0.10 for tomorrow rain and political forecaster wey get 0.18 for close elections no fit rank against each other. Compare scores only among forecasters wey answer the same questions for the same period.

Wetin Brier score of 0.25 mean?

Na the score for forecaster wey talk 50% for everything, because every squared miss go be 0.25, no matter wetin happen. Na the standard no-information benchmark for set of yes-or-no questions. But if the questions dey heavily favor one answer, use the base rate as the benchmark instead.

How Brier score different from log score?

Both na proper scoring rules. This mean each one get the lowest score when you state the probability wey you truly believe. Brier squares the miss and stays between 0 and 1. Log score punish confident misses much harder. If you state 0% for something wey later happen, the cost na infinity.

Option delta fit count as probability?

The absolute value of delta dey close to the market’s implied probability say the option go finish in the money. But na risk-neutral probability, no be real-world probability. The panels above grade am as forecast: stated delta dey for one axis, while the share of those contracts wey actually finish in the money dey for the other.

Every panel here come with the exact SQL underneath am, so person fit audit the grading line by line. If you wan score your own forecast log against the market forecast for the same questions, ask for the numbers in plain English on the Strasmore terminal.