By Matt Connor
Brier Score: How to Grade a Forecast
The Brier score grades probability forecasts as mean squared error against what happened. Lower is better, and 0.25 is what always saying 50% gets you.
The Brier score grades a probability forecast against what actually happened. Take the probability you stated, subtract the outcome (1 if the event happened, 0 if it did not), square that difference, then average the squares over every forecast you made. Lower is better: 0 is a perfect record, and 0.25 is the score you collect by saying "50%" about everything.
What is a Brier score?
A forecast is a claim about probability, and one probability can never be right or wrong on its own. You called something 30% likely and it happened. Were you wrong? Not yet. Glenn Brier, writing for weather forecasters in 1950, got around that by refusing to grade any single forecast. The score grades the whole log at once.
Here is a made-up log of ten forecasts. None of these numbers come from a market, they are invented to show the arithmetic.
- 90% stated, the event happened. Squared miss 0.01.
- 80% stated, happened. 0.04.
- 70% stated, did not happen. 0.49.
- 60% stated, happened. 0.16.
- 50% stated, did not happen. 0.25.
- 40% stated, did not happen. 0.16.
- 30% stated, happened. 0.49.
- 20% stated, did not happen. 0.04.
- 10% stated, did not happen. 0.01.
- 95% stated, happened. 0.0025.
The ten squared misses add to 1.6525. Divide by ten and the Brier score is 0.165. That is the entire calculation, and it runs in the python3 that already ships on any Mac or Linux machine. Nothing to install, no libraries.
rows = [(0.90, 1), (0.80, 1), (0.70, 0), (0.60, 1), (0.50, 0), (0.40, 0), (0.30, 1), (0.20, 0), (0.10, 0), (0.95, 1)]brier = sum((p - o) * (p - o) for p, o in rows) / len(rows)flat = sum((0.5 - o) * (0.5 - o) for _, o in rows) / len(rows)print(round(brier, 3), round(flat, 3))prints0.165 0.25
Why 0.25 is the number to beat
Say 50% about everything and every squared miss is 0.25, whatever happens. That fixed 0.25 is the no-information benchmark for any set of yes or no questions, and the skill score turns it into one figure: 1 minus your score divided by the benchmark's. The made-up log at 0.165 works out to a skill score of 0.34.
One caveat kills a lot of bad scorekeeping. If the events you are grading only happen 10% of the time, stating a flat 10% every time scores 0.09 while knowing nothing whatsoever about the individual questions. A score under 0.25 is not evidence of skill on a lopsided question set. The honest benchmark is the base rate of the questions you actually answered.
Calibration and confidence are graded together
Calibration means that of everything you called 70%, close to 70% of it happens. Confidence, which statisticians call resolution, is how far you are willing to move away from the base rate when you know something. A forecaster who says 50% to every question is perfectly calibrated and completely useless, and the Brier score prices both properties at once: miscalibration raises it, and earned confidence lowers it.
Bucketing the made-up log by stated probability shows what a calibration check looks like. In python that is one line per bucket, hits = [o for p, o in rows if p >= 0.8], then the average of hits.
- 10% to 30% stated, average 20%: 1 of 3 happened, 33%.
- 40% to 50% stated, average 45%: 0 of 2 happened, 0%.
- 60% to 70% stated, average 65%: 1 of 2 happened, 50%.
- 80% to 95% stated, average 88%: 3 of 3 happened, 100%.
Ten forecasts is nowhere near enough to read anything from those buckets. Calibration needs hundreds of resolved questions per bucket. The rest of this page uses a forecast log with millions of them.
Grading the market's own forecast
Every listed option carries a number that behaves like a stated probability. Delta measures how much the option's price moves for a one dollar move in the underlying stock, and for a contract that either finishes in the money (worth something at expiry) or worthless, the absolute value of delta sits close to the market's implied probability that it lands in the money. It comes off the same surface that sets AAPL implied volatility. Every contract expires, so every one of those forecasts gets graded.
The panel below takes every SPY option observed roughly a month before expiry, from January 2025 through May 2026, buckets them by stated delta, and counts how often the contract finished in the money.
The exact SQL behind every number
WITH settle AS
(
SELECT
date AS settle_date,
any(toFloat64(underlying_close)) AS settle_px
FROM global_markets.options_greeks
WHERE underlying_symbol = 'SPY'
AND date >= '2025-01-01'
AND date < '2026-08-01'
GROUP BY date
),
scored AS
(
SELECT
toUInt8(floor(abs(toFloat64(g.delta)) * 10)) AS bucket,
abs(toFloat64(g.delta)) AS stated,
startsWith(lower(toString(g.option_type)), 'c') AS is_call,
if(is_call,
s.settle_px > toFloat64(g.strike_price),
s.settle_px < toFloat64(g.strike_price)) AS finished_itm
FROM global_markets.options_greeks AS g
INNER JOIN settle AS s ON s.settle_date = g.expiration_date
WHERE g.underlying_symbol = 'SPY'
AND g.date >= '2025-01-01'
AND g.date < '2026-06-01'
AND g.expiration_date <= '2026-07-31'
AND g.days_to_expiry BETWEEN 28 AND 35
AND g.iv_converged = 1
AND g.volume > 0
AND abs(g.delta) > 0.02
AND abs(g.delta) < 0.98
)
SELECT
concat(toString(bucket * 10), ' to ', toString(bucket * 10 + 10), '%') AS stated_bucket,
round(avg(stated) * 100, 1) AS stated_pct,
round(avg(finished_itm) * 100, 1) AS finished_itm_pct,
count() AS sample_size
FROM scored
GROUP BY bucket
ORDER BY bucketRead the two series against each other. In the lowest bucket the market stated an average of 5.2% and those contracts finished in the money 3.7% of the time. In the highest bucket a stated 94% came with 96.5% finishing in the money. Across all 10 buckets the realized frequency sits in the same neighborhood as the stated one. That is the whole job of a calibration table: it exposes the gap between what a forecaster says and what happens, bucket by bucket, instead of hiding it inside a single average.
Does the market beat the coin flip?
Now the score itself. Same construction, several household names, each name's Brier score placed next to the flat 50% baseline computed over exactly the same contracts.
The exact SQL behind every number
WITH settle AS
(
SELECT
underlying_symbol AS sym,
date AS settle_date,
any(toFloat64(underlying_close)) AS settle_px
FROM global_markets.options_greeks
WHERE underlying_symbol IN ('SPY', 'AAPL', 'MSFT', 'NVDA', 'KO', 'TSLA')
AND date >= '2025-01-01'
AND date < '2026-08-01'
GROUP BY sym, settle_date
),
scored AS
(
SELECT
g.underlying_symbol AS symbol,
abs(toFloat64(g.delta)) AS stated,
startsWith(lower(toString(g.option_type)), 'c') AS is_call,
if(is_call,
s.settle_px > toFloat64(g.strike_price),
s.settle_px < toFloat64(g.strike_price)) AS finished_itm
FROM global_markets.options_greeks AS g
INNER JOIN settle AS s
ON s.sym = g.underlying_symbol AND s.settle_date = g.expiration_date
WHERE g.underlying_symbol IN ('SPY', 'AAPL', 'MSFT', 'NVDA', 'KO', 'TSLA')
AND g.date >= '2025-01-01'
AND g.date < '2026-06-01'
AND g.expiration_date <= '2026-07-31'
AND g.days_to_expiry BETWEEN 28 AND 35
AND g.iv_converged = 1
AND g.volume > 0
AND abs(g.delta) > 0.02
AND abs(g.delta) < 0.98
)
SELECT
symbol,
round(avg((stated - finished_itm) * (stated - finished_itm)), 4) AS market_brier,
round(avg((0.5 - finished_itm) * (0.5 - finished_itm)), 4) AS coin_flip_brier,
formatReadableQuantity(count()) AS graded_contracts
FROM scored
GROUP BY symbol
ORDER BY market_brierNVDA scores 0.1122 over 28.54 thousand graded contracts, against the flat baseline's 0.25 on the identical set. The weakest of the 6 names, SPY, still lands at 0.1375. Options are not magic here. Deep out of the money contracts carry deltas near 0.02 and mostly expire worthless, which is an easy forecast to get right, and that easiness is baked into the number. It is the main reason a Brier score quoted on its own tells you almost nothing.
The same forecaster scores differently on different questions
Split the identical method by how far out the question resolves and the score moves around under it.
The exact SQL behind every number
WITH settle AS
(
SELECT
date AS settle_date,
any(toFloat64(underlying_close)) AS settle_px
FROM global_markets.options_greeks
WHERE underlying_symbol = 'SPY'
AND date >= '2025-01-01'
AND date < '2026-08-01'
GROUP BY date
),
scored AS
(
SELECT
multiIf(g.days_to_expiry <= 7, 1,
g.days_to_expiry <= 14, 2,
g.days_to_expiry <= 30, 3,
g.days_to_expiry <= 60, 4,
g.days_to_expiry <= 120, 5,
6) AS horizon_rank,
abs(toFloat64(g.delta)) AS stated,
startsWith(lower(toString(g.option_type)), 'c') AS is_call,
if(is_call,
s.settle_px > toFloat64(g.strike_price),
s.settle_px < toFloat64(g.strike_price)) AS finished_itm
FROM global_markets.options_greeks AS g
INNER JOIN settle AS s ON s.settle_date = g.expiration_date
WHERE g.underlying_symbol = 'SPY'
AND g.date >= '2025-01-01'
AND g.date < '2026-06-01'
AND g.expiration_date <= '2026-07-31'
AND g.days_to_expiry BETWEEN 1 AND 250
AND g.iv_converged = 1
AND g.volume > 0
AND abs(g.delta) > 0.02
AND abs(g.delta) < 0.98
)
SELECT
multiIf(horizon_rank = 1, '1 to 7 days',
horizon_rank = 2, '8 to 14 days',
horizon_rank = 3, '15 to 30 days',
horizon_rank = 4, '31 to 60 days',
horizon_rank = 5, '61 to 120 days',
'121 to 250 days') AS horizon,
round(avg((stated - finished_itm) * (stated - finished_itm)), 4) AS market_brier,
round(avg((0.5 - finished_itm) * (0.5 - finished_itm)), 4) AS coin_flip_brier,
count() AS sample_size
FROM scored
GROUP BY horizon_rank
ORDER BY horizon_rankThe market's delta scored 0.1084 on contracts with 1 to 7 days to run and 0.1218 on contracts with 121 to 250 days to run. Same forecaster, same method, two different sets of questions. The flat 50% baseline sits at 0.25 in the first row and 0.25 in the last, which makes it the one fixed reference across every horizon. Any comparison between two forecasters has to run on the same questions over the same window, or it is comparing question difficulty rather than skill.
Why proper scoring rules matter
Mean absolute error, the average of the plain distance between your probability and the outcome, sounds like a reasonable alternative and grades badly. Suppose you genuinely believe an event is 70% likely. State 70% and your expected absolute error is 0.7 x 0.3 + 0.3 x 0.7 = 0.42. State 100% instead and it falls to 0.7 x 0 + 0.3 x 1 = 0.30. Under absolute error the honest number costs you, and a metric that pays you to overstate confidence is not measuring forecasting.
The Brier score has no such hole. Believe 70% and state 70%, and your expected score is 0.7 x 0.09 + 0.3 x 0.49 = 0.21. State 100% and it rises to 0.30. State 60% and it rises to 0.22. The minimum sits exactly on the number you believe. A rule with that property is called proper, and it is the reason Brier became the default in forecasting tournaments.
The log score is the other common proper rule: take the natural log of the probability you assigned to the outcome that happened, then flip the sign. Stating 1% on something that happens costs 4.6, and stating 0% costs infinity. On the made-up ten forecast log it comes to 0.483 against 0.693 for the flat baseline. Brier stays between 0 and 1, which reads more naturally as a scorecard. The log score is unbounded, which suits grading where the tails carry the risk.
What this means for event contracts
An event contract that pays $1 if something happens and $0 if it does not trades at a price that already states a probability. Sixty two cents is a 62% forecast, before the spread and fees are stripped out. Our guide to event contract prices as probabilities covers that conversion, and how event contracts settle covers what "it happened" means in the contract's own words.
That price is a forecast with a public track record and a settlement date, which makes it the benchmark your own log has to clear. Score your forecasts on the same questions over the same window. A trader whose Brier score comes in above the price's has no measured edge on that question set, however good the narrative attached to the trade. The same test travels to sports odds once you remove the vig from betting odds, and to rate markets, where Fed rate odds hand you a dated probability for a question with a known resolution day.
How these panels grade the market
Delta comes from the daily options greeks record for each contract. Only contracts with volume on the observation day and a converged volatility solve are included. The outcome is read from the underlying's close on the contract's expiration date: a call counts as in the money when that close sits above the strike, a put when it sits below. Real settlement runs through an exercise decision after the close, so contracts pinned within pennies of the strike can settle against this simplification. Every contract in these panels has already expired, and the longer horizon buckets necessarily draw from earlier observation dates in the window. A share split falling between the observation date and expiry would put the strike and the settlement price on different scales, one more reason each bucket carries its sample count.
Delta approximates a risk-neutral probability rather than a real-world one. The two differ by the risk premium embedded in option prices, and a calibration table is the instrument that measures that difference instead of assuming it away.
FAQ
Is a lower Brier score better?
Yes. The Brier score is an error measurement, so 0 is a perfect record and 1 is the worst possible, earned by stating 100% on everything that failed to happen. Anything under 0.25 beats the score you would get by answering 50% to every question.
What is a good Brier score?
There is no universal good number, since the score depends on how hard the questions were. A weather forecaster at 0.10 on tomorrow's rain and a political forecaster at 0.18 on close elections cannot be ranked against each other. Compare scores only across forecasters answering the same questions over the same window.
What does a Brier score of 0.25 mean?
It is the score of a forecaster who says 50% to everything, since every squared miss is then 0.25 whatever the outcome. It is the standard no-information benchmark for a set of yes or no questions, though a lopsided question set needs the base rate as its benchmark instead.
How is the Brier score different from the log score?
Both are proper scoring rules, meaning each is minimized when you state the probability you actually believe. Brier squares the miss and stays between 0 and 1. The log score punishes confident misses far harder, and a stated 0% on something that happens costs infinity.
Can option delta be read as a probability?
The absolute value of delta sits close to the market's implied probability that the option finishes in the money, and it is a risk-neutral probability rather than a real-world one. The panels above grade it as a forecast: stated delta on one axis, the share of those contracts that actually finished in the money on the other.
Every panel here ships with the exact SQL beneath it, so the grading is auditable line by line. To score a forecast log of your own against the market's on the same questions, ask for the numbers in plain English on the Strasmore terminal.