By Matt Connor · LLM fit find alpha factors?
LLM fit write one hundred alpha factors for one hour. See how 240 coin flip factors score on ten years of real prices, plus how to test survivors.
LLM fit fit alpha factors all day. Give capable model data dictionary plus scoring harness, e go write hundred plausible factor expressions before lunch. The harder question dey underneath: how person go know whether any one real, when the search wey produce am na machine for manufacturing winners from noise?
Wetin be alpha factor?
Factor na rule wey dey turn market data into one number for every stock on every date. Twelve-month price change na factor. Ratio of debt to equity sef na factor. Factor become strategy when you rank universe with am, buy the top slice, sell the bottom slice, then rebalance on schedule. Alpha na any return wey remain after you subtract wetin ordinary market exposure for give you anyway.
Dem dey score candidates with Sharpe ratio: average return divided by standard deviation of that return, then scale am to one year. Na return per unit of wobble. Long-run Sharpe near 1 for live strategy respectable, so remember am next time backtest quote 3.
How LLM factor research really dey work
Every project for this area dey run one version of the same loop.
- Model dey write factor expressions for small language wey harness fit evaluate.
- Backtester dey score each expression across fixed history of prices and fundamentals.
- Dem dey keep expressions wey pass score threshold. Dem dey drop the rest.
- Keepers dey return to model context as worked examples, then loop run again.
Multi-agent trading systems dey divide those jobs across separate roles, one proposing and one testing. The plumbing genuinely useful, and the market data skills wey AI agent need na the same ones person need.
Nothing for the loop dishonest. Search na how research dey happen. The problem na arithmetic, and e show immediately step 2 run more than small number of times.
Why LLM alpha factor search dey manufacture winners
One price history. Thousands of cheap hypotheses. Every hypothesis dey get score against the same finite sample, and that sample carry plenty luck. Test enough rules and some go fit the luck closely. The score no fit tell you which kind fit you get, because rule wey match noise and rule wey match market go print the same number.
See the null hypothesis, drawn 240 times. Each "factor" below na coin flip: hash of ticker, month, and trial number dey split 40 large US names into two halves every month, and strategy dey hold one half long and the other short. No information dey inside am, by construction. When dem score am on real month-end returns from January 2016 through June 2021, the 240 trials land like this.
The exact SQL behind every number
WITH month_end AS (
SELECT ticker,
toStartOfMonth(toDate(toTimeZone(window_start, 'America/New_York'))) AS month_start,
argMax(toFloat64(close), window_start) AS close_px
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL','ADBE','AMZN','BA','CAT','COST','CRM','CSCO','CVX','DE',
'DUK','GE','GOOGL','HD','HON','IBM','INTC','JNJ','JPM','KO',
'LMT','MCD','MMM','MRK','MSFT','NKE','NVDA','ORCL','PEP','PFE',
'PG','QCOM','SO','T','TGT','TXN','UNP','VZ','WMT','XOM')
AND toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2015-12-01')
AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2021-06-30')
AND toDayOfMonth(toTimeZone(window_start, 'America/New_York')) >= 22
AND (toHour(toTimeZone(window_start, 'America/New_York')) * 60
+ toMinute(toTimeZone(window_start, 'America/New_York'))) BETWEEN 570 AND 959
GROUP BY ticker, month_start
),
lagged AS (
SELECT ticker,
month_start,
close_px,
lagInFrame(close_px) OVER (PARTITION BY ticker ORDER BY month_start
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS prev_px
FROM month_end
),
monthly_return AS (
SELECT ticker, month_start, close_px / prev_px - 1 AS ret
FROM lagged
WHERE prev_px > 0
AND month_start >= toDate('2016-01-01')
),
trial AS (
SELECT arrayJoin(range(1, 241)) AS n
),
factor_month AS (
SELECT t.n AS trial_id,
m.month_start AS month_start,
avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1)
- avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) AS long_short_ret
FROM monthly_return AS m
CROSS JOIN trial AS t
GROUP BY trial_id, month_start
HAVING countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1) > 0
AND countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) > 0
),
scored AS (
SELECT trial_id,
avg(long_short_ret) / stddevSamp(long_short_ret) * sqrt(12) AS sharpe
FROM factor_month
GROUP BY trial_id
HAVING stddevSamp(long_short_ret) > 0
)
SELECT multiIf(sharpe < -1.2, 'below -1.2',
sharpe < -0.8, '-1.2 to -0.8',
sharpe < -0.4, '-0.8 to -0.4',
sharpe < 0.0, '-0.4 to 0.0',
sharpe < 0.4, '0.0 to 0.4',
sharpe < 0.8, '0.4 to 0.8',
sharpe < 1.2, '0.8 to 1.2',
'1.2 and above') AS sharpe_bucket,
count() AS factor_count,
round(100 * count() / 240, 1) AS share_pct
FROM scored
GROUP BY sharpe_bucket
ORDER BY min(sharpe)The spread na the whole point. Nothing for that chart dey predict anything, yet 1 trials land for top band (1.2 and above), 0.4% of the search, and 1 for bottom band (below -1.2). Researcher wey run one lucky trial and stop go hold chart and Sharpe ratio, with no way to know whether either one na discovery. Returns here na month-end close to month-end close; how dem dey measure monthly returns cover the arithmetic.
The number wey matter na how many you try
Backtest wey dem report alone dey miss its denominator. Read the same 240 trials as search wey keep widening: for every step, best score for board beside average of everything wey dem try so far.
The exact SQL behind every number
WITH month_end AS (
SELECT ticker,
toStartOfMonth(toDate(toTimeZone(window_start, 'America/New_York'))) AS month_start,
argMax(toFloat64(close), window_start) AS close_px
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL','ADBE','AMZN','BA','CAT','COST','CRM','CSCO','CVX','DE',
'DUK','GE','GOOGL','HD','HON','IBM','INTC','JNJ','JPM','KO',
'LMT','MCD','MMM','MRK','MSFT','NKE','NVDA','ORCL','PEP','PFE',
'PG','QCOM','SO','T','TGT','TXN','UNP','VZ','WMT','XOM')
AND toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2015-12-01')
AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2021-06-30')
AND toDayOfMonth(toTimeZone(window_start, 'America/New_York')) >= 22
AND (toHour(toTimeZone(window_start, 'America/New_York')) * 60
+ toMinute(toTimeZone(window_start, 'America/New_York'))) BETWEEN 570 AND 959
GROUP BY ticker, month_start
),
lagged AS (
SELECT ticker,
month_start,
close_px,
lagInFrame(close_px) OVER (PARTITION BY ticker ORDER BY month_start
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS prev_px
FROM month_end
),
monthly_return AS (
SELECT ticker, month_start, close_px / prev_px - 1 AS ret
FROM lagged
WHERE prev_px > 0
AND month_start >= toDate('2016-01-01')
),
trial AS (
SELECT arrayJoin(range(1, 241)) AS n
),
factor_month AS (
SELECT t.n AS trial_id,
m.month_start AS month_start,
avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1)
- avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) AS long_short_ret
FROM monthly_return AS m
CROSS JOIN trial AS t
GROUP BY trial_id, month_start
HAVING countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1) > 0
AND countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) > 0
),
scored AS (
SELECT trial_id,
avg(long_short_ret) / stddevSamp(long_short_ret) * sqrt(12) AS sharpe
FROM factor_month
GROUP BY trial_id
HAVING stddevSamp(long_short_ret) > 0
),
ladder AS (
SELECT arrayJoin([1, 2, 5, 10, 25, 50, 100, 160, 240]) AS n
)
SELECT l.n AS factors_tried,
round(max(s.sharpe), 2) AS best_sharpe,
round(avg(s.sharpe), 2) AS average_sharpe
FROM ladder AS l
CROSS JOIN scored AS s
WHERE s.trial_id <= l.n
GROUP BY factors_tried
ORDER BY factors_triedRunning maximum fit only rise, and na exactly the trap. The first rule wey dem test score 0.44. After 240 trials, best one for board read 1.59, while average across all of dem sit at 0.01. Headline improve without any rule improving. Harness wey evaluate ten thousand expressions dey run this curve far to the right of anything wey show here, and the number wey e report na the top of the curve.
Wetin held-out period dey do to winners
The standard defense na holdout: score for one period, then score survivors again for later period wey search never touch. Take the twelve best coin flips from training window and run the identical rules across the five years wey follow, July 2021 through June 2026.
The exact SQL behind every number
WITH month_end AS (
SELECT ticker,
toStartOfMonth(toDate(toTimeZone(window_start, 'America/New_York'))) AS month_start,
argMax(toFloat64(close), window_start) AS close_px
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL','ADBE','AMZN','BA','CAT','COST','CRM','CSCO','CVX','DE',
'DUK','GE','GOOGL','HD','HON','IBM','INTC','JNJ','JPM','KO',
'LMT','MCD','MMM','MRK','MSFT','NKE','NVDA','ORCL','PEP','PFE',
'PG','QCOM','SO','T','TGT','TXN','UNP','VZ','WMT','XOM')
AND toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2015-12-01')
AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2026-06-30')
AND toDayOfMonth(toTimeZone(window_start, 'America/New_York')) >= 22
AND (toHour(toTimeZone(window_start, 'America/New_York')) * 60
+ toMinute(toTimeZone(window_start, 'America/New_York'))) BETWEEN 570 AND 959
GROUP BY ticker, month_start
),
lagged AS (
SELECT ticker,
month_start,
close_px,
lagInFrame(close_px) OVER (PARTITION BY ticker ORDER BY month_start
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS prev_px
FROM month_end
),
monthly_return AS (
SELECT ticker, month_start, close_px / prev_px - 1 AS ret
FROM lagged
WHERE prev_px > 0
AND month_start >= toDate('2016-01-01')
),
trial AS (
SELECT arrayJoin(range(1, 241)) AS n
),
factor_month AS (
SELECT t.n AS trial_id,
m.month_start AS month_start,
avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1)
- avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) AS long_short_ret
FROM monthly_return AS m
CROSS JOIN trial AS t
GROUP BY trial_id, month_start
HAVING countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1) > 0
AND countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) > 0
),
scored AS (
SELECT trial_id,
avgIf(long_short_ret, month_start < toDate('2021-07-01'))
/ stddevSampIf(long_short_ret, month_start < toDate('2021-07-01')) * sqrt(12) AS in_sample_sharpe,
avgIf(long_short_ret, month_start >= toDate('2021-07-01'))
/ stddevSampIf(long_short_ret, month_start >= toDate('2021-07-01')) * sqrt(12) AS out_of_sample_sharpe
FROM factor_month
GROUP BY trial_id
HAVING countIf(month_start < toDate('2021-07-01')) >= 24
AND countIf(month_start >= toDate('2021-07-01')) >= 24
)
SELECT concat('trial ', toString(trial_id)) AS factor_label,
round(in_sample_sharpe, 2) AS in_sample_sharpe,
round(out_of_sample_sharpe, 2) AS out_of_sample_sharpe
FROM scored
ORDER BY in_sample_sharpe DESC
LIMIT 12Each pair of bars na one rule. Left bar na score wey earn am place for report. Right bar na same rule across the next five years. Top-ranked trial score 1.59 for training and -0.51 afterwards; twelfth score 0.74 and then 0.49.
Twelve rules na small sample by itself. Sort all 240 into fifths by training score, then average each fifth's holdout score, and you get clearer view.
The exact SQL behind every number
WITH month_end AS (
SELECT ticker,
toStartOfMonth(toDate(toTimeZone(window_start, 'America/New_York'))) AS month_start,
argMax(toFloat64(close), window_start) AS close_px
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL','ADBE','AMZN','BA','CAT','COST','CRM','CSCO','CVX','DE',
'DUK','GE','GOOGL','HD','HON','IBM','INTC','JNJ','JPM','KO',
'LMT','MCD','MMM','MRK','MSFT','NKE','NVDA','ORCL','PEP','PFE',
'PG','QCOM','SO','T','TGT','TXN','UNP','VZ','WMT','XOM')
AND toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2015-12-01')
AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2026-06-30')
AND toDayOfMonth(toTimeZone(window_start, 'America/New_York')) >= 22
AND (toHour(toTimeZone(window_start, 'America/New_York')) * 60
+ toMinute(toTimeZone(window_start, 'America/New_York'))) BETWEEN 570 AND 959
GROUP BY ticker, month_start
),
lagged AS (
SELECT ticker,
month_start,
close_px,
lagInFrame(close_px) OVER (PARTITION BY ticker ORDER BY month_start
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS prev_px
FROM month_end
),
monthly_return AS (
SELECT ticker, month_start, close_px / prev_px - 1 AS ret
FROM lagged
WHERE prev_px > 0
AND month_start >= toDate('2016-01-01')
),
trial AS (
SELECT arrayJoin(range(1, 241)) AS n
),
factor_month AS (
SELECT t.n AS trial_id,
m.month_start AS month_start,
avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1)
- avgIf(m.ret, bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) AS long_short_ret
FROM monthly_return AS m
CROSS JOIN trial AS t
GROUP BY trial_id, month_start
HAVING countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 1) > 0
AND countIf(bitAnd(cityHash64(m.ticker, toString(m.month_start), t.n), 1) = 0) > 0
),
scored AS (
SELECT trial_id,
avgIf(long_short_ret, month_start < toDate('2021-07-01'))
/ stddevSampIf(long_short_ret, month_start < toDate('2021-07-01')) * sqrt(12) AS in_sample_sharpe,
avgIf(long_short_ret, month_start >= toDate('2021-07-01'))
/ stddevSampIf(long_short_ret, month_start >= toDate('2021-07-01')) * sqrt(12) AS out_of_sample_sharpe
FROM factor_month
GROUP BY trial_id
HAVING countIf(month_start < toDate('2021-07-01')) >= 24
AND countIf(month_start >= toDate('2021-07-01')) >= 24
),
ranked AS (
SELECT trial_id,
in_sample_sharpe,
out_of_sample_sharpe,
row_number() OVER (ORDER BY in_sample_sharpe DESC) AS in_sample_rank
FROM scored
)
SELECT multiIf(in_sample_rank <= 48, 'best fifth in training',
in_sample_rank <= 96, 'second fifth',
in_sample_rank <= 144, 'middle fifth',
in_sample_rank <= 192, 'fourth fifth',
'worst fifth in training') AS training_group,
round(avg(in_sample_sharpe), 2) AS avg_in_sample_sharpe,
round(avg(out_of_sample_sharpe), 2) AS avg_out_of_sample_sharpe
FROM ranked
GROUP BY training_group
ORDER BY min(in_sample_rank)For training, groups run from 0.66 for top to -0.63 for bottom, wide and perfectly orderly ladder, because dem cut the groups with that same score. For holdout, the same two ends average 0.01 and 0.13. The ladder flatten. Holdout na the only part of pipeline wey nobody optimize against, and na why e worth spending carefully.
Defenses wey really work
Holdout wey you spend once. Every time you look at am, you turn am into training data. Walk-forward testing, where window dey roll and each score come from data after the fit, na the version wey survive repeated use.
Multiple-testing adjustment. Deflated Sharpe ratio, wey Bailey and López de Prado introduce for 2014, dey discount observed Sharpe based on how many trials dem run, how long sample be, how skewed returns be, and how fat their tails be. Give am honest trial count, and headline Sharpe from ten-thousand-expression search often deflate to nothing.
Audit trail wey cover every expression dem try, including the ones dem throw away. This one carry the main weight, and na why "auditable" na the interesting word for factor-research project description. Deflation need trial count. Pipeline wey log winners only don destroy input to its own correction. Discarded drafts, abandoned parameter sweeps, researcher's own restarts, and every earlier version of scoring code all count toward that number.
Cost and look-ahead bias checks before you believe the score. Factor ranked with fundamentals field stamped with date vendor load am, instead of date market fit see am, go backtest beautifully and trade badly.
How to read the word "auditable"
New repositories for this area dey appear almost every week, and project with few dozen stars na prototype, not track record. Star counts sef dey move faster than code, na why this page score the pattern instead of one project. This na wetin you suppose open first for whichever one land before you.
- E dey log every candidate with expression and score, plus timestamp, or only the keepers?
- Harness dey enforce holdout, or na researcher's self-discipline?
- Any reported score carry trial count beside am?
- Which market dem build am for? Library tuned for China A-shares inherit daily price limits and restriction on selling shares bought that same session, and factor behavior under those rules no dey carry over to US equities.
- You fit re-run am and reproduce the numbers? Pin exact commit wey you read, because project for this stage fit rewrite scoring code between weekends.
None of this make LLM useless for factor research. Generating hypotheses na real bottleneck and models good at am. The thing wey change na where burden fall: onto accounting for how many hypotheses dem burn. Before anything meet live order book, paper trading na where distance between backtest and fill become visible.
LLM alpha factor FAQ
LLM fit find alpha factors?
E fit propose dem by thousands, but proposal no be finding. The claim dey happen for scoring step, and score wey come from wide search carry selection problem wey score itself no fit see. Judge holdout discipline and recorded trial count before you judge expression.
Wetin be deflated Sharpe ratio?
Na correction wey turn observed Sharpe ratio into probability say search of that size for produce am even when no real edge dey. Bailey and López de Prado publish am for 2014. Its main input na number of trials, exactly the number wey unaudited research loop no fit provide.
How many backtests too many?
No threshold dey; na adjustment you must apply. One backtest wey score 1.0 and ten thousand backtests whose best scores 1.0 na different claims about the world. The coin flips above reach 1.59 across 240 trials, with nothing inside the data at all.
Why published factors dey weaken after publication?
Academic work don track how published anomalies decay during the years after dem appear. Crowding na one mechanism people dey give, and original result wey overfit its own sample na another; both draw the same shape for chart. The efficient market hypothesis frame the first one, and trials above demonstrate the second.
Every panel here na stored query over real month-end prices, with SQL open underneath. Copy one, increase the trial count, and watch the best number climb for the Strasmore terminal.