Agent backtests are demos, not track records
JPMorgan's eight AI agents beat 60/40 by 0.7pts across 20 years of backtests. The catch nobody in the amplification layer mentions: an LLM was trained on those same decades, so the answer key is partly in its weights.
JPMorgan disclosed that eight AI agents — built on OpenAI and Anthropic models to classify markets into four regimes (Goldilocks, reflation, stagflation, risk-off) and shift stock-bond allocation — beat the classic 60/40 portfolio by about 0.7 percentage points of annualized return, at lower volatility, across roughly two decades of backtests. Every one of the eight beat 60/40 on a risk-adjusted basis, and they also beat the bank's own rules-based regime model. In asset management, 70 basis points of consistent outperformance is genuinely a lot, and the finance-newsletter circuit spent the following week saying so.
To JPMorgan's real credit, the bank labeled these backtests, not live performance, and explicitly warned against reading them as proof AI can consistently beat markets. That caveat is the most important sentence in the disclosure, and it is precisely the sentence the amplification layer dropped — one widely-shared briefing rendered it as "AI Investors Beat Traditional Portfolios." So let me sharpen the caveat into the reason it matters, because it generalizes far beyond finance.
A 20-year backtest is the one arena an LLM-based agent can pass partly from memory. The models classifying whether 2008 was "risk-off" or 2020 was "reflation" were trained on text written about 2008 and 2020 — after the fact, with the outcomes known. The regime label for a historical period, and what worked in it, is effectively in the weights. This is look-ahead leakage, and the unsettling part is that it requires nobody to cheat: the contamination arrives through pretraining, silently, in a test where the "predictions" cover years the model has already read the history of. A human analyst backtesting a rule doesn't know the future at each step; an LLM has, in a real sense, already seen the answer key to the whole exam.
Notice where this bites hardest, because it is the counterintuitive part. The most impressive-sounding result — the agents beating JPMorgan's own rules-based model — is also the most contaminated comparison. The rules model has no memory of the test period; it applies a fixed formula blind. The LLM agents plausibly do have memory of it. So "the LLM beat the rules engine" may be measuring, in part, "the system that read the history beat the system that didn't." That is not a track record. It is a demo of a very specific capability: reconstructing known regimes from training data.
None of this means the work is worthless or dishonest — JPMorgan's framing is careful, and regime-aware allocation may well add value live. It means the backtest cannot tell you whether it will, because the one thing a backtest must guarantee — that the model didn't know the future — is the one thing an LLM trained on the past cannot guarantee.
And this is now everyone's problem, not just finance's. With surveys putting a majority of banks piloting agents, every enterprise AI pitch increasingly arrives wearing a backtest-shaped eval: "our agent would have caught this fraud," "would have flagged this outage," "would have won this deal" — evaluated on historical cases the model may have trained on. The finance example is just the most legible instance of a trap that's spreading.
So my first diligence question for any historical agent eval, in any domain: does the model's pretraining window overlap the test window? If yes, the comparison is compromised until proven otherwise, and the burden is on the vendor to show they controlled for it — held-out data after the training cutoff, or a decontamination method with receipts. "It beat the baseline on twenty years of history" is a sentence that should now trigger the question, not end it.
Steal this for your next agent eval review: demand at least one evaluation on data created after the model's training cutoff — genuinely unseen — and weight it above any historical backtest, however impressive. A model that performs on truly out-of-sample data has shown you something. A model that performs on its own training era has shown you its memory.
A backtest an LLM can pass from memory is a reading comprehension test, not a track record — grade the agent on the future it hasn't seen, not the past it was trained on.
