Security agents need patch acceptance metrics
OpenAI's GPT-5.5-Cyber found 24 kernel privilege-escalation exploits across 30M+ lines. Impressive — and the wrong scoreboard. Judge a security agent by validated fixes accepted upstream, not exploits generated.
OpenAI expanded its Daybreak security program this week with Patch the Planet, and the headline numbers are designed to travel: GPT-5.5-Cyber worked across more than 30 million lines of Linux kernel code and produced 24 local privilege-escalation exploits plus 8 kernel pointer-leak proofs-of-concept. On the ExploitGym benchmark, the specialized model scores 39.5% against 25.95% for base GPT-5.5. The newsletter version of this story is "AI found 24 kernel exploits", and that version will define how a thousand security programs think about agentic security tools.
Which is a problem, because exploit counts are the supply side of security, and supply was never the constraint.
Every security leader already lives this. The scanner era taught the lesson thoroughly: tools that produce findings are abundant; what drowns a program is the pipeline after the finding. Raw discovery counts mix real signal with duplicates of known issues, low-impact corner cases, findings that sit in disclosure backlog for months, and — the expensive category — issues that are technically valid but that no maintainer will ever accept a patch for, because the fix breaks something they value more. A wall of exploit trophies tells you the tool can break locks. Your risk went down only if a door somewhere actually got a better lock.
To OpenAI's credit, the program's structure understands this better than its headline does. Patch the Planet pairs the model with human engineers who review findings before maintainers see them, and the early sprint reporting emphasizes merged patches across participating projects — with cURL, Python, and Go among those signed on. That is the right pipeline. But the honest gap remains: I could not determine how many of those 24 kernel exploit proofs have become accepted upstream fixes, because disclosure is still in progress. Until that number exists, the 24 is a capability demo, not a security outcome.
So here is the scorecard I would hold any security agent to — vendor-supplied or homegrown. Five stages, each strictly harder than the last: validated issue (deduplicated, triaged as real and material), coordinated disclosure (reported through the project's actual process), accepted patch (a maintainer merged a fix — the stage where most trophy walls go quiet), time to merge (because a fix that lands in fourteen months protected nobody for fourteen months), and post-fix regression rate (did the patch hold, or did it break something and get reverted). Discovery counts appear nowhere on that card, and that is deliberate: discovery is the cheapest stage and the only one a model can inflate unilaterally.
At work, the version of this I now see in procurement: a security leader shown a vendor deck with a four-digit vulnerability count, and no way to see duplicates, acceptance, remediation time, or regressions behind it. The question I hand them is one line: of these findings, how many produced merged fixes, and what was the median time from report to merge? Vendors with a real pipeline answer with two numbers. Vendors with a trophy wall answer with an architecture slide.
Steal this framing for your own program too, internal tools included: pay bounties, bonuses, and renewals on stage-three-and-beyond events — accepted patches — never on finding counts. Incentives aimed at discovery buy you a backlog. Incentives aimed at acceptance buy you a smaller attack surface, which was the point all along.
An exploit proves the lock can be picked; security is the new lock, installed and holding — score your agents on doors fixed, not doors opened.
