Perfect accuracy cannot reveal the algorithm
Four compiled transformers returned every supported calculator answer correctly through different mechanisms. Accuracy can certify outputs; it cannot identify the algorithm or predict how it scales.
On August 9 Rob Porter published four transformer calculators. He reports exhaustively checking three across 3,000,000 supported expressions each and a fourth across 30,000. Every answer was correct.
The four checkpoints did not share an algorithm. One propagated grade-school carries through layers. One used hardware-style trees. One moved serial work into generated scratchpad tokens. One did no arithmetic at all; it retrieved a memorised answer from a finite table.
That is the cleanest answer I know to a common evaluation mistake: accuracy can certify the outputs observed in a tested domain. Even perfect accuracy cannot identify the procedure that produced them, predict its resource curve, or tell you what happens one step outside that domain. A score is an output receipt, not an algorithm receipt.
Four routes into one perfect column
Torchwright compiles restricted Python computation graphs directly into ordinary Phi-3 transformer weights. There is no training step, and the published three-digit grade-school checkpoint loads through the normal Hugging Face pipeline. That construction makes the hidden tradeoffs unusually visible.
The grade-school and hardware versions return the same answers, but at ten digits Porter's compiler produces 43 layers for the first and 36 for the second. The scratchpad construction holds at 18 layers through seven-digit operands by spending generated tokens instead. Add one digit to each operand of the memorising version and its table grows 100-fold.
Those are not cosmetic implementation details. They decide latency, token cost, parameter growth and the first failure you will meet when the problem gets larger. Yet an output-only benchmark collapses all four into the same cell: 100%.
Keep the evidence boundary tight. These exhaustive checks are Porter's reported results; I did not rerun the public checkpoints or his 17,290 frontier-model attempts. The domains are fixed and the frontier comparison used a different prompt grammar, which Porter explicitly calls not like-for-like. The construction proves mechanism blindness inside a score, not general-purpose mathematical intelligence.
The mirror image lives in the harness
The same weekend produced the opposite warning. Nvidia reported Claude Opus 5 moving from roughly 30% bare to 100.00 inside its AVO harness on ARC-AGI-3's 183-level public set. Same named model, different surrounding system, very different score—and Nvidia cautions that the comparison is not a controlled ablation.
Porter's calculators show different machinery, same score. AVO supplies the complementary warning: keeping the model name constant does not keep the evaluated system constant, and the score can move. Put them together and a leaderboard number is visibly under-specified: it identifies neither the weights nor the system that earned it.
The short below supports the harness argument, not the calculator claim. Its rendered audio, frames and Nvidia's dated source make that distinction clear. Keeping those two claims separate is exactly the discipline an evaluation review needs.
Put a second row under accuracy
For a model bake-off, keep the accuracy row. Add four mechanism checks beside it:
- Scale: extend length, context and workload until the resource curves separate.
- Intervene: remove scratchpads, memory, retries or tools one at a time.
- Trace: record tokens, tool calls, latency and provenance rather than inferring process from the answer.
- Cross the boundary: change the grammar or distribution just beyond the evaluated domain.
This will not magically recover an unknown algorithm from black-box outputs. It does something more useful for a buyer: it distinguishes reusable procedure from finite coverage, and it reveals which dependency is carrying the result. That extends the earlier rule that leaderboards cannot see the harness. A tied score is the start of due diligence, not the end.
Accept accuracy as evidence that the tested answers were right; demand scaling, intervention and provenance evidence before deciding what learned to produce them.
