Task cost is a harness property, not a model price

Same task, three prices. On a LangChain podcast carried in their July 23 newsletter, Factory CTO Eno Reyes ran a fixed code-review task and read out the bill: roughly $1.70 on GPT-5.5 inside Factory's harness, $5–6 on Opus 4.8 in a different harness, and about $3 when Opus 4.8 ran through Factory's own review workflow. The model swap moved the price less than the harness swap did. On Factory's cybersecurity benchmark the ranking flips entirely: GLM 5.2 tests strongest for the task but gets held back by its own harness assumptions.

Provenance first, because these numbers deserve their asterisk. Reyes co-founded Factory — a $1.5 billion company per the newsletter, a figure I could not verify — and he was demoing his own harness on a podcast run by LangChain, itself a harness vendor. I took the figures from LangChain's written summary, not a fetched transcript. Every number here is a vendor's claim about the vendor's product. Hold it accordingly.

But the direction of the claim matches what I see in production, and that is what makes it worth writing up.

The pricing page prices the wrong thing

The consensus way to compare models is dollars per million tokens. That works for a single completion. It collapses for agent tasks, because what a task costs is set by everything wrapped around the model: how much context gets assembled per step, how many retries the error policy allows, whether there's a review loop, how aggressively the harness caches and compacts. Comparing models by pricing page is comparing engines by fuel price while ignoring the drivetrain.

At work, every client cost model I have reviewed this year does the same arithmetic: token price times estimated volume, one row per model. Not one of them had a row for the harness. One client's projection was off by well over 2x against the first month's actual bill — not because the model was priced wrong, but because nobody had priced the retry policy and the context assembly that the orchestration layer added around every call. The pricing page was accurate. The forecast built on it was fiction.

Who gets to publish the honest benchmark

Here is the claim I'll defend, and it's the part nobody in that podcast said out loud: labs are structurally unable to publish honest task-level cost benchmarks, because their harnesses are tied to their own models — which quietly makes model-agnostic harness vendors the pricing authorities of the agent economy. Reyes gestured at this with his line about labs testing on blinders, but the consequence is bigger than a testing complaint. If cost-per-task is the number that matters and only cross-model harnesses can measure it fairly, then the entity buyers will end up trusting for price discovery is not OpenAI or Anthropic — it's whoever runs the neutral drivetrain. That's a strange amount of market power to hand to workflow companies, and nobody is treating them as the referees they're becoming. Referees with their own products in the race, mind you — which is exactly why the asterisk above matters.

Steal this before your next model decision: pick one real task from your backlog — a code review, a document triage, whatever you actually run. Execute it end to end through two different harnesses with the same model, and through two models in the same harness. Four cells, real bills, an afternoon's work. If the harness axis moves the cost more than the model axis — and Reyes's numbers say it can, by around 3x — then your model-comparison spreadsheet is measuring the smaller variable, and your negotiation leverage is in the wrong meeting.

Per-token price is what the vendor controls. Per-task cost is what you control. Confusing the two is how AI budgets die.

Benchmark the task, not the token — the harness sets your bill, and the pricing page never met your harness.