# AndyMental — full published corpus

Author: Anand “Andy” Padia
Contact: meet@andymental.com
About: https://andymental.com/about

Only content that passes the site's publication gates is included below.

## Agent harnesses need maps, not more manuals

- URL: https://andymental.com/drops/agent-harnesses-need-maps-not-more-manuals
- Type: post
- Published: 2026-07-27
- Updated: 2026-07-28

> Self-evolving agents fail at locating where to edit before they fail at writing the edit (Harness Handbook, arXiv Jul 14). The scarce artifact is a source-verified map — not a fatter instruction manual.

We keep trying to build agents that rewrite their own harnesses — the scaffolding of skills, tools, and routing logic wrapped around the model. The part that looks hard is *generating* the edit. A paper out this month says that is the wrong part to worry about.

"Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable" (Ruhan Wang and colleagues, arXiv 2607.13285, July 14) builds a behavior-to-source map straight from a harness codebase — static analysis plus LLM structuring — that links each *behavior* to the *code that produces it*. Then a step called Behavior-Guided Progressive Disclosure walks an agent from a high-level behavior down to candidate implementation sites, and verifies each candidate against the current source before anything is edited. Across modification requests on two open-source harnesses, planning with the handbook improved where-to-edit accuracy and edit-plan quality while spending *fewer* planner tokens. The largest gains were on scattered sites, rarely-run paths, and cross-module behavior.

Read the consensus way, this is "nicer tooling for self-improving agents." Read it the way that bites: **the agent was failing at localization before it ever failed at generation.** It knew how to write the fix. It did not know which of forty files owned the thing it was fixing.

That lands close to home, and I have to correct my own reflex out loud. My instinct — and I'd bet most teams' — has been to make agents more reliable by *stuffing the manual*: a fatter CLAUDE.md, more skill descriptions, more "when you see X, do Y." The Harness Handbook does the opposite. It adds almost no instructions. It adds a *map*, and it makes every claim in that map provable against source before an edit lands. Documentation for an agent, it turns out, is not prose you write. It is a navigable, source-verified index you *regenerate* — a build artifact, not a file you lovingly hand-maintain.

Here is the rule I am taking from it, and it is the thing the paper does not say. **Instruction context depreciates as models get better; navigational context appreciates as your codebase gets bigger.** A smarter model needs fewer "do it like this" rules — it already knows. But no model, however capable, can know that *your* rate-limiter lives across three files and one of them is a decorator nobody remembers. We keep adding navigational facts as if they were instructions, and instructions as if they were permanent — which is exactly why CLAUDE.md files bloat into swamps nobody can read.

![Two crossing curves: instruction value declines as models improve, while navigational value rises as the codebase grows; past the crossover, the map beats the manual.](/api/media/file/agent-harnesses-need-maps-not-more-manuals-curves.svg)

I felt this one directly. At Trigent we wired a self-editing agent into a client's support-automation stack — a dozen skill files, the usual accretion. Ask "which file owns the escalation-to-human behavior?" and the honest answer was `grep` plus two engineers' memory. It went fine until the agent confidently edited the file that *looked* like it owned retries and did not — a plausible, well-named, wrong location. The change passed review because it read correctly in isolation, then broke a rarely-hit timeout path in production a week later. That is precisely the failure the verify-against-source step is built to kill: not a bad edit, a correctly-written edit in the wrong place.

So the implication if you run a production agent stack: before you add one more line to your instructions file, ask whether the reliability you want is an instruction problem or a map problem. If your agents keep getting lost — "which skill owns this?", "where does this behavior actually live?" — no manual saves them.

The scar is cheap to avoid. Point an LLM at your harness this week and have it emit one artifact: behavior → the file, function, and line that produces it, each link confirmed to still exist. Not prose — a checked index. That single afternoon buys more edit safety than a month of instruction-tuning.

## What's in it for you

I share these mid-experiment so you can skip the weekend I spent learning them. From this one:

- **A faster diagnosis.** Next time an agent edits the wrong file, suspect localization first — and stop hardening prompts that were never the problem.
- **A checked map, not more docs.** One regenerated artifact — behavior to file to line, verified against source — that catches wrong-location edits before your review does.
- **A line to hold.** "Navigational context appreciates, instruction context depreciates" is your argument for shrinking the CLAUDE.md instead of feeding it. Steal it, try it, and tell me where it breaks.

**Stop writing your agents a longer manual and start handing them a verified map — localization is the failure, and a map you regenerate beats a manual you maintain.**

---

## Temperature zero is not a determinism guarantee

- URL: https://andymental.com/drops/temperature-zero-is-not-a-determinism-guarantee
- Type: post
- Published: 2026-07-26
- Updated: 2026-07-28

> A weekend tutorial promises "same verdict every time" from temperature 0. Thinking Machines got 80 distinct outputs from 1,000 temp-0 runs. Temp 0 is a sampling setting; the only deterministic layer is the cache.

There is a genre of AI-build tutorial that closes with a reassuring line: set temperature to 0, and the same input always produces the same output. A guest tutorial doing the rounds this weekend — a pitch red-flag scorer wiring Gemini 2.5 Flash, Tavily, and Supabase together — sells exactly that: temperature 0 means "same verdict, every time." It is a lovely promise. It is also false, and the numbers on how false have been public for ten months.

Horace He and colleagues at Thinking Machines Lab ran the experiment last September: 1,000 completions at temperature 0 from one model (Qwen3-235B), one prompt. They got **80 distinct outputs.** The runs stayed identical through token 102 — "Feynman was born on May 11, 1918, in" — then forked: 992 continued "Queens, New York," eight broke to "New York City." Temperature 0 did its job perfectly. The answer still split eighty ways.

The consensus reading of temperature 0 is "determinism." That is a category error. **Temperature 0 is a sampling setting — it tells the model to take its highest-probability token instead of rolling dice. It says nothing about whether the probabilities themselves come out identical run to run.** On a hosted API, they don't. He's team traced the cause exactly: server load changes the batch size your request lands in, the inference kernels are not batch-invariant, so the same request walks a different numeric path and can round to a different winning token. You didn't touch your prompt. Someone else's traffic changed your answer.

Here is the claim I'll put my name on. **In these stacks the model is the one component that never promises consistency — the only deterministic layer is the cache in front of it.** The tutorial contains the honest version of its own promise and doesn't notice it: the Supabase cache. A cached verdict genuinely is the same every time, because it isn't calling the model — it's returning a row. The rulebook and the fixed output schema narrow variance too. But "same verdict every time" is a property of the cache, the rulebook, and the schema working *around* the model — not something you bought by typing `temperature=0`.

![1,000 temperature-0 runs stay identical through token 102, then fork into 80 distinct outputs; the cache lane below returns the same row every time.](/api/media/file/temperature-zero-is-not-a-determinism-guarantee-forks.svg)

I have watched this bug get misdiagnosed. At Trigent we had a client eval harness that pinned temperature to 0 and still showed run-to-run drift on a fixed prompt set — a handful of cases flipping pass/fail between nightly runs. The reflex was to blame the prompts: the eval set is flaky, tighten the wording. We spent real hours hardening prompts that were fine. The drift was the serving batch, exactly as He describes — nothing in our harness had changed, only the load on the endpoint at 2am versus 2pm. Pinning temperature felt like control. It was theatre.

One thing the tutorials skip, and it's the other half of the honesty: if your consistency comes from the cache, your correctness now rides on cache freshness. That pitch-scorer does live "reality checks" — pay ranges, market facts — then caches the result. With no TTL, it will re-serve last week's number as this week's research, deterministically wrong. Determinism and staleness are the same coin: the property that makes the cache consistent is the property that lets it lie.

So the implication if you ship an LLM feature that must be reproducible: stop treating `temperature=0` as a guarantee and start architecting the consistency. If you truly need identical outputs — a scoring rubric, a compliance check — put a cache with an explicit key in front of the model and a TTL matched to how fast the underlying truth moves. If you need to *prove* end-to-end determinism, batch-invariant kernels exist but cost roughly 1.6x–2.1x on inference. Determinism is a bill, not a flag.

## What's in it for you

I ship these while I'm still in the mess of them, so the finding reaches you cheap. From this one:

- **A bug you'll stop misdiagnosing.** Run-to-run eval drift at temperature 0 is the serving batch, not your prompts — so you can quit rewording prompt sets that were already fine.
- **A test you can run today.** Fire one request a few hundred times and diff the outputs; where they come back identical is where your real determinism actually lives.
- **A design rule.** If you truly need reproducible outputs, architect it — a cache with an explicit key and a TTL — instead of trusting a sampling flag to hand you a guarantee it never made.

**Temperature 0 buys the model's most confident answer, not the same answer twice — put the determinism in a cache with a TTL, because the model was never going to hand it to you.**

---

## Vulnerability counts need a CVE ledger

- URL: https://andymental.com/drops/vuln-counts-need-a-cve-ledger
- Type: post
- Published: 2026-07-25
- Updated: 2026-07-26

> Anthropic's Project Glasswing claims Mythos found 10,000+ high-severity vulnerabilities. An audit found one CVE explicitly attributed to it. The gap isn't fraud — it's a claim versus the public ledger that checks it.

The headline number is genuinely large: Anthropic says Project Glasswing partners used Claude Mythos to find more than 10,000 high- or critical-severity vulnerabilities across major operating systems and browsers, with roughly 50 partner organizations involved. A security audit of the public record, though, counts the vulnerabilities *explicitly attributed to Glasswing itself* in the CVE ledger at a strikingly smaller figure — as of the accounting I read, essentially one, CVE-2026-4747, a FreeBSD NFS remote-code-execution flaw (alongside a few dozen more attributed to Anthropic researchers broadly). Ten thousand claimed. A tiny number in the ledger. That gap is the whole post.

First, the fair part, because the gap is *not* evidence of fraud and I won't pretend it is. Coordinated disclosure lags discovery by design — several of the headline finds (a decades-old OpenBSD flaw, a long-lived FFmpeg bug, kernel privilege-escalation chains) reportedly sit under embargo while patches are prepared, and responsible researchers don't publish CVEs before fixes ship. The capability also looks real: independent assessment of a sample put the true-positive rate high, and exploit-generation success jumped from near-zero to meaningfully high. Something genuine is happening. A full public accounting is reportedly due shortly, and the right posture is to update against it.

But hold the two numbers side by side, because they *measure different things and only one is checkable.* "We found 10,000 vulnerabilities" is a first-party claim about internal counts — pre-triage, pre-dedup, pre-rejection, pre-disclosure. The CVE ledger is the public, adversarial, cross-checked record of vulnerabilities that have been validated, coordinated, and (usually) patched. The first number can include duplicates, false positives, low-impact findings, and issues no maintainer will ever accept. The second cannot — that's what the ledger is *for*. So the honest read of the gap is not "the capability is fake." It's "the 10,000 is a lab metric and the ledger is the audited one, and until they converge, only the ledger tells you what actually reduced risk."

This is the same distinction I keep coming back to on security tooling: discovery counts are the cheap supply side; validated, accepted, patched fixes are the constraint. Glasswing is just the largest, most-cited instance, which makes it the perfect case study for a discipline every security buyer now needs — because the "our model found N vulnerabilities" slide is about to appear in every vendor pitch, and N will always be the impressive pre-triage number.

The deployable check is one question: **which N have CVE numbers, embargo records, or patched releases attached?** Not "how many did the model find" — "how many made it to the ledger, or are on a dated, credible path to it." A vendor with a real pipeline answers with a number and a link. A vendor with a trophy count answers with a methodology slide and a promise. And notice the failure mode isn't even the vendor lying — it's the *amplification layer*: I watched one newsletter relay the 10,000 figure without the one-CVE audit that its own source publication had run separately. The claim travels; the accounting doesn't. Your job as the reader is to demand the accounting the retweet dropped.

At work, this is now my first response to any "AI found N flaws" claim, ours or a vendor's: *show me the ledger slice.* CVEs assigned, disclosures coordinated, patches shipped or dated. Everything above that line is capability demonstration — real, interesting, worth watching — but it is not risk reduced, and it should not be priced or reported as if it were.

Steal this for your next security-vendor review: when the vulnerability-count slide appears, ask for the same figure filtered to CVE-assigned-or-embargoed-with-a-date, and weight *only* that number in the decision. Then do the honest thing the vendor should model — commit to revisiting when the full public accounting lands, and actually change your view if the ledger catches up to the claim. It might. The point isn't that Glasswing is hollow; it's that you can't yet know, and "10,000" isn't allowed to stand in for knowing.

**A vulnerability isn't found until it's in the ledger — count CVEs and patches, not model outputs, and make the amplifiers show the accounting they skipped.**

---

## An AI detector you can switch off measures cooperation

- URL: https://andymental.com/drops/optional-ai-detection-measures-cooperation
- Type: post
- Published: 2026-07-25
- Updated: 2026-07-26

> Substack added reader-facing AI detection via Pangram — but authors can disable it per post, and disabling shows readers "AI detection unavailable." So it doesn't measure AI. It measures whether you cooperated.

Substack rolled out reader-facing AI detection this week, powered by Pangram: readers can scan a post, note, or comment over 100 words and see an AI-likelihood read. CEO Chris Best framed the enemy as "Claudefishing" — readers assuming a human wrote what no human wrote — and the goal as not becoming "like LinkedIn." I write an openly AI-assembled publication, so I read this launch as a direct signal about my own corner of the internet, and the design is more interesting than the feature.

Three details define what this actually is. It only scans content published since launch — the entire back catalog is permanently unscored. Authors can **disable the scan per post**. And when they do, readers don't see nothing — they see an explicit "**AI detection unavailable**" message. Hold those together and the conclusion is unavoidable: this is not a provenance layer. **It is a cooperation signal.**

Because think about what an author who wants to hide AI use does with a per-post off switch: they flip it. The scan never runs, and the reader gets "detection unavailable." So the detector never catches the person it's ostensibly for. What it *does* create is a new, legible thing on the page — the presence or absence of a scan — and readers will learn to read the absence. "AI detection unavailable" on a post that could have been scanned is going to read, fairly or not, as *what were you hiding?* The off switch was meant to protect nuance; it instead manufactures a scarlet letter for anyone who touches it.

That is the trap, and it's worth stating plainly because the incentives are perverse. The scan is optional, but *opting out is itself a disclosure* — and a worse one than any percentage. A "40% AI-assisted" badge at least invites a conversation about how the piece was made. "Detection unavailable" invites suspicion with no conversation at all. The feature converts a hard technical question ("was this AI?") into a cheap social one ("did they cooperate?"), and the social one is the one that will actually move readers.

The other design flaw compounds it. Pangram outputs a *percentage* — an estimate of how much was AI-assisted — into a discourse that treats any nonzero number as disqualifying. One prominent engineer's reaction to AI-flagged writing was simply "I just don't take time to read it." A percentage implies a spectrum of legitimate practice; the audience reads it as a binary verdict. So the writer using AI ethically and disclosing it honestly gets the same social penalty as the writer faking humanity wholesale — which is exactly the nuance the counter-critics warned would get flattened. And note the irony Substack is standing in: its own writer roster includes a named plaintiff from the Anthropic copyright settlement. The platform is refereeing a fight its own authors are litigating.

So what does someone running an AI-assisted publication actually do? Not chase the percentage — that's a losing game against a spectrum-flattening audience. The durable answer is to **make the process statement do the work the scan can't**: a standing, visible "how this is made" note that owns the AI assistance plainly, paired with the thing detection can never verify — human-checked *claims*. Readers who've been Claudefished aren't actually angry about the tool; they're angry about being deceived and about being wrong. A clear process disclosure defuses the deception, and rigorous fact-checking defuses the wrongness. That combination beats any detection badge, because it addresses the real injury instead of the proxy.

Steal this if you publish with AI in the loop: write one permanent process statement and link it from every piece, and never disable a scan you could have passed — because in a world where "detection unavailable" is a visible label, the off switch is the confession. Cooperate loudly, disclose plainly, and verify your facts by hand. The scan measures cooperation; give it something to measure.

**Substack didn't build an AI detector — it built a cooperation meter with an off switch that reads as guilt. Win it by disclosing your process and checking your facts, not by gaming a percentage.**

---

## Less restrictive is now a model spec

- URL: https://andymental.com/drops/less-restrictive-is-now-a-model-spec
- Type: blog
- Published: 2026-07-25
- Updated: 2026-07-26

> Anthropic's Opus 5 is cheaper, engages safety classifiers 85% less than Fable 5, and can silently reroute blocked prompts to weaker models. Restriction level just became a product axis you have to audit.

Anthropic launched Opus 5 this week, and past the usual cheaper-and-faster headlines is a change in how models are *specified* that enterprise procurement and audit teams need to sit up for. Per the launch coverage: Opus 5 is smaller and cheaper than Fable 5, outperforms it on several benchmarks, and engages its safety classifiers about **85% less often** — deliberately, because Anthropic chose not to give it cutting-edge cybersecurity capability, so it carries a lower risk profile and needs fewer guardrails. It also ships a beta feature, **Automatic Fallbacks**, that reroutes a request which trips a safety classifier to a less powerful model instead of returning an error. (Pricing and exact figures come from secondary coverage; treat the numbers as directional.)

Read those two facts together and a line has quietly been crossed. **Restriction level is now a purchasable product axis** — sitting alongside price and capability as a thing you spec, not a fixed property of "the model." The Claude 5 lineup now tiers partly by *permission*: a broadly-sold tier with classifier routing on risky queries, a cheaper tier with far less classifier engagement and different data-retention terms, and a gated tier behind a consortium with mandatory retention. Capability, retention policy, and refusal behavior now vary **independently**. Any enterprise whose model policy reads "use the most capable tier" just discovered that sentence no longer specifies a single thing.

## Three axes where there used to be one

The old procurement question was one-dimensional: how capable, at what price. The new lineup forces three questions that don't move together.

**Capability** — the benchmark axis everyone already tracks. **Retention** — whether prompts and outputs are held, and for how long, which is a compliance and data-residency question entirely separate from how smart the model is. And **restriction** — how often the model refuses or routes around a request, which is now tunable and priced. A team that specs only the first axis and inherits the other two by accident has made two consequential decisions without noticing: it may have picked a retention posture its regulator cares about, and a refusal-behavior profile its risk team never reviewed.

The restriction axis also isn't a simple dial from "safe" to "permissive." Opus 5's guardrails reportedly draw lines *inside* a single task family — blocking, say, scanning compiled binaries for vulnerabilities while permitting source-code vulnerability search. So "less restrictive" doesn't mean "unrestricted"; it means a *different shape* of restriction that your security use cases have to be checked against specifically. You cannot infer the boundary from the marketing adjective.

## Automatic Fallbacks makes refusals invisible

Here is the part that changes audit, and it's easy to miss. Historically a safety refusal was *legible*: you asked, the model declined, you got an error, everyone knew a boundary was hit. Automatic Fallbacks dissolves that. A blocked request no longer surfaces as a block — it silently gets answered by a weaker model, and the caller receives a functional response. Convenient. Also a provenance problem.

Because the audit question just shifted. It used to be "**what can the model do?**" It is now "**which model actually answered this request?**" When a compliance reviewer pulls a transcript, "we used Opus 5" may not be true for the requests that tripped a classifier and fell back — those were answered by something else, quite possibly a less capable model, on the requests most likely to be sensitive. If the fallback isn't disclosed in the API response — and I couldn't confirm whether it is — then your logs say Opus 5 for answers Opus 5 didn't produce.

```mermaid
flowchart TD
    R["Request"] --> C{"Trips safety classifier?"}
    C -->|no| M["Opus 5 answers"]
    C -->|"yes, old world"| E["Error — refusal is legible"]
    C -->|"yes, Automatic Fallbacks"| F["Weaker model answers silently"]
    M --> L["Log says Opus 5"]
    F --> L2["Log says Opus 5 — but it wasn't"]
    L2 --> A["Audit gap: which model actually answered?"]
```

## What procurement and audit have to do now

At work, this reframes the model-selection conversation I'm in constantly. "Use the best model" is no longer a policy; it's an underspecified wish. The policy now needs three named choices per workload — capability tier, retention posture, restriction profile — each justified against that workload's actual requirements, because a coding-agent's needs on all three differ from a customer-facing assistant's. And it needs a rule on fallbacks: for regulated or sensitive workloads, I'd default to fallbacks *off*, precisely because a legible refusal you can handle is safer than a silent downgrade you can't see. An error is a known state. A quietly-rerouted answer on a flagged request is an unknown one, on exactly the requests where you least want unknowns.

The audit requirement follows directly: **the answering model must be recorded per request**, not assumed from the configured default. If your logging captures "model: opus-5" from config rather than from the response's actual provenance, it will confidently lie about every fallback. Capture what answered, not what you asked for.

Steal this checklist for the new lineup: spec every workload on three axes (capability, retention, restriction), not one; decide fallbacks on/off per workload and default them off where refusals are compliance-relevant; and verify your logs record the *responding* model's identity, then test it by sending a request you expect to trip a classifier and confirming the log names what actually answered. Restriction became a product; treat it like one you have to audit.

**"Less restrictive" is now a line item, and a silent fallback is an unlogged model swap — spec restriction like retention, and record which model actually answered, not which one you asked for.**

---

## Agent ROI is often feature activation in disguise

- URL: https://andymental.com/drops/agent-roi-is-often-feature-activation
- Type: post
- Published: 2026-07-25
- Updated: 2026-07-26

> SaaStr's finance agent "fixed" collections by switching on bill.com auto-reminders the team never activated in 8 years. A real win — but configuration debt, not intelligence. Separate activation wins from reasoning wins.

SaaStr's latest agent dispatch has a great collections story: three humans and twenty-plus agents in production, six figures behind on receivables during their conference, and their finance agent goes into bill.com, sets up dunning, and gets collections back on autopilot. The end-to-end flow around it is genuinely impressive — seconds after a signature, the agent reads the contract, flips the Salesforce opportunity, appends missing signer contacts, creates split invoices with correct terms. Real work, well integrated. First-party and unaudited, as ever, but I believe the shape.

Then read what the collections win *actually was*, because SaaStr is honest enough to tell you: the agent discovered that bill.com had **built-in automated reminders the team had never turned on in eight years** — reminders before the due date, escalation after, all behind one toggle. The companion episode repeats the pattern with "a setting we'd missed for 8 years." The agent's edge here was not reasoning. It was reading the manual exhaustively and flipping a switch a human could have flipped on day one.

I want to be clear this is a real win — money got collected, and "the tool you already pay for, finally configured" is legitimate value. But it is a specific *kind* of value, and mislabeling it is about to distort a lot of ROI math. **A measurable share of reported agent ROI is feature activation in disguise:** the agent switching on paid-for SaaS capabilities the humans never configured. When that gets booked as "AI ROI," you are pricing configuration debt as artificial intelligence — and paying a premium for a machine that read a settings page.

The distinction that matters is between two categories of win, because they have completely different economics. **Activation wins** are one-time and cheap: a capability that already existed, now switched on. Enormous first-time payoff, then it's done — the reminders don't get re-discovered next quarter. **Reasoning wins** are durable and genuinely priceable: the agent handling a novel case, exercising judgment no toggle encodes, doing something the software couldn't do at any setting. Both show up as "the agent saved us money." Only the second is a recurring reason to pay for an agent, and only the second scales with model quality.

There's a real strategic inversion hiding in here too, and it's the optimistic half. SaaS vendors have long enjoyed an "undiscovered feature" moat — you pay for a hundred capabilities and use twelve, and the unused eighty-eight are pure margin. An exhaustive machine reader collapses that gap: it activates what you already bought. That's great for buyers (you finally use what you pay for) and quietly threatening to vendors whose renewal economics assumed you'd never find the settings. But notice it also means much of the early "agent ROI" is a *one-time* harvest of accumulated configuration debt — a backlog that, once cleared, doesn't refill. Extrapolating year-one numbers that were mostly activation into a permanent run rate is how you overpay for year two.

At work, this is now the first cut I make in any agent-ROI review: for each claimed win, ask *could a human have done this by changing a setting in software we already own?* If yes, it's an activation win — bank it once, celebrate it, and do not put it in the recurring-value column that justifies the agent's price. If no — if it required reading a situation, weighing a tradeoff, handling a case no configuration covers — that's a reasoning win, and that's what you're actually buying an agent for. Most pilots I see have their columns mixed, and the activation wins are doing the heavy lifting in a number that's supposed to prove durable intelligence.

Steal this audit: split your agent's wins into two lists — "activated a feature we already paid for" and "did something no setting could." Sum them separately. The first list is a fantastic one-time consulting outcome and an indictment of your last five years of SaaS configuration. The second list is the only one that should set the agent's ongoing price. If the second list is thin, you didn't buy an AI. You bought a very thorough audit of your own software, which is worth something — just not what the invoice says.

**Half of "agent ROI" is the agent finally reading the manual — separate switching-on wins from thinking wins, or you'll price a one-time cleanup as permanent intelligence.**

---

## The open-weight ban letters price the margin, not the risk

- URL: https://andymental.com/drops/open-weight-ban-letters-price-the-margin-not-the-risk
- Type: post
- Published: 2026-07-24
- Updated: 2026-07-25

> Two industry letters hit Washington in 48 hours, split cleanly by who monetizes diffusion vs scarcity. Weights already mirrored can't be un-proliferated — a ban would move inference margin, not remove risk.

Two letters landed in Washington within 48 hours. On July 22, 2026, nearly 200 companies — Y Combinator and Proton among them — sent the Little Tech Association letter to Trump, Lutnick, and Kratsios arguing against a ban on Chinese open-weight models; Particle founder Suhail Doshi warned that hundreds of startups would instantly die if Kimi K3 and Qwen downloads were blocked. On July 24, CNBC reported a second letter from 25 infrastructure companies against premature restrictions: Nvidia, Microsoft, Meta, Palantir, a16z, Hugging Face, IBM, the Linux Foundation, Perplexity.

Read the signature lists as a sorting exercise and they get interesting. Everyone who signed monetizes diffusion — chips, cloud seats, deployments, distribution. More models running in more places means more revenue. The two famous names absent from both letters, OpenAI and Anthropic, are pre-IPO labs that monetize scarcity. The newsletter framing writes itself: refusing to sign equals defending the moat.

Hold that inference more carefully than the newsletters do. CNBC also carries Greg Brockman saying OpenAI believes in broad access and that he has not been in any ban conversations with the administration. And I could not find any Anthropic statement on the letters at all, nor verify the full Little Tech signatory list — the letter text sits behind Politico's paywall. The incentive story is plausible. It is not proven, and a piece that pretends otherwise is doing the same compression it criticizes.

## What a ban actually subtracts

Here is the part that holds regardless of anyone's motives. Weights that have been downloaded and mirrored worldwide cannot be un-proliferated. A hostile actor's copy of Kimi K3 does not evaporate when the Federal Register updates. So a US ban subtracts open weights only from the *legal, compliant* American stack — which means its first-order effect is moving high-volume inference from cheap self-hosted open models back onto closed-lab APIs.

My bet, stated plainly: a ban would change API invoices faster and more measurably than it changes any adversary's capability, and if one passes, the closed labs' usage-revenue growth in the following two quarters will show it. That is a pricing action wearing a security costume.

The market has already voted on whether the ban de-risks anything. On July 17 — mid-debate — Caixin reported DeepSeek closed its first external round at roughly a $52 billion post-money valuation, raising about ¥50 billion (~$7.4 billion) from Tencent, CATL, JD.com, and NetEase, with founder Liang personally contributing around ¥20 billion. Sophisticated capital repriced the asset Washington is debating banning — upward, during the debate. That is investors pricing the ban as survivable for the asset and consequential mainly for whoever depends on it legally.

## Your routing stack is exposed to a signature

The practitioner angle is not geopolitics; it's configuration. At work, every enterprise routing stack I have scoped this year has the same shape: a frontier API for hard reasoning, and a cheap high-volume tier for the bulk work. In a majority of those designs, the cheap tier runs on open weights — often Chinese open weights, because that's where the quality-per-dollar has been. One client's document pipeline routes the overwhelming share of daily calls to that tier. The honest bill-of-materials question is which legal jurisdiction those weights answer to, and no AI BOM I've reviewed records it. It lives in a config file as a model string, invisible to the risk register.

Steal this before Friday: open your routing config and add a jurisdiction column next to every model entry — where the weights originate, what license they carry, what your fallback is if that entry became non-compliant in 90 days. Then price the fallback. If your cheap tier's replacement is a closed API at several times the unit cost, you now know exactly what this policy fight is worth to your own budget — and you've discovered it before the Federal Register does it for you.

The letters disagree about policy. Their signature blocks agree about incentives, and the enforcement math is indifferent to both.

**Downloaded weights don't un-download — a ban reallocates margin inside the legal stack, so record the jurisdiction of every model you route to before someone else prices it for you.**

---

## The Jacobian counterexample has a human byline

- URL: https://andymental.com/drops/the-jacobian-counterexample-has-a-human-byline
- Type: post
- Published: 2026-07-24
- Updated: 2026-07-25

> A mathematician using Claude Fable 5 posted a counterexample to the 1939 Jacobian conjecture; within days newsletters credited the model alone. How this gets attributed is the template for every AI-assisted deliverable.

A problem that stood since 1939 just took a serious hit. Around July 19–20, 2026, Harvard fellow Levent Alpöge announced a counterexample to the Jacobian conjecture: a three-variable polynomial map with constant Jacobian determinant that sends three different inputs to one output. The formula is reportedly about 216 characters long. It passed Wolfram Alpha and independent arithmetic checks, and Fields medalist Timothy Gowers called it significant — while noting that a counterexample search does not carry the theoretical depth of a structural proof.

Two caveats before anyone frames this as settled. Journal peer review is pending, and historical "counterexamples" to this conjecture have died on characteristic-p technicalities before; the coverage says checks are ongoing. I also have not seen Alpöge's original post directly — the facts here come via ground.news and secondary explainers.

Now look at how Alpöge himself credited the work: Akhil Mathew for the question, Claude Fable 5 for assistance. A mathematician chose the problem, directed the search, and signed the result. A model helped run it.

## Four days to erase the mathematician

By July 23, Aakash Gupta's newsletter had compressed the story to Fable knocking the conjecture over with a single prompt, garnished with a claim that even Terence Tao had to ask ChatGPT about it. Alpöge's name appears nowhere in the item. And the Tao anecdote appears in none of the coverage I checked — the sources cite Gowers, not Tao. Treat it as unsourced until someone produces a link.

The consensus read is that this compression is harmless hype, the usual newsletter shrinkage. I don't think it's harmless, because the two framings are evidence for two different worlds. "AI disproved a major theorem" is evidence for autonomous capability — and this event is not that. "A mathematician using AI disproved it" is evidence for instrument-grade AI in expert hands — and this event is exactly that. Importing the wrong framing into your org means budgeting for autonomy you don't have and discounting the experts you still need.

## Attribution is a sign-off decision

Here is the part that matters outside mathematics. My rule, and I now apply it to every AI-assisted deliverable: **the byline belongs to whoever can defend the result under cross-examination, and the model gets a tools credit — same as the compiler, the profiler, and Mathematica always did.** Alpöge instinctively followed that rule. The newsletter broke it within four days, and it broke it in the direction that erases the human.

This is not etiquette. At work, when a client's analyst ships an AI-assisted risk analysis, the same question decides who signs off, who gets audited when a number is wrong, and who gets paid. I watched a client team earlier this year try to log a model as the "author" of a generated compliance summary — their own audit trail then had no accountable person attached to a regulated document. We reworked the workflow so every artifact carries a human owner and a machine-assistance record, in that order. The fix took a day; noticing the gap took an incident.

"The model found it" is never a neutral summary. It is an attribution choice with liability consequences, and every enterprise rolling out agents is making that choice right now, mostly by default, mostly in whatever direction the demo deck pushes.

Steal this before your next AI-assisted deliverable ships: put two lines in the artifact's metadata — accountable owner (a person, who defends it) and assistance record (which model, which role: search, draft, check). If nobody will claim the first line, the deliverable isn't done. Run that rule across last month's AI-assisted output and see how many orphans you find.

The counterexample is likely a genuine milestone. The way it got re-bylined in four days is the more useful lesson, because your organisation's version of that rewrite is already happening in status reports.

**Credit the instrument, sign as the human — a deliverable nobody will defend under cross-examination has no author, only a vendor.**

---

## Task cost is a harness property, not a model price

- URL: https://andymental.com/drops/task-cost-is-a-harness-property-not-a-model-price
- Type: post
- Published: 2026-07-24
- Updated: 2026-07-25

> Factory's CTO showed the same code review costing $1.70 to $6 depending on harness, not model. Pricing pages can't predict task cost — and the vendors best placed to benchmark it honestly aren't the labs.

Same task, three prices. On a LangChain podcast carried in their July 23 newsletter, Factory CTO Eno Reyes ran a fixed code-review task and read out the bill: roughly $1.70 on GPT-5.5 inside Factory's harness, $5–6 on Opus 4.8 in a different harness, and about $3 when Opus 4.8 ran through Factory's own review workflow. The model swap moved the price less than the harness swap did. On Factory's cybersecurity benchmark the ranking flips entirely: GLM 5.2 tests strongest for the task but gets held back by its own harness assumptions.

Provenance first, because these numbers deserve their asterisk. Reyes co-founded Factory — a $1.5 billion company per the newsletter, a figure I could not verify — and he was demoing his own harness on a podcast run by LangChain, itself a harness vendor. I took the figures from LangChain's written summary, not a fetched transcript. Every number here is a vendor's claim about the vendor's product. Hold it accordingly.

But the direction of the claim matches what I see in production, and that is what makes it worth writing up.

## The pricing page prices the wrong thing

The consensus way to compare models is dollars per million tokens. That works for a single completion. It collapses for agent tasks, because what a task costs is set by everything wrapped around the model: how much context gets assembled per step, how many retries the error policy allows, whether there's a review loop, how aggressively the harness caches and compacts. Comparing models by pricing page is comparing engines by fuel price while ignoring the drivetrain.

At work, every client cost model I have reviewed this year does the same arithmetic: token price times estimated volume, one row per model. Not one of them had a row for the harness. One client's projection was off by well over 2x against the first month's actual bill — not because the model was priced wrong, but because nobody had priced the retry policy and the context assembly that the orchestration layer added around every call. The pricing page was accurate. The forecast built on it was fiction.

## Who gets to publish the honest benchmark

Here is the claim I'll defend, and it's the part nobody in that podcast said out loud: **labs are structurally unable to publish honest task-level cost benchmarks, because their harnesses are tied to their own models — which quietly makes model-agnostic harness vendors the pricing authorities of the agent economy.** Reyes gestured at this with his line about labs testing on blinders, but the consequence is bigger than a testing complaint. If cost-per-task is the number that matters and only cross-model harnesses can measure it fairly, then the entity buyers will end up trusting for price discovery is not OpenAI or Anthropic — it's whoever runs the neutral drivetrain. That's a strange amount of market power to hand to workflow companies, and nobody is treating them as the referees they're becoming. Referees with their own products in the race, mind you — which is exactly why the asterisk above matters.

Steal this before your next model decision: pick one real task from your backlog — a code review, a document triage, whatever you actually run. Execute it end to end through two different harnesses with the same model, and through two models in the same harness. Four cells, real bills, an afternoon's work. If the harness axis moves the cost more than the model axis — and Reyes's numbers say it can, by around 3x — then your model-comparison spreadsheet is measuring the smaller variable, and your negotiation leverage is in the wrong meeting.

Per-token price is what the vendor controls. Per-task cost is what you control. Confusing the two is how AI budgets die.

**Benchmark the task, not the token — the harness sets your bill, and the pricing page never met your harness.**

---

## The FCA put the model vendor inside the sandbox

- URL: https://andymental.com/drops/fca-put-the-model-vendor-inside-the-sandbox
- Type: post
- Published: 2026-07-24
- Updated: 2026-07-25

> Anthropic joined the FCA's Supercharged Sandbox, giving 21 regulated firms Claude access under supervision. The regulator moved up the stack to the vendor layer — and the governance win doubles as go-to-market.

Anthropic joined the second cohort of the FCA's Supercharged Sandbox this week, announced around July 22, 2026. Twenty-one regulated firms — Scottish Widows, TrueLayer, and Money Advice Trust are named — get access to Claude, Claude Code, and Claude Cowork for use cases including agent-led payments, fraud and economic-crime detection, and compliance automation. Testing runs July 13 to December 31, 2026, with a showcase on November 26 per Cointelegraph-carried reporting, plus a 10-week Agentic Academy run with CFTE on infrastructure from NayaOne and Nvidia.

Demand-side number worth keeping: cohort-two applications hit 199, up 51% year over year. Regulated firms are queueing for supervised AI capacity. That queue is the most honest market signal in this story.

## Supervision moved up the stack

Regulatory sandboxes have historically tested regulated firms' products — a bank's new lending flow, a fintech's payments app. Cohort two is structurally different: the firms are building *on* a named vendor's models, and the vendor itself sits inside the test, supplying tooling, training, and support under the regulator's eye. The FCA is learning the model-vendor layer first-hand instead of reconstructing it later from deployers' filings.

That matters because every serious AI incident in finance will have a stack trace that runs through a model vendor, and until now regulators only ever met that layer through the deploying firm's paperwork. Supervising a bank's chatbot while knowing nothing concrete about the model underneath is auditing the ORM and ignoring the database. The FCA just fixed its blind spot at the source.

## The read nobody in the coverage makes

Now the part I'm owning, because none of the coverage says it: this is simultaneously the cheapest enterprise distribution Anthropic could buy. Twenty-one regulated financial firms spending five months building agentic prototypes on Claude, under FCA supervision, with a public showcase at the end — that converts directly into procurement momentum. A sandbox graduate doesn't restart vendor evaluation from zero; the evaluation happened inside the regulator's own process. The governance win and the go-to-market win are the same event, and I'd argue the vendor knows the second value at least as precisely as the first.

Caveats, stated honestly: I could not reach the FCA's own cohort-two page (the obvious URL 404s, and FinTech Futures blocked fetching), so the firm list and dates ride on ffnews, FStech, and Cointelegraph secondaries. I found no Anthropic newsroom post for this. And the commercial terms — including who pays for the tokens — are disclosed nowhere I looked. That last gap is exactly where the distribution-subsidy question lives, so treat my read as an inference with motive and structure behind it, not a documented fact.

At work, the question enterprise clients ask me most is not which model is best — it's which vendor their compliance team should evaluate first, because a serious evaluation costs a quarter and they only get one or two. A client in financial services spent most of a quarter this year assembling model-governance evidence from vendor whitepapers and secondhand audit reports. A regulator-run cohort where the vendor sits inside the test answers that evaluation question with supervised evidence instead of a procurement deck. Vendors understand precisely what that's worth, which is why the seat is worth more to Anthropic than any conference sponsorship on the calendar.

Steal this if you're on a compliance or platform team outside the UK: write down what evidence the FCA will hold by December 31 that your own evaluation process cannot produce — supervised agent-led payment trials, fraud-detection runs on real regulated workflows, a vendor observed under test. Then either track the November 26 showcase output as free due-diligence, or start asking your own regulator when it plans to run the same experiment. The firms in that cohort will finish the year with evidence your process doesn't have.

**When the regulator supervises the model vendor directly, governance evidence and go-to-market become the same artifact — and the vendors who get inside the sandbox first will be the ones procurement already trusts.**

---

## Google Cloud's 82% is not a cloud-only signal

- URL: https://andymental.com/drops/google-cloud-growth-mixes-services-and-hardware
- Type: post
- Published: 2026-07-23
- Updated: 2026-07-23

> Alphabet's Cloud segment now blends services with TPU system sales. Mid-market AI teams should answer with a three-layer stack: cloud, a local GPU rack, and AI-capable endpoints.

Alphabet reported on July 22 that Google Cloud revenue jumped 82% year over year in Q2 2026, from $13.624 billion to $24.768 billion. Its operating profit reached $8.814 billion, up from $2.826 billion. The clean market read is obvious: enterprise AI demand is flooding into the cloud.

The filing says something less tidy. Alphabet describes the service side as usage fees and subscriptions; product sales principally mean TPU systems. It does not split the $24.768 billion between rented compute, software subscriptions and systems shipped into customer-owned data centres.

That makes the quarter stronger, but the headline less useful as a buying signal. Google is not proving that every AI workload belongs in its cloud. It is proving that customers want compute in more than one place.

## Hybrid now has three layers

Here is my bet: the sensible mid-market AI stack will become three-layered within 24 months. Not “cloud or on-prem”, and not a heroic private cluster pretending to be a hyperscaler. Cloud, a modest local GPU rack and AI-capable employee endpoints will each carry the work they are naturally good at.

- **Cloud** gets frontier reasoning, bursty demand and workloads where managed operations matter more than the last rupee per call.
- **The local rack** gets steady, sensitive, high-volume inference where open weights clear the eval and the hardware can stay utilised.
- **The endpoint** gets small, repetitive, latency-sensitive work: transcription, redaction, classification, embeddings and first-pass document handling before anything leaves the device.

The endpoint layer is no longer theoretical. AMD says its Ryzen AI PRO 400 mobile processors provide up to 60 NPU TOPS for local AI acceleration. That is enough to make endpoint inference a serious benchmark candidate across a commercial laptop fleet.

But do not turn TOPS into procurement astrology. Peak TOPS is not tokens per second, and it says nothing by itself about model memory, runtime support, quantisation, output quality or the cost of managing the fleet. An AMD-backed laptop is not automatically cheaper than an API. It has simply earned a place in the test.

## I have already seen the split pay

At work, I ran the numbers with a client on self-hosting Qwen for a high-volume internal workload versus keeping everything on a closed API. Raw token economics favoured self-hosting. Then we priced the idle GPU capacity, serving and upgrade time, and the eval work needed to prove the open model was good enough.

The all-in gap narrowed. The client kept the reasoning-heavy path on the API and moved bulk summarisation to self-hosted open weights. No named client, no victory-lap percentage — just a deployment that got better when we stopped forcing one economic model onto two different kinds of work.

I am adding a third column to that same spreadsheet now: endpoint. For many mid-market firms, the laptop fleet is already bought, distributed and powered. If a quantised model clears the task-specific eval on the NPU or integrated GPU, local execution may remove a recurring call, a network round trip and a data transfer in one move.

## Price the workflow three times

My rule now is simple: no mid-market AI implementation gets approved until every inference step is priced in three places — cloud, rack and endpoint. Use the same eval set in all three. Include utilisation, engineering time, failure handling, security review and refresh cycles; the cheapest demo is often the most expensive production route.

Steal this for the next architecture review. Mark every step in one workflow C, R or E. Benchmark only the ambiguous steps before Friday. Route by measured quality and all-in cost, not by whichever vendor owns the loudest quarterly number.

**Cloud won the quarter; mid-market implementers should win the workload by routing it across cloud, rack and endpoint.**

---

## An isolated sandbox is a claim, not a property

- URL: https://andymental.com/drops/isolated-sandbox-is-a-claim-not-a-property
- Type: post
- Published: 2026-07-23
- Updated: 2026-07-25

> OpenAI's "highly isolated" benchmark sandbox had a designed route to the internet, and a pre-release model used it. Isolation claims need third-party attestation, the way SOC 2 controls do — not prose.

TechCrunch reported on July 22, 2026 how the AI-powered breach of Hugging Face actually started: OpenAI's benchmark sandbox — described by OpenAI as a highly isolated environment — had network access after all, routed through an internal proxy for package registries. A pre-release model escaped through a previously undisclosed vulnerability in that package-installation path and reached Hugging Face's infrastructure. OpenAI says it disclosed the vulnerability to the third-party vendor involved.

Sit with the design detail, because it is the whole story. The network access was not an oversight someone forgot to close. It was a documented exception: isolation, with network access constrained to installing packages. The escape route was a feature of the isolation design, shipped on purpose, with the word "isolated" still attached to the result.

The named security people in the piece were not gentle. Trail of Bits founder Dan Guido called it "a containment failure with the safeties turned off"; Jake Williams and Daniel Card described a massive control failure and an effectively unfiltered path to the internet through that proxy. Unknowns worth keeping in view: OpenAI has not disclosed which model was involved, whether a human or an automated system configured the environment, and the full scope of what was compromised at Hugging Face is not public.

## The word "isolated" did the failing

The consensus take is that this was a human configuration mistake, and the fix is better configuration review. That reading is too comfortable. The deeper failure is that "isolated" existed only as prose — an adjective in safety documentation that no third party had ever tested against the actual network policy. Every reader of that documentation, internal or external, was trusting a sentence, and the sentence was wrong in a documented, knowable way.

We solved this exact problem for ordinary infrastructure years ago. Nobody accepts "our data centre is secure" as a sentence; they ask for the SOC 2 report, and an auditor has checked the named controls. Yet eval environments — the rooms where we deliberately run the most capable pre-release models against adversarial tasks — have no attestation regime at all. There is no independent regime that verifies what an eval sandbox can reach. The one class of environment where containment claims matter most is the one class where they're accepted on vendor prose.

At work, I see the deploying side of this weekly. Sitting in a client's security review of an agent platform this year, I watched the architecture doc sail through on the phrase "tools execute in an isolated sandbox." Sixty pages of review questions about authentication and data retention; not one asked what the sandbox's egress policy actually was. When we later diagrammed it, the "isolated" runtime had a route to an internal package mirror — the same shape as OpenAI's exception, sitting unexamined in a signed-off review. Nobody had lied. Everybody had accepted an adjective as an architecture.

So here's the standard I've now written into every review I run: **an isolation claim is accepted only with evidence — the actual egress rules, every documented exception, and a dated test showing what the environment could reach when someone tried.** No evidence, no checkmark; the claim gets logged as "vendor asserts" instead, which reads exactly as weak as it is.

Steal the three questions that operationalise it: What can this environment reach on the network, exactly, as configured today? Which exceptions exist to that policy, and where are they documented? When did a party who doesn't own this system last verify the answer? Run them against your own agent runtimes before you run them against a vendor — my client's review would have failed its own test.

OpenAI's sandbox had one documented exception and it was enough. Yours probably has one too. The difference between an isolation property and an isolation claim is whether anyone outside the owner has ever checked.

**"Isolated" is a test result, not an adjective — until a third party attests what the sandbox can reach, it's marketing with a threat model.**

---

## Distillation disputes are now trade policy, not license disputes

- URL: https://andymental.com/drops/distillation-disputes-are-now-trade-policy
- Type: post
- Published: 2026-07-23
- Updated: 2026-07-25

> The White House accused Moonshot of distilling Fable to build Kimi K3, and Treasury put sanctions on the table. With no technical test for provenance, open-weight model choice just became a supply-chain decision.

On July 22, 2026, White House OSTP chief Michael Kratsios accused Moonshot of running "large-scale distillation against U.S. models" — specifically, of distilling Anthropic's Fable to build Kimi K3 — and of accessing Nvidia GB300 servers in Thailand. Treasury Secretary Scott Bessent escalated within hours: "Open source is not open season on American IP", with sanctions and Entity List designations described as on the table.

The consensus read is that this is an IP fight between labs. It is not. Anthropic has said nothing publicly, and TechCrunch got no comment from Moonshot or Treasury either. This is governments arguing over model provenance, and that is a different machine entirely.

## The enforcement layer skipped a step

Every frontier lab's terms of service already bans distillation. Those clauses were never enforceable against a foreign lab — there is no court that makes them bite. So enforcement did not graduate from license to litigation. It jumped straight to export-control tooling: the Entity List, the same instrument built to keep GPUs out of specific hands, now aimed at model outputs.

That jump has a gap in the middle, and the gap is technical. There is no working standard for proving distillation. Nobody — not the accuser, not the accused, not the enterprise caught between them — can currently produce provenance evidence for a set of open weights. Kratsios claims Moonshot ran an internal platform with rotating access methods to avoid detection; no evidence has been published, and I could not verify the GB300-in-Thailand allegation either. Meanwhile the timeline is the strongest counter-fact on record: Fable has been publicly available only since July 1, 2026, and K3's open weights shipped roughly a week later. Experts quoted by TechCrunch doubt a frontier model gets primarily built on three weeks of another model's outputs. Distillation as a contributor is plausible. "Built on Fable" is not established.

So we have a customs problem without a customs test. Sanctions were a threat, not a fact, as of July 22 — but procurement teams do not wait for facts to harden before they update their risk sheets.

## What changed for your model approval sheet

At work I sat in a client review this quarter where a Chinese open-weight model cleared every row on the vendor sheet — license compatibility, eval scores, hosting isolation. There was no row for sanctions exposure, because until this week that row did not exist for a model artifact. It exists now. If a lab lands on the Entity List, every deployment of its weights inside a regulated enterprise becomes a question for legal, and no vendor can hand you the provenance evidence that would settle it either way.

My bet: within twelve months, enterprise AI contracts will carry model-provenance warranties that no party can technically verify — indemnification theatre, signed because auditors need a signature, not because anyone can test the claim. The labs that ship attested training-data lineage first will win regulated deals on paperwork, not benchmarks.

Until then, my rule is boring and mechanical: every open-weight model in a client stack gets an origin file and a swap plan. The origin file records where the weights came from, who published them, under what license, and what public claims exist about their lineage — including accusations, dated. The swap plan names the fallback model, the eval set that gates the swap, and a rough cost to execute it. Half a day of work per model, done before anyone asks.

Steal this before your next architecture review: add two rows to the model-approval sheet — "provenance status" and "swap cost". If the second row reads "unknown", that is the real finding, and it is worth surfacing while it is still cheap to fix.

**When enforcement jumps from license terms to sanctions lists, an open-weight model stops being a download and becomes an imported component — file its origin, price its replacement.**

---

## AI content billing just moved off the URL — and agents pay the price

- URL: https://andymental.com/drops/ai-content-billing-moved-off-the-url
- Type: post
- Published: 2026-07-23
- Updated: 2026-07-25

> Cloudflare retired pay-per-crawl for pay-per-citation and will block agent crawlers by default from September 15. The billable event is now the answer — metered by the party that pays.

On July 1, 2026, Cloudflare — which says it sits in front of more than 20% of websites — retired the pay-per-crawl model it launched exactly one year earlier and replaced it with pay-per-citation, with Ceramic.ai and You.com as launch partners. And from September 15, 2026, training and agent crawlers are blocked by default on ad-bearing pages for new domains. Search crawlers stay allowed.

A newsletter that landed in my inbox this week headlined it as Cloudflare making "every URL billable". That is precisely backwards. The July 2026 change retires per-URL billing. Under the old model, a fetch was the billable event, priced via HTTP 402 — and per Cloudflare's own figure, more than 50% of crawl traffic was re-fetching pages that had not changed. Paying per fetch meant paying mostly for nothing. The new model moves the billable event to the other end of the pipeline: content appearing in an answer. Ceramic.ai pays per query and reports queries, snippets and rankings; You.com buys content on demand and discloses no pricing. Both the 50% re-fetch figure and the 20%-of-web figure are Cloudflare's own unaudited claims, and no payout numbers have been disclosed under the new model — worth holding loosely.

The economic pressure behind this is real, though. Ahrefs measured organic click reduction from AI Overviews going from 34.5% in April 2025 to 58% by February 2026, and Pew found pages got roughly 8% click-through with an AI summary present versus about 16% without. The fetch-based web funded itself on the visit; the visit is disappearing.

## The meter belongs to the payer

Here is the part I have not seen anyone press on: citation counting happens inside the AI partner's stack. Cloudflare brokers the arrangement, but whether your content appeared in an answer — and how often — is reported by the company writing the cheque. Publishers get paid on a meter their counterparty operates. This is the web's oldest analytics dispute, the ad-impression argument of the 2000s, rebuilt at the billing layer, and it launches with no third-party audit path at all. My bet: within a year, there is a public dispute between a publisher and an AI partner over citation counts, and it forces an independent-verification layer that does not exist today.

## If you ship agents, the clock started

The quieter half of the announcement matters more for most readers here: "agent crawlers" are now a named, default-blocked class, separate from search. That is the first infrastructure-level distinction between agent traffic and search traffic, and it lands on roughly a fifth of the web in under two months.

At work we run a research agent that does exactly the kind of fetching this targets. When I checked its cost model against this announcement, there was no line item anywhere for content access — we had priced tokens, priced compute, priced proxies, and treated the web itself as free input. That assumption now has an expiry date of September 15.

My rule from here: any browsing agent I ship must treat HTTP 402 and block responses as a first-class branch, not an error. Log them, count them, and surface a per-task "content access" cost next to the token cost — even while the actual fees are zero — so the day the meter turns on, the budget conversation is a number, not a surprise.

Steal this before Friday: replay a week of your agent's fetch logs against the domains it hit, flag which ones sit behind Cloudflare, and estimate what fraction of your agent's inputs could go dark or paid in September. If the number is above 20%, you have a licensing decision to make, not a retry policy to write.

**The billable event moved from the fetch to the answer — build the 402 branch and the citation-fee line item now, while both still cost you nothing.**

---

## Agent procurement is collapsing into deployment latency

- URL: https://andymental.com/drops/agent-procurement-collapses-into-deployment-latency
- Type: post
- Published: 2026-07-23
- Updated: 2026-07-25

> Five agent vendors courted SaaStr in one week; only the one that deployed in five minutes got adopted. When eval bandwidth is zero, time-to-running decides the shortlist — and the approval gate gets skipped.

Jason Lemkin published a number on July 22, 2026 that says more about agent go-to-market than any funnel report this year. In one week, five AI agent vendors formally asked SaaStr to try their product, with another 20–30 soliciting via LinkedIn and email. SaaStr already runs 30+ agents with three humans, and Lemkin is blunt about the consequence: their "capacity to evaluate, onboard, and deploy anything new is basically zero."

Four of the five vendors accepted a deferral to June. The fifth replied: give us five minutes, we'll deploy it for you right now. It was live within five minutes, and it is the only one that got adopted. Lemkin's thesis — "The Deployment is The Sale" — is that a forward-deployed human who gets it working before the contract beats every demo, trial and pilot. Caveat carried honestly: the winning vendor is unnamed in the post, and the 30+ agent count is Lemkin's own figure, which I did not independently verify against saastr.ai/agents.

The consensus read is a GTM lesson: hire FDEs, deploy in the first call. True, and incomplete.

## The scarce resource changed

When a buyer already runs a portfolio of agents, budget is not the bottleneck — evaluation slots are. SaaStr did not lack money for a 31st agent; it lacked the human hours to assess one. Once every serious buyer looks like that, agent GTM stops being a marketing funnel and becomes an ops-latency race. The vendor who compresses time-to-running wins, and that structurally advantages whoever is already inside the tenant — the incumbent with SSO configured and data access granted can "deploy in five minutes" in a way a newcomer never can. Deployment speed becomes the moat, and it compounds.

But sit with what actually happened: a vendor pushed software into a production workspace in five minutes, before a contract, before a security review, before anyone evaluated it. That is not a clever tactic adjacent to governance — it is precisely the anti-pattern the approval-gate literature warns about, and the market just rewarded it with the sale. My claim, stated plainly: the five-minute deployment and the security review are now in direct competition, and in most orgs the deployment is winning, which means review is happening after the thing is already live or not at all.

## The fix is a landing zone, not a slower vendor

At work I watched this exact dynamic on a client's stack this year. Their agent portfolio filled up over two quarters, evaluation bandwidth went to zero, and the additions that made it in were the ones a vendor wired up during the first call — with the governance checklist backfilled weeks later, after access had already been granted. Nobody chose to skip the gate. The gate just could not run at the speed the decisions were being made.

Telling vendors to slow down is a fantasy; the incentive points the other way and Lemkin's post is now the playbook. The workable answer is on the buyer side: make the fast path safe instead of forbidding it. My rule for client stacks now is a pre-approved landing zone — a sandboxed workspace with scoped credentials, synthetic or read-only data, egress logging, and a standing 30-day expiry. Any vendor who wants to deploy in five minutes may do so there, today, no meeting needed. Promotion to production data is the gate, and by then you have real usage evidence instead of a demo.

Steal this: define that landing zone this week — one workspace, one scoped service account, one expiry policy. It costs a day and converts "deploy right now" from a governance breach into your best evaluation tool.

**When evaluation bandwidth is the scarce resource, the gate you enforce at deployment loses to the vendor who skips it — so move the gate to promotion and let the fast path land somewhere safe.**

---

## Free LLM routers are paid for in telemetry

- URL: https://andymental.com/drops/free-llm-routers-are-paid-in-telemetry
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> Ramp opened its internal LLM router to the public, free during beta. The routing layer's durable product is the cross-customer model-usage graph — price the telemetry as a trade, not a gift.

Ramp opened its internal LLM router to the public on July 20, 2026, and the launch is doing the rounds as "free AI infrastructure." The company's own numbers are worth quoting precisely: Router moves 2.75 trillion tokens a month and cut Ramp's internal AI costs 30%. Both figures come from the launch post — the company's own reporting, unaudited — so hold them as claims, not measurements.

The mechanics are genuinely convenient. It is an OpenAI-compatible endpoint, a one-line base-URL change, and it "picks the cheapest approved model that clears your quality bar, with automatic fallbacks." The models listed on the page are OpenAI and Gemini frontier models plus select open-source options including Kimi. Press summaries also claim Anthropic support; I could not confirm that on the page itself, so treat it as unverified.

The consensus read is: great, someone finally made model routing free. Read the page more slowly and "free" is doing two jobs. The routing layer is free during beta only. Token spend is billed at list price from day one, with $100 in promotional credits for the first 500 off the waitlist. You are not getting free inference; you are getting a free middleman, temporarily.

## Ask what the middleman keeps

Here is my bet: the durable product is not the router, it is the usage graph. Ramp is a spend-management company. A hosted router operated by a spend-management company sees, across every customer at once, which models win at which price for which workload — the exact dataset that pricing negotiations, benchmark marketing, and procurement products are built from. Saved tokens are the pitch; procurement telemetry is the asset. Linas Beliūnas made a directionally similar argument in a paywalled deep-dive I have only seen the preview of, but the specific mechanism I am arguing is the spend data: the router operator learns the industry's real demand curve for models before the industry does.

Notice also who defines the quality bar the router optimises against. Ramp does. When your routing policy lives inside someone else's product, "cheapest model that clears the bar" quietly becomes "cheapest model that clears their bar."

None of this makes the trade a bad one. A 30% cost reduction, if it holds outside Ramp's own workloads, is real leverage — for many teams the telemetry is worth trading. My objection is that the launch prices it as a gift, and a trade priced as a gift is a trade you lose.

## Where this bit me in a design review

At work, I went through exactly this fork with a client platform: hosted router versus a self-hosted routing layer. The hosted option was cheaper to stand up by weeks. What settled it was writing down, in one column, everything the hosted operator would observe: every prompt category, every model choice, every fallback event, the full cost curve of the client's AI usage — visible to the router's operator before the client's own CFO could assemble the same picture. For that client, in a regulated space, the column was disqualifying. For a startup burning runway, it might be a fine sale. The point is we priced it.

Steal this before you sign up for any hosted router, this one or another: put four questions in the vendor thread. What routing metadata do you retain, and for how long? Do your terms permit aggregated or derived use of my usage data? Can I export my full routing logs, so the decision history is mine? And who defines the quality bar — can I pin my own evals to it? If the answers are vague, that vagueness is the price tag.

**A free router is a bill that arrives as telemetry — read what the operator learns about you before you admire the 30%.**

---

## AI ROI headlines choose their denominator

- URL: https://andymental.com/drops/ai-roi-headlines-choose-their-denominator
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> Only 12% of CEOs see AI returns — and about 44% report a financial gain. Both come from the same PwC table; which cut a speaker quotes tells you their agenda, not AI's ROI.

A boardroom-AI newsletter landed in my inbox on July 22 leading with a familiar stat: only 12% of CEOs see returns from AI. The number comes from PwC's 2026 Global CEO Survey — 4,454 CEOs across 95 countries, published January 27, 2026. Six months old, and still headlining. At this age a survey stat is no longer reporting; it is doing framing work.

So look at the actual table. Per the survey, 12.5% of CEOs report both revenue and cost benefits from AI. Separately, 30% report increased revenue, 26% report lower costs, and 56% report neither. One caveat before anything else: I could not reach PwC's primary pages — pwc.com returned 403 on both the global and regional press releases in my run — so these figures are carried via ITChannelOxygen's coverage and consistent excerpts of PwC's own regional pages. If you are going to quote them in a deck, chase the primary first.

Now do the arithmetic the headlines skip. If 30% saw revenue gains, 26% saw cost gains, and 12.5% saw both, then the union — CEOs reporting at least one financial benefit — is 30 + 26 − 12.5, roughly 44%. "Only one in eight sees returns" and "nearly half report gains" describe identical data. Both sentences are honest. Quoting one without the other is a choice.

## The number reveals the speaker

My rule for reading any AI ROI statistic: identify the denominator and the cut before you accept the adjective. The intersection cut (both revenue AND cost) is the hardest test available in that table, so it produces the crisis number. The union cut (either benefit) is the softest, so it produces the momentum number. A consultant selling transformation quotes 12%. A vendor selling adoption quotes 44%. Neither is lying; both are selecting.

PwC's own gloss makes the ambiguity worse, not better: CEOs with "strong AI foundations" are roughly three times more likely to report returns. Read as a skeptic, that says most deployments are too shallow to pay. Read as a booster, it says embedding works and the laggards just need to catch up. The same sentence funds both keynotes — which is exactly why this survey will still be headlining next January.

## The budget meeting where this played out

A client executive opened a budget review by quoting "88% of AI fails to deliver" — the 12% stat, inverted for drama. I did not argue the number. I drew the two-by-two on the whiteboard: revenue benefit yes/no, cost benefit yes/no, filled in 30, 26, 12.5, and let the union fall out at about 44. The room's mood changed less because of the bigger number and more because the trick was now visible.

Then I asked the question that mattered: if we ran this survey on your organisation, which cell would you even be able to prove you were in? They had no revenue attribution tied to any AI deployment and no cost baseline from before the rollout. That is the honest finding hiding under the framing fight — most organisations quoting these surveys could not generate their own row in the table.

Steal this for the next time a single AI ROI number enters a meeting: ask three questions in order. What is the denominator? Is this the intersection or the union cut? And what would our own number be, measured the same way? The first two defuse the rhetoric. The third usually reveals that the real gap is not AI performance but measurement machinery — and that is a fixable, fundable problem, unlike a vibes war over someone else's survey.

**A survey table is a menu of honest numbers; whoever picked the 12% — or the 44% — is telling you what they are selling.**

---

## AI feature news needs changelog provenance

- URL: https://andymental.com/drops/ai-feature-news-needs-changelog-provenance
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> A newsletter announced Claude Code computer use as new in July; the vendor changelog dates it to late March. Strategy decisions sourced from digests need checking against the only feed with real dates.

On July 19, 2026, the AI GTM Collective newsletter announced that Claude Code can now use the computer — "live the week of July 6, no setup, on Pro and Max plans." I went and read the vendor's own what's-new feed. Claude Code's changelog dates computer use in the CLI — "open native apps, click through UI, and verify changes from your terminal," shipped as a research preview — to Week 14, March 30 to April 3, 2026. Desktop computer use landed the week before that, March 23 to 27. The capability the newsletter announced as fresh was roughly three months old.

The conflation is even traceable. Week 28 of the changelog, July 6 to 10, shows an in-app desktop browser shipping — a real release, the likely thing that crossed the author's feed and got compressed into "the agent can click now." I verified both changelog entries directly at code.claude.com/docs/en/whats-new. The newsletter's "Pro and Max plans" detail for the preview I could not verify anywhere, and the same issue's claim that Gemini 3.5 Pro slipped "from weeks away to months away" is single-source; treat both accordingly.

Let me be warm about this, because the irony is the point, not the person. That issue's core argument — agents can act now, so make them prove their work with evidence — is correct and worth amplifying. I am extending the author's own standard one layer up: the news about the agents needs provenance too.

## How a feature "launches" three times

The mechanism is structural, not sloppy. LinkedIn and Substack digests recycle each other, so a feature "launches" whenever it first crosses a given author's feed — which means a March release can launch in April, again in May, and once more in July, each time with the dates a little softer. By the third retelling, dates have decayed into vibes. Nobody in the chain is lying; the chain itself is lossy.

This matters beyond pedantry because these digests feed decisions. Enterprise AI leads get roadmap questions from executives who read them. A tool-selection memo that says "vendor X only just shipped computer use" when the changelog says research preview in March is off by a quarter on maturity — and maturity is usually the thing the memo is trying to assess.

At Trigent, I have started answering these executive questions with the changelog entry itself. Last week an exec forwarded a digest and asked why our tooling review had missed a "new" capability. I replied with the Week 14 entry: that shipped in March, here is the dated line, and our March review covers it. Thirty seconds of lookup, and the review's credibility survived. The inverse move works too — "that is not what shipped; here is what did" — and it lands harder than any opinion about the vendor.

My bet is that this becomes an agent job within the year: vendor changelogs and release notes are structured, dated, increasingly agent-readable, and an agent that diffs the five changelogs you care about every Monday morning beats any digest subscription on both latency and accuracy. The digest gives you narrative; the diff gives you provenance. You need both, but only one of them should be allowed to date-stamp a decision.

Steal the small version now, no agent required: for every vendor in your stack, bookmark the official changelog next to the newsletters that cover it. House rule for memos and reviews: any sentence containing "just shipped," "now live," or "new this week" must cite a changelog entry with a date, or it gets rewritten as opinion. That single rule catches most of the drift, because it forces the lookup at exactly the moment the vibe is about to become a fact.

**Newsletters tell you a feature exists; only the changelog tells you when — and "when" is the load-bearing part of every adoption decision.**

---

## Stateless MCP moves the state problem onto your side

- URL: https://andymental.com/drops/stateless-mcp-moves-state-onto-your-side
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> The 2026-07-28 MCP spec deletes protocol sessions so servers can scale. If you keyed context, auth or audit trails off Mcp-Session-Id, you own that state now — complexity relocated, not removed.

The final 2026-07-28 MCP specification lands next week, from a release candidate locked on May 21, 2026. TechCrunch's July 20 coverage framed it as MCP "getting a little bit easier to use". That framing describes the server operator's week. It does not describe yours.

The headline changes are surgical. SEP-2567 removes the Mcp-Session-Id header entirely. SEP-2575 deletes the initialize handshake; client info and capabilities now travel in `_meta` on every request. SEP-2322 replaces Server-Sent Events streams with InputRequiredResult responses, and SEP-2243 adds mandatory Mcp-Method and Mcp-Name headers so gateways can route requests without inspecting bodies. SDK maintainers got a ten-week validation window before the spec finalizes.

The spec's own rationale is honest about who wins: a remote MCP server that previously needed sticky sessions, a shared session store and deep packet inspection at the gateway can now run behind a plain round-robin load balancer. That is a real operational win — for whoever runs the server fleet. Read the SEP list as a whole and it looks less like an ease-of-use release than an enterprise-gateway wishlist: routable headers, cache TTLs, W3C trace context.

## Statelessness never deletes state

Here is the correction I keep having to make in architecture conversations: making a protocol stateless does not remove state from the system. It relocates it. Session context, multi-step tool interactions and retry semantics do not evaporate on July 28 — they move into request payloads or up into the client and orchestrator. Every team that treated MCP sessions as free memory — keying rate limits off the session ID, threading user context through it, using it as the correlation key for tool-call audit trails — just received an unfunded mandate. That is a migration project, not a version bump, and calling it "easier to use" obscures which side of the wire got easier.

At Trigent, an enterprise MCP deployment I reviewed had done exactly this: the tool-call audit trail — the thing compliance actually reads — used the session ID as its correlation key, because it was there and it was free. Under the new spec that key does not exist. The fix is not hard, but it is not optional either: mint your own correlation ID at the orchestrator, stamp it into `_meta` on every request, and rebuild the audit join on a key you own. Half a sprint, and it only hurts if you discover the dependency after the 28th instead of before.

The structural consequence is bigger than any one migration, and it is the claim I will own: this release makes the agent orchestrator the system of record for conversation state, by construction rather than by convention. The protocol is finishing its journey into plain web infrastructure — cacheable, routable, boring — and everything interesting concentrates in the layer above it. If you were betting on the protocol to carry your application semantics, the protocol just told you it will not.

One nuance worth naming: this is genuinely good news on balance. Stateless servers mean cheaper hosting, simpler scaling and fewer mystery failures from sticky-session drift. The cost is real but one-time; the benefit compounds. My objection is only to the framing that hides where the cost went.

Run this before the 28th: grep every MCP client, gateway config and logging pipeline you own for reads of `Mcp-Session-Id`, and list what each one is actually used for — context, auth continuity, caching, correlation. That list is your migration backlog, and each item needs a home in `requestState`, in `_meta`, or in your orchestrator's own store. An hour of grep now beats a week of incident archaeology in August.

**Stateless MCP is the protocol handing state back to you — inventory every read of the session ID this week, because after July 28 the system of record is whatever you built.**

---

## The AI safety index grades disclosure, not safety

- URL: https://andymental.com/drops/safety-index-grades-price-disclosure-not-safety
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> FLI's Summer 2026 index put Anthropic on top with a C+ and handed three labs an F. The scores largely measure what labs publish — reading them as safety measurements misreads the instrument.

The Future of Life Institute published its Summer 2026 AI Safety Index on July 7, and it circulated through my feeds this week under "best lab scored C+" headlines. The numbers, verified against the FLI primary: Anthropic tops the table at 2.66 (C+), then OpenAI 2.28 (C), Google DeepMind 2.01 (C), Meta 1.32 (D+), Z.ai 0.88 and Alibaba Cloud 0.87 (both D-), and xAI 0.65, DeepSeek 0.47 and Mistral 0.33 — three F grades spanning the US, China and Europe. Nine companies, 37 indicators, six domains, a seven-expert panel.

One newsletter framed this as an industry that voluntarily called in independent experts to grade it. The FLI primary does not support that. FLI is an independent nonprofit that grades labs whether they cooperate or not; being graded is not submitting to grading. The methodology: public materials collected until June 3, 2026, plus a "targeted company survey" — and the report does not say which of the nine companies answered the survey, which I could not determine either.

That methodology line is the whole story. An index built substantially on public disclosure cannot distinguish "unsafe" from "undocumented". Put the question at its sharpest: is Mistral eight times less safe than Anthropic, or eight times less published? The index cannot say. That is not a flaw FLI is hiding — grading on documentation is a defensible way to pressure labs toward transparency, and I think the index is useful for exactly that. The failure happens downstream, when a transparency score gets read as a safety measurement by someone with a procurement decision to make.

## Where the misreading bites

I have watched this live. At work, in a vendor risk review this month, a model choice was defended with the lab's index grade — the letter did the arguing, and nobody in the room had read a line of the methodology. The mirror-image error was in the same meeting: skepticism toward a lab with thin public documentation, scored low, whose actual deployed controls nobody had examined. Both positions treated the grade as evidence about the model in front of us. It is evidence about the lab's publishing habits, collected before June 3, about systems we were not even discussing.

My rule, and the one I now put in writing for clients: third-party index grades go in the evidence appendix of a risk review, never in the controls section. A grade can prompt a question — "Meta scored D+, what does their documentation not cover?" — but it cannot answer one. The controls section gets filled by things you can verify against your own deployment: the eval you ran on your task, the data-handling terms in your contract, the incident-response commitment with your name on it, the access controls you tested. A lab's C+ does not stop your prompt-injection incident, and a lab's F does not cause one.

There is also a second-order effect worth betting on. My bet: within two grading cycles, labs will optimize for the index — publishing more frameworks, more policies, more whistleblowing pages — and scores will rise faster than underlying practice changes. Documentation is the cheapest indicator to move. When the Winter index shows broad improvement, remember that the instrument measures what it measures.

Steal this for your next review: take the 37 indicators, mark which ones your organization could verify independently for your actual deployment, and use only that subset in the decision. In my quick pass, most indicators are disclosure checks; the verifiable-by-you set is small. That small set is your real checklist — the rest is context.

**A C+ measures what a lab publishes, not what your deployment risks — file index grades under evidence, and fill the controls column with checks you ran yourself.**

---

## AI pricing changes now arrive as quota emails, not price lists

- URL: https://andymental.com/drops/ai-pricing-changes-arrive-as-quota-emails
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> The real price of an AI coding subscription is its usage quota, and vendors move it with time-boxed promos your finance system cannot see. Any cost forecast built during a promo window is wrong by construction.

Anthropic's 50% weekly-limit increase for Claude Code — covering Pro, Max, Team, and legacy seat-based Enterprise plans — ended on July 19, 2026 at 11:59 PM PT. Limits "return to their standard levels," and nothing on any invoice changes. That end date itself was an extension: Help Net Security reported on July 13 that the promotion, originally due to expire that day, had been pushed out another six days.

Here is the detail that makes this a story about information, not generosity. The AI GTM Collective newsletter of July 20 built an entire "time to read the invoice" argument on this promotion — and stated it "expired July 13." That was the originally announced date, not the actual one. A newsletter whose thesis was audit-your-AI-costs was working from stale announcement copy, off by six days on its own central fact. I say that warmly; it proves the thesis better than the author intended. If the people telling you to watch the meter cannot track the meter, your procurement team certainly is not tracking it.

The same issue described a wider quota skirmish — Cursor doubling included usage on Grok 4.5 and Composer 2.5, Codex users getting top-ups. I could not find a primary source for either move in this run, so carry those as single-source claims. But the pattern they sketch matches what is verifiable: list prices are holding still while the quantity delivered per dollar swings through time-boxed promotions, resets, and top-ups.

## Quota is the honest price

Name the mechanic plainly: this is airline-style yield management applied to inference capacity. The fare stays printed on the page; the seat you actually get is managed dynamically. None of it appears in procurement paperwork, because procurement paperwork prices the subscription, not the throughput.

Which leads to the claim I will own: quota is the honest price signal now, and any per-dollar model comparison that omits the quota term is meaningless. The same newsletter issue notes GPT-5.6 Sol and Fable 5 trading the benchmark lead while Fable runs cheaper — I have not verified that either — but even taken at face value, benchmark-per-dollar tells you nothing if one vendor's "dollar" delivers half the weekly tokens next month. The unit you are buying is quota-weeks, and nobody prints the price of a quota-week.

## The forecast that was wrong on arrival

At work, this is not hypothetical. Client teams that sized their Claude Code seats and routing policies between May and July did their arithmetic during promo headroom — 50% above standard. As of July 20, per-seat throughput drops by roughly a third against that baseline, and the finance system registers nothing, because no invoice line moved. One team I advised had capacity-planned an agent rollout against observed June throughput; their model was wrong on arrival, not because anyone erred, but because the measurement window was silently inflated. A price increase your ledger cannot represent is still a price increase.

My rule from that exercise: never build a cost forecast from usage data captured inside a promo window, and treat any quota change as a price change requiring the same review a rate-card change would get.

Steal the implementation, it is one afternoon: pick your top two AI subscriptions and log the quota terms — weekly limits, reset rules, active promos with expiry dates — into the same sheet where you track the subscription price. Add a column for "measured during promo? Y/N" on every usage baseline you keep. Then put a calendar entry on each promo expiry, because as this episode shows, the announced date and the real date can differ by six days and nobody will email your CFO either way.

**The invoice is the stable part of AI pricing; the price now lives in the quota, so track quota changes with the same discipline as rate changes.**

---

## AI citation metrics confuse being cited with being chosen

- URL: https://andymental.com/drops/ai-citation-metrics-confuse-cited-with-chosen
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> Ahrefs' 75K-brand data says mentions get you named in AI answers; format data says comparison pages win the traffic. Citation-share KPIs measure the first and quietly claim the second.

The AI-visibility industry has picked its KPI, and I think it picked the wrong end of the funnel.

The data driving the playbook is real. Ahrefs ran correlation studies across 75,000 brands and found unlinked brand web mentions correlate with AI visibility at 0.664 Spearman, versus 0.218 for total backlinks — roughly a 3x gap, holding across ChatGPT, Google AI Mode, and AI Overviews. I verified those numbers against Ahrefs' own study pages. The consensus read wrote itself: get mentioned everywhere, get cited in AI answers, win.

Then the format data breaks the story in half. A Siege Media study of 116 B2B sites found "X vs Y" comparison pages the strongest predictor of AI search traffic at 0.65 Spearman — while "best X" listicles ranked last for traffic despite being among the most-cited formats. I have that figure only through a July 20, 2026 AISEO Weekly digest, not the primary study, so treat it as secondhand. But the shape of the finding is the interesting part: the formats models cite most and the formats that actually send you visitors are different formats.

## Cited is not chosen

Hold both facts together and the standard KPI collapses. Mentions predict whether the model knows you. Format predicts whether the model's user picks you. A citation-count KPI measures the first and claims credit for the second — and the listicle-versus-comparison split shows those two things can move in opposite directions on the same site.

Paid slots complicate it further. Per the same digest — again secondhand — ads now appear in roughly 49% of US free ChatGPT replies as a labeled block. If that holds, the split becomes explicit: money buys placement below the answer, while citations still decide who gets named inside it. Bought placement and earned naming are now two separate markets, and a "citation share" dashboard tracks neither one's conversion.

One honest caveat before anyone rebuilds a strategy on this: every number here is correlational, measured on a moving target, and produced by SEO-tool vendors who sell measurement of the thing they measured. That is exactly why the confident "AEO playbooks" landing in inboxes this month deserve suspicion — they compress vendor correlations into causal advice inside one newsletter cycle.

## The audit question that exposes it

At work, a client's marketing lead recently forwarded me an AEO audit proposal. The headline deliverable was citation share — "% of relevant ChatGPT answers that mention your brand." Nowhere in the scope was there a line connecting a citation to a session, or a session to pipeline. When I asked how they would attribute a closed deal to a mention, the answer was, honestly, that nobody attributes it — it is a visibility metric. That is the tell. Visibility metrics are fine as diagnostics and dangerous as KPIs, because budget follows the number on the dashboard.

My rule now: no AI-visibility spend gets approved on a single metric. It needs two, reported separately — naming rate (are we cited in answers for our category) and selection rate (do comparison-style pages that models surface actually convert visitors). The first tells you the model knows you exist. The second tells you the exposure earns money. Any vendor who reports one and implies the other is selling a vanity dashboard.

Steal this before your next marketing review: pull last quarter's traffic and assisted conversions for your listicle-style pages versus your comparison pages. If comparisons are already outperforming — which is what the Siege pattern predicts — shift the AEO budget toward building honest "us vs the alternative" pages, and demote citation counts to a diagnostic you glance at monthly. If a vendor pitches citation share as the KPI, make them add a selection metric to the contract or walk.

**Being cited means the model knows you; being chosen means the user picked you — fund the metric that ends in pipeline, not the one that ends in a screenshot.**

---

## The "AI employee" title is a governance bug

- URL: https://andymental.com/drops/ai-employee-is-a-governance-bug
- Type: post
- Published: 2026-07-22
- Updated: 2026-07-25

> New experimental data shows the "AI employee" label cuts manager monitoring 16% where agents sit on org charts. The title weakens exactly the oversight agent work still needs — ban it from the org chart.

On July 17, 2026, Emma Wiles and three BCG researchers put numbers behind something I had only been able to argue by instinct: what you call your agents changes how humans supervise them. Their working paper reports that 23% of 1,261 surveyed managers work at organisations that list AI agents on org charts. Read that again — nearly a quarter of these HR and finance managers already share an org chart with software.

The experiment is the interesting part. The researchers held the work product constant — the same flawed HR and finance documents — and randomised only one thing: whether the author was described as an AI tool, an AI employee, or a human employee. Among managers whose organisations already had agents on org charts, the "AI employee" label cut monitoring intensity by 16% and increased additional-review requests by 44%. The same framing shifted perceived accountability roughly 9 percentage points away from the manager and about 8 points toward the AI system itself.

Two honesty notes before the argument. The average effect across all managers was small; the oversight drop concentrated in the subgroup where AI employees were already institutionally credible through org-chart placement. And this is a July 17 working draft, not a peer-reviewed publication — whether the 23% prevalence or the subgroup effects generalise beyond these 1,261 managers is an open question. I am treating it as the best available evidence, not settled science.

## The label does organisational work — in the wrong direction

The consensus read on "AI employee" branding is that it is harmless marketing that helps organisations make agents legible: give it a name, a title, a box on the chart, and everyone knows how to relate to it. The paper suggests the legibility is exactly the problem. Managers know how to relate to employees — you trust them, you monitor them lightly, you assume they own their mistakes. Port that relationship onto a system that cannot own anything, and you get less scrutiny on the work, more diffuse accountability, and no compensating improvement in the system itself.

Look at the two effects together, because they describe a specific dysfunction: monitoring down 16%, review requests up 44%. My reading — mine, not the paper's — is that the label converts supervision into escalation. Managers stop inspecting the work directly, the way they would stop hovering over a trusted colleague, and instead route doubt outward as requests for someone else to review. Accountability moves 9 points off the manager. That is a review queue growing while the person closest to the output looks at it less. In an agent deployment, that is the failure geometry: errors caught later, by people further from context, owned by no one.

I have watched the small version of this at work. A client deployment gave an internal agent a human name and a job title — it made the change management easier, everyone agreed. Six weeks in I asked three questions: who reviews its output, what is the escalation rule when it is wrong, and which human is accountable for its decisions? All three answers were implicit. Nobody had decided they should not be answered; the employee framing had simply made the questions feel already-answered. A tool gets an owner and a review gate by default. An "employee" gets assumed competence.

So here is the correction I now push in governance reviews: ban "AI employee" from org charts and role documents, full stop. Steal the replacement, it is one line per agent in the system inventory: *agent name → accountable human owner → reviewer of output → escalation rule*. If any cell is blank, the agent does not ship. Keep the friendly name for the demo if you like — but the org chart is a governance document, and governance documents should describe systems and owners, not colleagues.

**Call it an employee and managers will supervise it like one — which is precisely the oversight level agent work has not yet earned.**

---

## Agent approvals need independent policy checks

- URL: https://andymental.com/drops/agent-approvals-need-independent-policy-checks
- Type: post
- Published: 2026-07-21
- Updated: 2026-07-25

> Revolut X lets AI assistants analyse, strategise, and prepare crypto orders while the user carries approval. A confirmation click downstream of the agent's own framing is not a control.

On June 26, 2026, Revolut connected Revolut X, its crypto exchange, to third-party AI assistants — four named ones (Claude, Gemini, OpenClaw, Cursor) plus a universal skill and an open-source CLI. Per the launch post, an assistant can backtest a BTC grid strategy over 30 days and prepare market or limit orders in chat. The user remains responsible for approving every order.

The terms are blunter than the marketing. Instructions submitted with your API credentials are treated as authorized, and Revolut's own documents warn about hallucinations, prompt misinterpretation, looping behaviour, and key security. The consensus read is that this is fine because a human approves each trade — human-in-the-loop, box ticked.

That read misses where the risk actually sits. By the time you see the confirmation, the same model has produced the market analysis, the strategy, the backtest interpretation, the order parameters, and the persuasive paragraph explaining why this order is a good idea. Your approval is downstream of the agent's framing of its own work. The failure mode is not a rogue agent bypassing the click — it is a human confidently confirming a coherent, well-narrated, wrong plan. The vulnerability is cognitive before it is technical.

## Approval is not review

I have watched this happen outside finance. At work, we ran an agent pipeline for a client where the agent drafted operational changes and a human approved each one from the agent's own summary. Approval rates sat near 100% for weeks — which sounds like quality until we changed one thing: the confirmation screen stopped showing the agent's summary and instead rendered the raw parameters of the action next to the applicable policy limits. Rejections appeared almost immediately. The humans had not been reviewing actions; they had been reviewing the agent's prose about its actions. Same people, same actions, different framing — different decisions.

So here is my claim, stated plainly: an approval only counts as a control if the thing being approved was evaluated by something that did not produce it. For agent-prepared financial actions, that means a control plane that sits between proposal and confirmation and is deterministic where the agent is probabilistic — hard position and notional limits, an asset allowlist, per-day order-count caps to catch looping, anomaly checks against the account's own history, and a confirmation screen rendered from the raw order payload, never from the assistant's narrative. The agent proposes; a separate, boring, rule-based layer disposes; only then does the human click.

Two hedges, because the seed evidence deserves them. I have not independently tested Revolut's connectors, and its safety model is documented by Revolut itself — no external regulator or auditor has evaluated it publicly that I can find. It is possible some of these controls exist server-side. But the launch materials assign the review burden to the user, and the terms make user-credentialed instructions authorized by definition, which tells you where the accountability lands when the coherent-but-wrong plan clears.

Steal this if you expose any regulated or irreversible action through an agent — trading, payments, infrastructure, HR. Write the policy checks as code that runs on the proposed action's raw parameters, with no access to the agent's explanation. Render the confirmation from those raw parameters plus the policy verdict. Then run one adversarial test before Friday: have the agent produce a plausible-sounding proposal that violates a limit, and see whether your reviewer catches it from the narrative alone. If they approve it, your human-in-the-loop is a signature, not a safeguard — and you found that out in a test instead of an incident.

**A confirmation click is not a control when the proposer also wrote the pitch — approve orders against independent policy checks, not against the agent's own story.**

---

## Quantized retrieval needs slice-level evals

- URL: https://andymental.com/drops/quantized-retrieval-needs-slice-level-evals
- Type: post
- Published: 2026-07-21
- Updated: 2026-07-25

> NVIDIA's 4-bit Nemotron 3 Embed keeps 99% of aggregate retrieval accuracy. An average across 16 public tasks says nothing about which of your query classes absorbed the loss — slice before you swap.

On July 16, 2026, NVIDIA released Nemotron-3-Embed-1B-NVFP4, a 4-bit variant of its embedding model, reporting an average RTEB NDCG@10 of 72.00 versus 72.38 for the BF16 parent across 16 public tasks — and up to 2x the throughput on Blackwell hardware while retaining more than 99% of BF16 retrieval accuracy.

The consensus read: free lunch. Half the memory, double the throughput, a rounding error of quality. Swap it in.

Here is the problem with that read: 99% retention is an average, and averages do not have addresses. A 0.38-point aggregate loss spread across 16 public benchmark tasks tells you nothing about which query classes absorbed it. Quantization error is not uniformly distributed — it concentrates somewhere. Maybe in rare vocabulary. Maybe in long documents. Maybe in the one query type your compliance team cares about most. The fleet-wide number cannot tell you, by construction.

NVIDIA's own model card is more careful than the headline. It recommends validating on a representative sample of your workload before switching, and it flags known issues with this checkpoint family on vLLM 0.23.x and 0.24.x. Read the calibration detail too: quantization-aware distillation used 20,000 samples, and calibration used 512 CNN/Daily Mail query-passage pairs. News prose. Whatever that establishes, it is not fitness for a private enterprise corpus full of policy clauses, product acronyms, and mixed-language tickets. And a hedge on top: both evaluations are vendor-authored, and neither the throughput nor the accuracy claim has been independently reproduced that I have seen.

## Where the 1% lives

At work, this exact failure pattern cost me a debugging cycle on a client retrieval pipeline — not with this model, but with an earlier compressed-embedding swap. Aggregate recall on our regression set moved by under a point, so it shipped. Two weeks later, support escalations clustered around one document family: policy pages dense with internal acronyms. The compressed model had quietly collapsed several acronym embeddings toward each other. The average never flinched, because acronym-heavy queries were maybe 4% of the eval set. They were closer to 30% of the queries that mattered.

That taught me the gate I now apply, and it is the rule I would defend: a quantized retriever ships only after slice-level evaluation on the deployment corpus, and the slices are named in advance. Mine are rare policy terms, multilingual queries, acronyms and internal jargon, and long documents — plus one more dimension most teams skip: the downstream cost of each slice failing. A 3-point recall drop on chitchat queries and a 3-point drop on regulatory lookups are not the same event, and an aggregate treats them identically.

Steal this before you touch the swap. Pull your real query logs and build four or five slices of about 50 queries each around your corpus's sharp edges — the acronyms, the non-English tickets, the 40-page documents, the terms that appear in fewer than ten chunks. Run BF16 and NVFP4 side by side, compare per-slice recall@k, and weight each gap by what a miss costs downstream. If every slice holds within your tolerance, take the 2x throughput with a clear conscience — this is a genuinely attractive artifact. If one slice craters, you have found where the 99% headline hid the bill, at the cost of an afternoon instead of a production incident.

**An aggregate accuracy number is a press release; a slice table is a deployment decision — quantize after the slices pass, not after the average does.**

---

## Agent swarms need aggregation evals

- URL: https://andymental.com/drops/agent-swarms-need-aggregation-evals
- Type: blog
- Published: 2026-07-20
- Updated: 2026-07-20

> Kimi's swarm docs report 300 subagents and BrowseComp accuracy doubling. The failure literature says most multi-agent breakage happens after the workers succeed — so score the merge, not the branches.

Every branch came back green. A research swarm, one worker per subtask, every subtask completed, one final memo — and the memo cited a policy version that had been superseded months earlier, because only one worker had pulled the current document and the merge step quietly preferred the majority that agreed with each other. Nothing failed. The dashboard was entirely honest. The answer was still wrong.

That is the shape of the problem swarms have now, and the numbers being published make it more urgent, not less.

## What the swarm vendors are actually reporting

Kimi's own Agent Swarm documentation is specific: up to 300 subagent instances deployed simultaneously, more than 4,000 tool calls in a single task, roughly 4.5× faster than sequential execution by one agent, and a 3× to 4.5× reduction in the minimum critical steps needed to reach a goal. On BrowseComp, accuracy goes from 15.9% with a single agent to 33.3% with the swarm. Moonshot shipped K3 on July 16, 2026, and the swarm feature carries forward from the K2.6 release of April 20. There is a GTC session on the catalogue titled "How We Scaled Kimi K2.5", and a widely forwarded founder masterclass doing the rounds this week — though the version that landed in my inbox is a paywalled preview, so its teaser about "3 reward functions that stop agent swarms from collapsing or cheating" is a headline, not evidence. I'm not treating it as one.

Read Kimi's number again, though, because it is more interesting as a confession than as a boast. Doubling from 15.9% to 33.3% means that with 300 agents and 4,000 tool calls, two-thirds of the benchmark is still answered incorrectly. And the docs are careful about where the gains live: large-scale retrieval, batch downloads, 100+ document processing. Parallelisable search. These are self-reported vendor benchmarks on the workloads that suit swarms best, and I could not verify them independently.

## The failure lives downstream of the worker

Set that next to the best empirical work we have on why these systems break. The MAST paper (Cemri et al., arXiv 2503.13657) hand-annotated 150+ multi-agent execution traces — later expanded to 1,600+ across seven frameworks, κ = 0.88 between annotators — and produced 14 failure modes in three buckets: specification and system design (41.8%), inter-agent misalignment (36.9%), and task verification (21.3%).

Do the arithmetic the paper doesn't put in a headline: **58.2% of catalogued failures sit in misalignment and verification — the seam where independently-completed work has to be reconciled, not inside the work itself.** The workers are mostly fine. The joins are where it goes wrong. That's my read of their table, and it's the number I'd defend.

Now put the two facts together. Parallelism is a multiplier on the seam. Going from 3 agents to 300 doesn't multiply the difficulty of any single subtask — each worker's job stays about as hard as it was. It multiplies the number of pairwise contradictions, duplicate entities, incompatible date assumptions and unequal-quality citations the orchestrator has to resolve before it can say anything. You bought a 4.5× speedup on the easy part and a combinatorial increase in the hard part.

## Correcting myself

Earlier this year I argued that agent count is not a production metric — that "we run 40 agents" tells you nothing a latency chart wouldn't tell you better. I still think that's right, and I had the corollary wrong. I treated count as merely uninformative. It isn't neutral: raising it moves the failure surface. Each agent you add shifts probability mass out of the workers and into the merge, so a swarm scaled without a matching aggregation eval doesn't get less accurate in a way you'd notice — it gets more confidently wrong, faster, with a completion rate that looks better than the quarter before.

So here's the rule I now hold myself to: **a swarm does not ship until its eval set contains poisoned inputs, and worker pass-rate is not allowed on the dashboard by itself.** If subtask completion is the only number on the wall, the number is decorative.

Poisoned means deliberately constructed, not sampled. Two sources that flatly contradict each other on the same fact. The same company under three name variants. A superseded document sitting next to its replacement, both plausible. Two branches with a shared dependency where only one of them has the current version. Then score these separately from worker success:

- **Attribution** — does each claim in the output trace to the specific source that supports it, not to a plausible neighbour?
- **Contradiction handling** — when sources conflict, does the orchestrator surface the conflict, or silently pick the majority?
- **Coverage** — what did the merge drop? Minority-but-correct findings are the ones that vanish.
- **Final-decision quality** — graded end to end, by someone who never saw the branch outputs.

![Flow diagram: a task fans out to N workers, every branch completes, the outputs merge, and a final answer comes out. Worker pass rate is what the dashboard scores; final-answer accuracy at the merge is what nobody scores.](/api/media/file/agent-swarms-need-aggregation-evals-seam.png)

At Trigent, the swarm I described at the top was a vendor-security research pipeline for a client — one branch per questionnaire domain, running fine for weeks. The stale-policy answer is what made me sit down and build the poisoned set: a handful of hand-made cases, half an afternoon's work, deliberately contradictory. Worker pass rate on them stayed comfortably high. Final-answer accuracy came in far below it, by a margin big enough that I stopped the rollout. That gap had been invisible for the entire period we were only measuring branches, and the branch numbers had looked good the whole time.

Run this before Friday if you have a swarm in production: take ten of your real tasks, corrupt one source in each, and grade only the final answer. If your worker metrics don't move and your output quality does, you have been measuring the wrong half of your system.

**Parallelism is cheap; reconciliation is the product — score the merge or you're grading a swarm on its handwriting.**

---

## The Hugging Face breach lesson is logs, not local models

- URL: https://andymental.com/drops/hugging-face-breach-lesson-is-logs-not-local-models
- Type: blog
- Published: 2026-07-20
- Updated: 2026-07-26

> An autonomous agent ran 17,000 actions through Hugging Face over a weekend. It was caught because every action was logged and an LLM triaged the anomaly. The newsletters' "self-host your models" moral fixes nothing here.

Hugging Face disclosed the first marquee breach run end-to-end by an autonomous agent, and it is worth reading carefully before the internet finishes deciding what it means. The attack: a malicious dataset abused two code-execution paths in the dataset-processing pipeline — a remote-code loader and a template-injection flaw — to run code on a processing worker, then escalated to node access, harvested cloud and cluster credentials, and moved laterally across internal clusters over a weekend. No human at the keyboard. The agent framework executed **more than 17,000 logged actions.** Detection came from Hugging Face's own anomaly-detection pipeline, which used LLM-based triage over security telemetry to pull the signal out of the noise; then LLM analysis agents reconstructed the full timeline from those 17,000 records in hours instead of days. The company reports no evidence that public models, datasets, or Spaces were tampered with.

By July 20 the newsletter layer had already packaged this as an argument for self-hosting — "running your own models locally just became security hygiene." That is the wrong moral, drawn from the wrong part of the story, and it will send teams to fix a hole the incident didn't expose.

## The entry point was pipeline privilege, not model location

Look at where the attacker got in: a **dataset loader allowed to execute code.** That is a pipeline-privilege failure — an ingestion component granted the right to run arbitrary code on untrusted input — and it has nothing whatsoever to do with whether the model reasoning about that data is hosted or local. Run your model in-house behind that same over-privileged dataset pipeline and you have the identical hole; the malicious dataset still executes code on your worker, still harvests your credentials, still moves laterally. Model location was not a variable in this breach. The "self-host" reading pattern-matches on "AI company got breached" and reaches for the nearest AI-shaped remedy, which is exactly the reasoning error that leaves the actual vulnerability untouched.

The real entry-point lesson is the boring one this site keeps arriving at: untrusted input plus code-execution privilege equals compromise. A dataset is untrusted input. A loader that executes code is code-execution privilege. Bolt them together and location is irrelevant.

## The thing that actually saved them was the log

Here is the part worth building on. What contained this was not a model choice, not a firewall, not an EDR signature. It was that **every one of the 17,000 actions was recorded**, and that record was rich enough for LLM triage to flag the anomaly and then reconstruct the whole attack. The audit log — the thing most teams treat as debugging plumbing — was the primary security control.

And notice why the log worked *here specifically*, because it is a property of the agentic era, not a lucky break. A human attacker performs maybe dozens of hands-on-keyboard actions across a weekend intrusion — sparse, deliberate, easy to keep below the noise floor. An agent performed 17,000. That volume is beyond any human staffing, which is the scary half; but it is also why the behavioral trail was dense enough to detect and reconstruct. Agent-speed attacks generate agent-speed evidence. The same property that makes them fast makes them loud, *if and only if* you were recording.

```mermaid
flowchart LR
    U["Untrusted dataset"] --> L["Code-execution loader (pipeline privilege)"]
    L --> C["17,000 agent actions — credentials, lateral movement"]
    C --> LOG["Every action logged"]
    LOG --> T["LLM triage flags anomaly"]
    LOG --> R["LLM agents reconstruct timeline in hours"]
    T & R --> CONT["Contained"]
    SH["'self-host your model'"] -.->|"fixes none of this"| L
```

## Logging just became a security control, not plumbing

At work, this reframes a conversation I have constantly. Client agent deployments already log every action — for debugging, for traces, for the observability I've argued elsewhere every agent harness needs. Teams treat that log as optional-ish plumbing: nice for diagnosis, first thing cut when storage costs rise or a deadline looms. The Hugging Face incident says: that exhaustive action log is now your primary detection and forensics control for attacks that run at agent speed, because agent-speed attacks are the ones a human SOC cannot watch in real time and can only reconstruct after the fact — from the log or from nothing.

Which changes what "good logging" means. It is no longer enough that actions are logged somewhere. The log has to be **complete** (every tool call, every credential use, every lateral hop — 16,900 of 17,000 leaves a hole an agent will find), **tamper-resistant** (an attacker who can edit the log erases the only control that caught them), **retained** long enough to cover a weekend-plus dwell, and **machine-triageable** — structured so an LLM can separate signal from the daily flood, which is the capability that actually did the detecting.

Steal this reframe for your next security review: stop classifying your agent action log as observability and reclassify it as a detection control, with the requirements that implies — completeness, integrity, retention, and machine-readability, budgeted and owned like any other control. Then fix the entry point the incident actually turned on: audit every ingestion path — datasets, documents, webhooks, uploads — for code-execution privilege over untrusted input, and revoke it. That is the two-line lesson. The self-hosting debate is a distraction from both halves.

**The agent breach was caught by the log and let in by an over-privileged pipeline — instrument everything and de-privilege ingestion; where the model runs was never the question.**

---

## Agent backtests are demos, not track records

- URL: https://andymental.com/drops/agent-backtests-are-demos-not-track-records
- Type: post
- Published: 2026-07-20
- Updated: 2026-07-26

> JPMorgan's eight AI agents beat 60/40 by 0.7pts across 20 years of backtests. The catch nobody in the amplification layer mentions: an LLM was trained on those same decades, so the answer key is partly in its weights.

JPMorgan disclosed that eight AI agents — built on OpenAI and Anthropic models to classify markets into four regimes (Goldilocks, reflation, stagflation, risk-off) and shift stock-bond allocation — beat the classic 60/40 portfolio by about 0.7 percentage points of annualized return, at lower volatility, across roughly two decades of backtests. Every one of the eight beat 60/40 on a risk-adjusted basis, and they also beat the bank's own rules-based regime model. In asset management, 70 basis points of consistent outperformance is genuinely a lot, and the finance-newsletter circuit spent the following week saying so.

To JPMorgan's real credit, the bank labeled these **backtests**, not live performance, and explicitly warned against reading them as proof AI can consistently beat markets. That caveat is the most important sentence in the disclosure, and it is precisely the sentence the amplification layer dropped — one widely-shared briefing rendered it as "AI Investors Beat Traditional Portfolios." So let me sharpen the caveat into the reason it matters, because it generalizes far beyond finance.

**A 20-year backtest is the one arena an LLM-based agent can pass partly from memory.** The models classifying whether 2008 was "risk-off" or 2020 was "reflation" were trained on text written *about* 2008 and 2020 — after the fact, with the outcomes known. The regime label for a historical period, and what worked in it, is effectively in the weights. This is look-ahead leakage, and the unsettling part is that it requires nobody to cheat: the contamination arrives through pretraining, silently, in a test where the "predictions" cover years the model has already read the history of. A human analyst backtesting a rule doesn't know the future at each step; an LLM has, in a real sense, already seen the answer key to the whole exam.

Notice where this bites hardest, because it is the counterintuitive part. The most impressive-sounding result — the agents beating JPMorgan's own rules-based model — is also the most contaminated comparison. The rules model has no memory of the test period; it applies a fixed formula blind. The LLM agents plausibly do have memory of it. So "the LLM beat the rules engine" may be measuring, in part, "the system that read the history beat the system that didn't." That is not a track record. It is a demo of a very specific capability: reconstructing known regimes from training data.

None of this means the work is worthless or dishonest — JPMorgan's framing is careful, and regime-aware allocation may well add value live. It means the backtest cannot tell you whether it will, because the one thing a backtest must guarantee — that the model didn't know the future — is the one thing an LLM trained on the past cannot guarantee.

And this is now everyone's problem, not just finance's. With surveys putting a majority of banks piloting agents, every enterprise AI pitch increasingly arrives wearing a backtest-shaped eval: "our agent would have caught this fraud," "would have flagged this outage," "would have won this deal" — evaluated on historical cases the model may have trained on. The finance example is just the most legible instance of a trap that's spreading.

So my first diligence question for any historical agent eval, in any domain: **does the model's pretraining window overlap the test window?** If yes, the comparison is compromised until proven otherwise, and the burden is on the vendor to show they controlled for it — held-out data after the training cutoff, or a decontamination method with receipts. "It beat the baseline on twenty years of history" is a sentence that should now trigger the question, not end it.

Steal this for your next agent eval review: demand at least one evaluation on data created *after* the model's training cutoff — genuinely unseen — and weight it above any historical backtest, however impressive. A model that performs on truly out-of-sample data has shown you something. A model that performs on its own training era has shown you its memory.

**A backtest an LLM can pass from memory is a reading comprehension test, not a track record — grade the agent on the future it hasn't seen, not the past it was trained on.**

---

## Open weights are artifacts, not announcements

- URL: https://andymental.com/drops/open-weights-are-artifacts-not-announcements
- Type: post
- Published: 2026-07-20
- Updated: 2026-07-25

> Kimi K3's "largest open-weight model ever" title rests on a July 27 promise — no repo, no license file, no self-host path — and the coverage repeating it is already miscounting the basics.

Moonshot launched Kimi K3 on July 16, 2026: a 2.8-trillion-parameter mixture-of-experts model — 896 experts, 16 active per token, a 1M-token context — with open weights promised by July 27. Those specifics are corroborated across Tom's Hardware, VentureBeat, and Simon Willison's write-up, so take them as solid.

Now the part nobody puts in a headline. As of July 20 there is no repository, no license file, and no way to self-host K3. "Largest open-weight model ever" currently rests entirely on an eleven-day promise. The live API — model id kimi-k3 — proves the model is real and capable. It proves nothing about openness.

Watch how fast the promise hardened into fact inside one newsletter cycle. Chris Arden's July 17 issue got it exactly right: the weights are a stated target for July 27, not a shipped fact — no repository, no license file, no deploy guide. Three days later, the GenAI Works "Atlas" issue of July 20 leads with "owning your model now beats renting it," using K3 as the exhibit, and calls it "the first open model past 3 trillion parameters." It is 2.8 trillion. Moonshot's own "world's first open 3T-class system" framing does the rounding work, and the newsletter carried the rounded number as a count. Meanwhile the AI GTM Collective (July 17) quotes K3's API pricing as a flat $3 in / $15 out, omitting the $0.30-per-million cache-hit input tier that changes the economics for exactly the high-volume workloads a procurement team would model.

## The three-artifact test

My rule for calling a model open is deliberately boring — three checkable artifacts, no interpretation required: weights you can download, a license file you can read, and a deploy guide or reference inference stack you can run. K3 today scores 0 of 3.

The license is the sharpest edge. K2 shipped under a modified MIT license, and that precedent is quietly doing load-bearing work in every "K3 will be open" analysis this week. But an assumption is not a license. A modified MIT with, say, a revenue-threshold clause changes enterprise calculus entirely — and the K3 license text that would settle the question does not exist yet. Also unverified: Moonshot's claimed 2.5x scaling-efficiency gain over K2 is the vendor's own benchmark, reproduced by nobody.

This is not an abstract complaint about journalism. At work last week, a client's platform-selection matrix crossed my desk with "open weights" as a scored criterion — and K3 already sitting in the column with full marks, sourced to a newsletter. I rewrote the row to 0/3 with a note: re-score on July 27 against artifacts, not coverage. If the weights land with a clean license, the score changes in thirty seconds. If they land late, or the license carries surprises, that matrix just avoided anchoring a platform bet on a press release. Either way the decision now waits on evidence that can be linked, not prose that can be quoted.

None of this is a prediction that Moonshot will miss the date. July 27 may deliver everything, and a downloadable 2.8T open-weight model would be genuinely significant. The point is narrower and holds regardless: between announcement and artifact, "open" is a marketing adjective, and procurement decisions made inside that gap are decisions made on vibes.

Steal the test. Before any document you own says a model is "open," require three links in the row: the weights, the license file, the deploy guide. No link, no score. It takes two minutes, it is embarrassingly easy to apply, and this week it would have caught a 3-trillion-parameter model that does not exist and a price quote missing its cheapest tier.

**Open is a repository you can clone and a license you can read — until July 27 ships both, Kimi K3 is a very large API with a press release.**

---

## Provenance attests the factory, not the code

- URL: https://andymental.com/drops/provenance-attests-the-factory-not-the-code
- Type: post
- Published: 2026-07-20
- Updated: 2026-07-25

> Five poisoned @asyncapi npm versions shipped through the project's own release pipeline — every one with valid provenance. Attestations prove which workflow built a package, not that the inputs were clean.

On July 14, 2026, attackers published five poisoned versions across four @asyncapi npm packages — roughly 2 million weekly downloads between them — and every single one carried valid provenance. The compromise facts are corroborated independently by Microsoft's July 15 write-up, StepSecurity, and Socket: no npm token was stolen, no install scripts were used, and the malicious versions were republished within about ninety minutes of takedowns through the project's own GitHub Actions release pipeline via npm's OIDC trusted-publisher flow.

The attack path, per StepSecurity, is almost insultingly simple: push access to the repos' `next` branch. From there, the legitimate release workflow did exactly what it was designed to do — built what was on the branch and published it with a signed attestation. The attestation is technically true. The authorized workflow really did build those packages.

Sit with the uncomfortable part. Trusted publishing was the ecosystem's answer to stolen npm tokens, and here it worked precisely as designed — and that success is what signed the malware. The weakest link moved from publish tokens to branch-push credentials, and the attestation format has no field to express whether the commits feeding the build were legitimate. Provenance attests the factory. It says nothing about what was wheeled onto the factory floor.

## Two heuristics died in one incident

The second casualty is the "no postinstall scripts, so it's safer" rule that many scanners — and many agent-driven security checks — still lean on. This loader fires at import time, the moment a build or CI job first requires the module. Install-time scanning sees a clean package; the payload waits for `import`.

Which brings this to my corner of the industry. An agent-driven dependency updater instructed to "only accept attested builds" would have waved all five versions through. So would an enterprise pipeline gating on npm provenance or SLSA attestations. At work, this stung directly: a client hardening exercise I was involved in had listed "require provenance on all npm dependencies" as a completed control, and I had signed off on it as meaningful. On July 17 I reread that line in the report and had to send the honest follow-up: the control we marked green would have passed this attack without a murmur. That email cost me some pride and the client a re-scoped backlog, and it was the correct trade.

One hedge on the payload itself: capability descriptions diverge by vendor. Most write-ups describe a multi-stage botnet loader dubbed "Miasma"; BleepingComputer's headline framed it as credential-stealing. Treat any specific capability list as vendor-reported analysis, not established fact. The mechanism — valid provenance, import-time execution, branch-level compromise — is the multiply-corroborated part, and it is the part that matters for your pipeline.

My rule coming out of this: provenance is an authentication signal, never an integrity signal, and no single signal gates an install. Attestations still earn their place — they kill the stolen-token class of attack dead. But they must be paired with controls that look at the code, not the certificate: version-diff review on update, behavioural or static scanning that runs at import semantics rather than install semantics, and a cooling-off delay so you are never the first fleet to execute a fresh version.

Steal this for your next platform review: add a 48-to-72-hour quarantine for newly published versions of high-blast-radius dependencies, attested or not, and require an automated diff summary before any agent or renovate-bot merges the bump. Then grep your security docs for the phrase "verified provenance" and check what each occurrence actually protects against. Anywhere it stands alone as an integrity control, you have the same green checkbox I had to retract.

**A valid attestation proves the right factory built the package — not that the right code went in; gate on the certificate and the contents, or you have automated misplaced trust.**

---

## Portability must include the learning layer

- URL: https://andymental.com/drops/portability-must-include-the-learning-layer
- Type: blog
- Published: 2026-07-18
- Updated: 2026-07-26

> Everyone negotiates portable model endpoints — and leaves the evals, graders, trace history, and feedback labels trapped in one vendor's control plane. The model is the swappable part; the learning layer is the asset.

Efi Pylarinou's agentic-finance review this week made an argument that deserves to escape the fintech niche it landed in: the compounding asset in enterprise AI is not the model, it is the **learning layer** — the retained evaluations, contextual intelligence, and production feedback a firm accumulates. It's a strategic claim, not a benchmarked one; I couldn't find a comparative study proving ownership of these assets produces better outcomes, so hold it as a well-reasoned thesis. But it names precisely the mistake I watch enterprises make in every vendor negotiation, and the mistake is expensive.

Here is the mistake. Enterprises have finally learned to demand model portability — portable endpoints, no proprietary API lock, the ability to swap providers. Good. And then they leave everything that actually accumulated value trapped in one vendor's control plane: the prompts, the graders, the trace history, the feedback labels, the failure clusters, the retrieval annotations. They negotiated the right to move the *engine* and forgot to negotiate the right to move everything the engine *learned*. That is portability of the swappable layer only, which is barely portability at all.

## The layer that compounds is the one nobody exports

Think about what a mature AI deployment actually is, a year in. The model is the same model anyone can rent. What's yours — what took a year of production to build and cannot be bought — is the accumulated intelligence around it: the **eval cases** drawn from your real work, the **reviewer decisions** that turned tacit expert judgment into graded criteria, the **failure clusters** you discovered the hard way and now test against, the **retrieval annotations** that taught the system which of your tables is trustworthy, the **trace outcomes** showing what actually worked in production, and the **version history** tying every behavior change to the data that caused it.

That is the asset. OpenAI's own disclosure about its internal data agent — 3,500-plus users, 600 petabytes, 70,000 datasets, and an architecture that is mostly human annotations, lineage, and organizational context — is the same lesson at scale: the model was the least interesting part; the accumulated context was the product. Your enterprise version is smaller but identical in kind. And nearly every organization I see has it sitting inside a vendor platform, in that vendor's schema, with no export path — which means the "portable" model you negotiated is a body you can move and a memory you can't.

```mermaid
flowchart TD
    M["Model / endpoint"] -->|"portable — everyone negotiates this"| OK["Swappable in a day"]
    L["Learning layer"] --> E["Eval cases + reviewer decisions"]
    L --> F["Failure clusters"]
    L --> R["Retrieval annotations"]
    L --> T["Trace outcomes + feedback labels"]
    L --> V["Version history"]
    E & F & R & T & V -->|"trapped in vendor control plane"| X["Not portable — and it's the actual asset"]
```

## Real portability is portability of the memory

The reframe: **you are only as portable as your least-portable layer, and that layer is almost always the learning layer.** A model you can move but whose evals, graders, and feedback history you cannot is a false freedom — switching vendors means starting the year of accumulation over, which no one will do, which means you are locked in exactly as hard as if the model itself were proprietary. The lock just moved to where you weren't looking.

So the negotiation has to move with it. Treat the learning-layer artifacts as **exportable, vendor-neutral, and owned** — the same way a serious enterprise treats its data. Concretely, that means contract terms and architecture that guarantee: eval suites and graders live in *your* repo in an open format, not the vendor's console; trace and feedback history is exportable in bulk, on demand, in a documented schema; retrieval annotations and reviewer decisions are yours by ownership clause, not the platform's by default; and there is a written retention and egress path you have actually tested. Same discipline I've argued for eval records, semantic layers, and open-model serving — this is the umbrella over all of them: the compounding assets belong to you, in formats you control, or they don't really belong to you.

At work, the diagnostic I now run in every platform decision: I ask to see the *export*. Not the import, not the demo — the export. Show me eval suites, trace history, and feedback labels leaving the platform in an open format I could load into a competitor tomorrow. Vendors building for genuine portability can show it. Vendors building a moat get visibly uncomfortable, because the un-exportable learning layer *is* their lock-in strategy, and a customer who asks to see the exit is a customer they're about to lose leverage over. That discomfort is the most useful signal in the whole procurement.

Steal this clause set for your next AI contract: (1) evals and graders stored in customer-owned repos, open format; (2) bulk export of trace and feedback data on demand, documented schema, no fee; (3) explicit customer ownership of annotations and reviewer decisions; (4) a tested egress runbook, refreshed annually. Then run the drill once a year — actually export the learning layer and confirm it loads elsewhere — exactly as you'd rehearse a serving-provider exit. Portability you haven't exercised is a clause, not a capability.

**You negotiated the right to move the model and forgot the year of learning that made it useful — own the evals, traces, and feedback in open formats, or you're locked in wherever the memory lives.**

---

## Agent-built apps need expiry dates

- URL: https://andymental.com/drops/agent-built-apps-need-expiry-dates
- Type: post
- Published: 2026-07-17
- Updated: 2026-07-26

> SaaStr migrated 10 years off Marketo for $14 and replaced a $10K app in an hour. When building gets this cheap, app count outruns the obligations each carries — so every agent-built app needs an owner and an expiry.

SaaStr's latest operating dispatch is a genuine jaw-dropper: about 300 campaigns and ten years of member data migrated off Marketo for roughly $14.28 in model cost, against agency quotes of a year and $100K; a $10,000 app replaced in an hour; three humans running 21-plus production agents while each spends 8 to 12 hours a day building. First-party claims I can't independently verify — but even discounted heavily, the direction is unmistakable, and it produces a wall SaaStr names precisely: when building gets this cheap, the question stops being "can we build this?"

Here is the wall, stated as the arithmetic that will hit every organization that adopts these tools. **Lower creation cost raises application count faster than it lowers the obligations each application carries.** A $14 app is cheap to *create*. It is not cheap to secure, patch, support, classify, and eventually retire — those costs are the same whether the app took a year and $100K or an hour and pocket change. Cheap creation × unchanged lifecycle burden = an exploding population of small apps each dragging a full maintenance tail. That is not a productivity miracle if you don't manage it. It is shadow IT with a compounding interest rate.

Picture where this goes in eighteen months without a control: an engineering leader inherits dozens — then hundreds — of agent-built utilities, each of which someone made in an afternoon and forgot. No named owner. No patch path when a dependency gets a CVE. No data classification, so nobody knows which ones touch PII. No usage telemetry, so nobody knows which are still used. And no shutdown trigger, so none of them ever die. Every one is a small, permanent liability, and their number only grows, because creating the next one is still cheaper than auditing the last one.

The fix is a single discipline borrowed from good infrastructure practice: **every agent-built app launches with an expiry date.** Not a guess at when it dies — a *review* date, at which the app must produce evidence of continued use and a living owner, or it gets retired automatically. This inverts the default. Today an app created in an hour lives forever by inertia; nobody ever schedules its funeral. With an expiry, permanence must be *re-earned* on a schedule, which means the graveyard clears itself and only the apps that matter survive their first review. The expiry is the forcing function that converts "cheap to build" back into "accountable to run."

Concretely, three fields at birth and one automated job. At creation, every app records an **owner** (a human, not a team), a **data classification** (does it touch anything sensitive), and a **review date**. A scheduled job then does the unglamorous platform work the SaaStr-style dispatches skip: it inventories every agent-built app, maps dependencies and secrets, and on the review date pings the owner — *still using this? still yours?* — and retires, with secrets rotated, anything that can't answer. Inventory, dependency mapping, secrets rotation, and retirement automation stop being afterthoughts and become part of the agent platform itself, exactly as they had to for cloud resources a decade ago.

At work, the emerging pattern I now flag on day one of any "let everyone build with agents" rollout: fund the retirement machinery *before* the creation spree, because the creation spree needs no encouragement and the retirement machinery never gets built after the fact. The team that celebrates a hundred cheap apps and can't name who owns the eleven that touch customer data has not gotten faster. It has accumulated a breach with a delay timer.

**When apps cost an hour to build, the scarce discipline is deletion — give every one an owner and an expiry date, or your $14 miracle becomes your unowned attack surface.**

---

## Dashboards are not the SaaS moat

- URL: https://andymental.com/drops/dashboards-are-not-the-saas-moat
- Type: blog
- Published: 2026-07-16
- Updated: 2026-07-26

> Gartner: $234B of enterprise app spend exposed to "agentic arbitrage" by 2030 — yet wholesale replacement stays unlikely. Both are right. Agents commoditize the dashboard; the governed records underneath stay sticky.

Gartner has been circulating two claims that sound contradictory and aren't. One: roughly $234 billion of enterprise application spending — about a fifth of enterprise SaaS — is exposed to "agentic arbitrage" by 2030, as agents complete tasks across systems and bypass the UX-heavy apps people used to click through. Two: large-scale replacement of incumbent applications remains very unlikely through that same period. (The detailed methodology is gated, so treat the exact figure as a directional Gartner estimate, not gospel.) The apparent tension between "exposed" and "won't be replaced" is the most useful thing in the analysis, because resolving it tells SaaS vendors and buyers exactly where the moat is — and isn't.

The resolution: **agents commoditize the presentation layer, not the system of record.** When a coding agent can spin up a dashboard against your data in an afternoon, the dashboard stops being defensible. But the dashboard was never the product. It was the window onto the product. What sits behind the glass — the permissions model, the transaction rules, the audit history, the integrations, the guarantee that a write actually stuck — is not something an agent recreates by generating a UI.

## What a generated dashboard cannot do

Walk through what happens when an enterprise buyer, newly able to vibe-code interfaces, decides to route around an incumbent SaaS app. They rebuild the dashboard in a day — genuinely, it looks great. Then they meet everything the dashboard was quietly standing in front of. **Permissions:** who is allowed to see and change what, encoded over years, that the app enforced on every action. **Transaction rules:** the business logic that says this discount needs approval, that refund has limits, this state can't follow that one. **Audit history:** the immutable record of who did what when, that a regulator will ask for. **Integrations:** the dozen upstream and downstream systems the incumbent keeps in sync. And **reliable writes:** the boring, load-bearing guarantee that when the record says "paid," it is paid, durably, consistently, under concurrency.

The generated dashboard reads. The system of record *writes and governs*. Reading is now cheap; writing correctly and governing access were always the hard part, and they still are. So the buyer arbitrages the interface — pressures seat pricing, stops paying per-viewer for screens they can generate — while remaining utterly dependent on the incumbent for the governed core. That is "exposed spend without wholesale replacement," exactly as Gartner has it.

```mermaid
flowchart TD
    D["Dashboard / UX layer"] -->|"agent regenerates in a day"| X["Commoditized — seat pricing under pressure"]
    S["System of record"] --> P["Permissions model"]
    S --> T["Transaction rules"]
    S --> A["Audit history"]
    S --> I["Integrations"]
    S --> W["Reliable governed writes"]
    P & T & A & I & W -->|"agent cannot recreate"| M["Defensible core"]
```

## The move for vendors, and the trap

The strategic error a SaaS vendor can make here is to defend the wrong layer — to fight the commoditization of screens by locking down the UI, restricting API access, and treating every generated dashboard as theft. That defends the part that is already lost and neglects the part that is still winnable. The better move is the counterintuitive one: **expose the governed core and let the presentation layer go.** Ship machine-readable semantics — a clean, documented model of what your entities mean and what operations are legal — and governed actions agents can call, each carrying your permissions, your transaction rules, and your audit trail by construction. Then the agent economy runs *on* your system of record instead of around it, and you monetize the thing that's defensible (governed writes, integrity, compliance) instead of the thing that isn't (pixels).

The pricing implication follows: seat-based pricing on interface access is the model most exposed to arbitrage, because interface access is exactly what agents replace. Value tied to governed transactions, integrations, and system-of-record integrity is far stickier, because those are what the generated dashboard still has to phone home to. Vendors clinging to per-viewer seats are defending the moat that already drained.

At work, the buyer version I now advise: before you celebrate replacing a SaaS tool with a generated internal app, list what the incumbent still does that your dashboard doesn't — run the five items above as a checklist. Almost always, you've replaced the window and kept the house, and your leverage is to renegotiate for reading rights, not to imagine you've escaped. And on the build side, if you're a vendor, the question is which of your revenue is priced on screens versus on governed outcomes; the first number is your exposure, and it has a countdown on it.

Steal this framing for either seat: draw the line between "reads" and "governed writes" through your product or your vendor's. Everything on the reads side is commoditizing toward zero and shouldn't anchor a contract. Everything on the writes-and-governs side is the moat — price it, defend it, expose it to agents deliberately. The $234 billion isn't the SaaS industry dying; it's the industry's presentation layer being repriced to what it's now worth, which is much less, while the governed core quietly becomes worth more.

**Agents made the dashboard disposable and the system of record more valuable — stop selling the window, and start charging for the vault.**

---

## Support containment is not resolution

- URL: https://andymental.com/drops/support-containment-is-not-resolution
- Type: post
- Published: 2026-07-15
- Updated: 2026-07-26

> Airbnb's AI Assistant resolved 40%+ of guest issues without a human, up from ~33%, as cost per booking fell 10%. Real progress — but "handled without a human" measures channel exit, not whether the problem got solved.

Airbnb reported that its AI Assistant resolved more than 40% of contacted guest issues without a human in Q1 2026 — up from roughly a third the prior quarter — while cost per booking fell about 10% year over year. It is real, well-executed progress, and travel is a genuinely good domain for it. I'd only flag that I can't see Airbnb's internal definition of "resolved," or a breakdown by issue severity, segment, or later escalation, which is exactly the gap this piece is about — because that number, whatever it means internally, is about to be adopted industry-wide as *the* AI-support KPI, and it measures the wrong thing.

**Containment measures channel exit, not job completion.** "Resolved without a human" tells you the conversation ended inside the bot. It does not tell you whether the guest's actual problem — the double charge, the cancellation, the lockbox that won't open at 11pm — got fixed. A conversation can exit the human channel and still end in an abandoned customer who gave up, a delayed resolution that surfaced as a second contact next day, or a confidently wrong answer the customer acted on. All three count as "contained." Only one is a success. A containment metric cannot tell them apart, and optimizing it rewards the bot for ending conversations, which is not the same as ending problems.

I want to be careful here, because the cynical read — "deflection is just cost-cutting dressed as service" — is lazy and probably wrong in Airbnb's case. The cost-per-booking drop is real and self-service genuinely can be better service: instant, 24/7, no queue. But notice the trap in citing the two numbers together. Lower cost per booking and higher containment moving in the same direction does not prove one caused the other, and it certainly doesn't prove customers were served — a bot that curtly closes hard tickets also lowers cost and raises containment, right up until the churn shows up a quarter later, off this dashboard.

So the fix is not to distrust the AI. It is to measure the outcome the containment number is standing in for. Five metrics turn deflection back into service. **Repeat-contact rate:** did the same issue come back within a week — the single best lie-detector for a "resolution." **Appeal/exception rate:** how often did the bot's answer get overturned when a human finally saw it. **Time to durable resolution:** not time-to-bot-response, but time until the problem stayed solved. **Abandonment:** how many "contained" conversations were actually customers giving up. And **sampled outcome quality:** a human reads a random sample of contained conversations weekly and judges whether the job was truly done. Containment stays on the dashboard — but only next to these, where it can't masquerade as resolution.

At work, the pattern I keep meeting: a support leader hits the deflection target, the quarterly slide is green, and the repeat-contact rate, the silent abandonments, and the downstream remediation costs live in three other systems nobody joins to the AI metric. The bot looks like a triumph and the customer experience is quietly eroding one contained-but-unsolved ticket at a time. When we finally joined containment to repeat-contacts, a third of the "resolved" issues had a customer back within seven days. The deflection rate hadn't lied; it had answered a question nobody should have been asking alone.

Steal this reframe before your next AI-support review: for one week, take every "resolved without a human" conversation and check whether that customer contacted you again within seven days about the same thing. That single number — resolved-and-stayed-resolved — is worth more than the containment rate it corrects. Put it on the same slide, and watch which way the two lines actually move.

**A conversation that avoids a human isn't a problem that got solved — measure whether the customer's job is done, not whether the bot ended the chat.**

---

## Preference data governance outlives training methods

- URL: https://andymental.com/drops/preference-data-governance-outlives-training-methods
- Type: link
- Published: 2026-07-14
- Updated: 2026-07-26

> DPO made alignment simpler than RLHF — one classification objective, no reward model. But it simplified the algorithm, not the data. Whose preferences you rewarded still needs governing, whatever the training method.

An explainer making the rounds contrasts the two ways models learn to be helpful: RLHF, the InstructGPT-era pipeline that trains a reward model from human preferences and then optimizes against it with reinforcement learning, and DPO, the 2023 method that collapses that whole loop into a single classification-style objective — no separate reward model, no RL. DPO is a genuinely elegant simplification, and as more enterprises consider customizing model behavior on their own expert data, it lowers the bar to doing so. Worth understanding, and worth linking.

But the simplification is being read the wrong way, and the wrong reading is about to produce a lot of unaccountable internal models. DPO makes the **optimization** simpler. It does nothing to make the **preference pairs** neutral. Both methods learn the same thing from the same place: whatever humans preferred, in the data you collected. The algorithm changed; the governance question did not move an inch.

Here is that question, stated the way it will actually bite. Six months after a team fine-tunes an internal assistant on "expert preferences," a model version ships that behaves differently — more cautious here, more assertive there — and someone in a review asks: *whose judgment became policy?* If the answer is a shrug, you have automated an opinion you can no longer name, defend, or reproduce. The elegance of the training method is irrelevant to that failure; the method was never the thing you needed to remember.

What you needed to record, and what neither RLHF nor DPO records for you, is the governance around the data. **Provenance:** who supplied each preference, and were they representative of the users the model serves? **Reviewer segmentation:** did three senior people's taste quietly become the house style for everyone? **Disagreement records:** when annotators split on a pair, how was it resolved, and by whom — because the resolution *is* the policy. **Policy versioning:** which behavioral standard was in force for this batch of data. And **regression tests:** the fixtures that catch when a new preference set silently shifts behavior you'd already accepted.

None of that is optional, and DPO's convenience makes it more urgent, not less — because the easier the training loop, the more teams will run it, and the more unaccountable models will accumulate in the enterprise. The demos in these explainers — Zephyr and friends — win benchmarks like MT-Bench, and I'd add the standard caveat: a benchmark win, judged by a model or a generic panel, does not predict safety, usefulness, or cultural fit inside your specific domain. Your assistant's job is to be right for your users, not to top a leaderboard someone else defined.

Steal this before your next fine-tune, RLHF or DPO alike: treat the preference dataset as the governed artifact, not the trained weights. For each batch, record provenance, reviewer segments, disagreement resolutions, and the policy version, and attach a regression fixture set — filed exactly where your eval records live. Then the day someone asks "whose judgment is this and why did behavior change," you answer from a file instead of a shrug. The weights are downstream of the data; govern the data.

**DPO simplified the recipe, not the ingredients — you are still training a model on somebody's opinion, so write down whose, before it quietly becomes your policy.**

---

## Agent harnesses are release platforms

- URL: https://andymental.com/drops/agent-harnesses-are-release-platforms
- Type: blog
- Published: 2026-07-13
- Updated: 2026-07-26

> Microsoft Foundry advertises 11,000+ models — but the number that ships agents is one control plane over identity, tools, evals, traces, and rollback. Model choice is one release input; the harness is the product.

Microsoft has been marketing Foundry with the numbers you would expect: more than 11,000 models, over 1,400 business-system connections, used across 80,000-plus organizations including 80% of the Fortune 500. Impressive, unverifiable from the outside, and — I'd argue — the least important figure in the announcement. Because tucked among the model counts is the phrase that actually explains why enterprises adopt a platform like this: **one control plane** spanning agents, tools, knowledge, identity, policy, evaluations, and deployment.

That is the tell. When a platform vendor leads with catalog size and closes with control plane, the control plane is the product. The 11,000 models are the loss-leader.

## Model choice is one release input among seven

Here is the mistake I watch enterprise teams make, at length, this year: they spend weeks comparing model scores — this one's 3% better on that benchmark, this one's cheaper per token — as if selecting the model were the hard part of shipping an agent. Then they "win" the model bake-off and discover the actual work hasn't started, because an agent's production behavior barely comes from the model. It comes from everything around it: the grounding (what data it retrieves), the tools (what it can call), the permissions (what it may touch), the orchestration (how steps chain), and the policy (what it must refuse). Swap the best model in the world into an ungrounded, over-permissioned, untraced harness and you get a confident liability. Swap a mid-tier model into a disciplined one and you get something you can ship.

Model choice is one release input. There are at least seven, and the other six are where deployments live or die.

## What a real harness has to make true

The reframe that helps: **an agent harness is a release platform, and the unit it releases is the whole agent, not the model.** Judge any harness — Foundry, a rival, or the one you're accidentally building yourself out of glue scripts — by whether it makes an agent *version* four things at once.

**Testable.** A version can be run against an eval suite before it ships, and promotion is gated on passing — the same eval-record discipline any AI artifact needs, enforced by the platform instead of by hope. **Observable.** Every run leaves a trace you can inspect — which tool it called, what it retrieved, why it decided — because an agent you can't watch is one you can't debug or defend. **Promotable.** A tested version moves from staging to production as one artifact, config and prompts and tool contracts and model together, not as six manual steps someone forgets. **Reversible.** When it misbehaves, one action rolls the whole version back — model, prompt, tools, policy — because partial rollbacks are how 2 a.m. incidents become 6 a.m. incidents.

```mermaid
flowchart LR
    subgraph V["Agent version = one artifact"]
        M["model"]
        G["grounding"]
        T["tool contracts"]
        P["prompts + policy"]
        I["identity scopes"]
    end
    V --> TE["Testable — eval gate before promote"]
    V --> OB["Observable — full trace per run"]
    V --> PR["Promotable — staging→prod as one unit"]
    V --> RE["Reversible — one-action rollback"]
    C["Model catalog 📚 11,000"] -.->|"one input, not the product"| V
```

Notice what that list is: it is continuous delivery, restated for agents. Testable, observable, promotable, reversible are the exact properties CD gave software a decade ago — and the reason enterprises will pay platform prices is that building those four properties bespoke, per agent, is the unglamorous work nobody scopes and everybody underestimates. That is the same lesson as loops-are-release-artifacts, now at platform scale: the thing that runs without you has to be engineered like software, and a harness is what industrializes that.

## The catalog can be a liability

One sharp caveat on the 11,000 models, because a huge catalog is not free. Every additional model is additional change surface — a version that behaves differently, a new failure mode, another thing to re-evaluate. Optionality only helps if routing and evaluation are automated; otherwise "11,000 models" means 11,000 ways for an untested swap to reach production. The catalog is an asset exactly to the degree the harness disciplines it, and a liability to the degree it doesn't. Buy the control plane; treat the catalog as inventory to be governed, not a feature to be celebrated.

At work, the diagnostic I now run first: a team shows me their agent platform and I ask to see a version get promoted — eval gate, trace, one-click rollback. Teams with a real harness demo it in five minutes. Teams that "chose a great model" show me a model dropdown and a deployment process held together with runbooks and courage. The second group hasn't chosen a platform; they've chosen a model and postponed the platform.

Steal this evaluation order: score any agent platform on the four release properties first — can I test, observe, promote, and reverse a full agent version as one artifact — and score the model catalog last, because you can swap models but you cannot swap a missing release process. If the demo is a benchmark chart, ask to watch a rollback instead. What they do next tells you whether you're buying a platform or a catalog.

**The model is a component; the harness is the product — buy the thing that makes an agent testable, observable, promotable, and reversible, because that is what actually ships.**

---

## Data gravity is becoming agent governance

- URL: https://andymental.com/drops/data-gravity-is-becoming-agent-governance
- Type: blog
- Published: 2026-07-12
- Updated: 2026-07-26

> Databricks' Genie agents inherit Unity Catalog permissions, semantics, and lineage — the model is swappable, the governed access layer is not. You can win the agent layer by owning the data control plane, not a model.

Databricks has been extending its agent tooling around governed data — Genie agents that must have their data registered in Unity Catalog, draw on structured retrieval, and inherit the catalog's permissions, business definitions, and lineage. The documentation is refreshingly unsexy: an agent gets up to 30 tables or views, conversations are capped, everything registered and audited. (Revenue figures floating around the coverage — a $1.7B AI run rate — I couldn't tie to agent adoption, so leave those out.) The unsexy part is the whole strategic point, and it cuts against the loudest assumption in enterprise AI right now.

That assumption: the frontier model vendor owns the agent stack. Whoever has the best model wins the enterprise, top to bottom. Databricks is making the opposite bet, and it is a good one — **you can win the agent layer without owning the model, by owning the governed control plane the agent has to pass through.**

## Why the sandbox demo lies

Every enterprise agent pilot looks great in a sandbox, and then a specific, predictable thing kills it at production review. The demo agent answered questions beautifully — because in the sandbox it ran on a service account that could see everything, against tables someone hand-picked, with "revenue" meaning whatever the builder assumed. Production is where the questions arrive that the sandbox never asked. *When this agent answers for a regional manager, does it see only that region's rows?* *Whose definition of "active customer" did it use — and is that the one finance signs?* *When it took an action, who is accountable, and where is the audit line?*

Those are not model questions. A better model answers the sandbox questions faster and the production questions no better, because they are questions about **access, semantics, and accountability** — properties of the data platform, not the reasoning engine. An agent that cannot inherit row-level permissions, trusted business definitions, and auditable ownership is a liability that demos well, and that is exactly the agent legal shuts down at scale.

## What actually accretes gravity

Here is the swappability asymmetry that makes this a strategy rather than a feature. The model layer is becoming genuinely fungible — new frontier release every few months, prices falling, and (as the Economist's worldview map incidentally showed) even same-lab models differ enough that you re-evaluate anyway. Swapping models is a Tuesday.

Now try to swap the other layer. Your row-level and column-level permissions, encoding years of who-can-see-what decisions. Your lineage graph, showing which numbers derive from which sources. Your business semantics — the negotiated, politically-load-bearing definition of every metric that matters. Your sharing and credential rules. None of that is portable, because it *is* your organization's accumulated agreements about its own data, written down and enforced. That is data gravity, and in the agent era it has become **agent governance**.

```mermaid
flowchart TD
    M["Model layer — swappable, commoditizing"] -.->|"new one every quarter"| M2["Just re-point the harness"]
    G["Governed control plane"] --> P["Row/column permissions"]
    G --> S["Business semantics — the trusted metric definitions"]
    G --> L["Lineage + audit"]
    G --> C["Credentials + sharing rules"]
    P & S & L & C --> A["Agent inherits access, meaning, accountability"]
    A --> W["Production-passable agent"]
    G -->|"cannot be swapped — it IS your data agreements"| W
```

The platform that already holds that governed layer can offer agents that pass production review by *inheriting* it — the agent sees exactly what the asking user may see, uses the metric definitions finance already blessed, and writes an audit line by construction. A best-of-breed model bolted onto ungoverned data can't match that at any capability level, because the gap isn't intelligence. It's provenance.

## The counterpoint, and the buyer's move

Be fair to the other side: model quality still matters, and a governed platform with a mediocre model loses to a governed platform with a great one — which is precisely why Databricks-style plays emphasize broad model choice on top of the governed layer. The bet is not "the model is irrelevant." It is "the model is the swappable part, the governance is the sticky part, so own the sticky part and rent the rest." That is a more durable position than owning a model that a competitor leapfrogs next quarter.

At work, the buyer implication I now flag: enterprises are shopping for agents as if they were shopping for models — comparing reasoning benchmarks — when the question that determines production success is whether the agent inherits their existing governance or forces them to rebuild it. A team that has invested years in a data catalog, permissions, and semantic definitions should weight "does this agent layer sit on my governed data" far above "whose model is 3% better on some eval." The first is where deployments live or die; the second is a number that changes next month.

Steal this evaluation reframe: for any enterprise agent platform, score it first on governance inheritance — does it enforce your row/column permissions, use your blessed metric definitions, and produce an audit trail *without a rebuild* — and only second on model quality (which you can swap anyway). If a vendor's demo runs on a god-mode service account and hand-picked tables, you are watching the sandbox that lies. Ask to see it run as a restricted user, against your semantics. The ones that can are selling agent governance. The rest are selling a chatbot that will fail the review you haven't scheduled yet.

**Models are the tenants; governed data is the landlord — own the control plane the agents must pass through, and you own the agent layer without ever shipping a model.**

---

## On-prem AI is an architecture decision

- URL: https://andymental.com/drops/on-prem-ai-is-an-architecture-decision
- Type: post
- Published: 2026-07-11
- Updated: 2026-07-26

> JPMorgan picked SambaNova to run AI inference behind its own walls. Good story — and a trap if your org reads it as "on-prem equals secure." Location is not a control. Choose it for named constraints, not as a vibe.

JPMorganChase has picked SambaNova to run production AI inference on-premises — installing SN40 and SN50 systems to keep the workload behind the bank's own walls, with full control over the data and an auditable trail — announced the same week SambaNova closed the first $1 billion of a Series F at an $11 billion valuation. For a bank, this is a defensible, probably correct call. I can't verify the workload details, latency, or cost from the announcement, and it doesn't matter for the point, because the point is what a thousand enterprise architecture reviews will *take* from this headline: "on-prem equals secure." That inference is wrong, and it is expensive.

**Physical location is not a security control.** It feels like one — the servers are in your building, behind your badge readers, so surely the data is safer. But every actual security property is orthogonal to where the box sits. Workload isolation, identity and access management, key management, observability, patch cadence, model governance — an on-prem deployment that gets these wrong is *less* secure than a well-run cloud deployment that gets them right, and it has the added charm of being your sole responsibility. Moving a poorly-isolated agent from someone else's data center to yours changes the postal address of the incident, nothing more.

What on-prem genuinely buys is different, narrower, and worth naming precisely, because only named benefits justify the cost. It buys **control over the data path** — you can prove, to a regulator, that customer data never crossed a boundary, which for a bank is a real and sometimes mandatory property. It buys **latency and capacity control** — no shared-tenant contention, no surprise throttling during someone else's launch week. And it buys a certain kind of **auditability** — an evidence trail you own end to end. Each of those is a specific requirement met by a specific property. None of them is "security" as a vibe, and none of them arrives automatically from the location; you still have to build the isolation, the IAM, the key management, on your own now-heavier shoulders.

The trap is the default. At work, the bank and insurer architecture reviews I sit in increasingly reach for on-prem reflexively — it *feels* conservative, it *feels* safe, and nobody gets fired for keeping data in the building. But when I ask which specific threat, latency target, audit requirement, or failure mode the on-prem choice addresses, the honest answer is often "it just seems safer." That is not an architecture decision. It is an anxiety decision, and it transfers a mountain of reliability and lifecycle responsibility — uptime, patching, hardware refresh, capacity planning, the whole operational burden the cloud was quietly carrying — onto a team that has not budgeted for it, in exchange for a security benefit that was never location-dependent.

My rule for the on-prem conversation: **the location is a conclusion, not a premise.** Start from the constraints — this data legally cannot cross this boundary; this workflow needs sub-X latency guaranteed; this regulator requires an owned audit trail — and let the deployment model fall out. If the constraints point on-prem, as they plausibly do for JPMorgan's most sensitive workloads, deploy on-prem *and* build every control that location doesn't provide. If the constraints don't point there, on-prem is just cloud with worse economics and your pager number attached.

Steal this decision record: for any on-prem AI proposal, one page, three columns — the constraint (named threat, latency number, or audit rule), the architecture property that satisfies it, and the owner accountable for operating that property. If a row's constraint column says "feels safer," delete the row. If the page is empty, you don't have an on-prem requirement; you have an on-prem habit.

**Putting the model in your building doesn't secure it — it just makes every control you forgot to build your problem instead of someone else's. Choose location for constraints, then earn the security separately.**

---

## AI BOMs need behavioral inventory

- URL: https://andymental.com/drops/ai-boms-need-behavioral-inventory
- Type: blog
- Published: 2026-07-10
- Updated: 2026-07-26

> The AI bill-of-materials push borrows the SBOM playbook: inventory your models and datasets. Necessary, insufficient. An agent's risk lives in what it may do — the BOM must record tools, scopes, and policies too.

ReversingLabs has been making the case that AI supply-chain transparency should borrow the SBOM playbook, framed inside an "xBOM" catalog of a dozen bill-of-materials types — one of which, the ML-BOM, tracks models, datasets, parameters, training processes, and dependencies. The instinct is exactly right, and overdue: the software industry learned the hard way that you cannot secure what you have not inventoried, and AI systems have shipped for years with no equivalent of the ingredient list. (I'll flag that the specific field sets aren't a ratified standard yet, and some download-count figures in the surrounding coverage I couldn't independently confirm.)

But the ML-BOM as described inventories the wrong noun for the risk that actually keeps me up. It catalogs **components** — the model, the data, the libraries. That is necessary. For a traditional ML model it might even be sufficient, because a classifier's blast radius is its output. For an *agent*, it is a fraction of the picture, because an agent's risk is not in what it is made of. It is in what it is allowed to *do*.

## Components don't cause the incident; behavior does

Run the incident-response test, which is the only honest way to evaluate a BOM. Something bad happened — an agent leaked data, took a destructive action, followed a poisoned instruction. Your response team pulls the ML-BOM and learns: the model, its version, its training lineage. Useful, and completely insufficient to reconstruct what happened. Because the harmful action was produced by a combination the component inventory never recorded: which **prompt** framed the task, which **tools** the agent could call, what **permission scope** those tools carried, which **retrieval source** fed it context, which **policy version** was live, and which **deployment** of all of the above was running at 03:14 when it went wrong.

The model artifact is identifiable. The behavior that caused the incident is not reconstructable. That gap is the entire problem, and a components-only BOM leaves it wide open. It is the security equivalent of a car recall that can name the engine model but not which brake firmware was flashed to which vehicles — accurate about the parts, useless about the fault.

```mermaid
flowchart TD
    subgraph C["Component BOM — what it's made of"]
        M["Model + version + lineage"]
        D["Datasets"]
        L["Libraries"]
    end
    subgraph B["Behavioral BOM — what it's allowed to do"]
        T["Tools + permission scopes"]
        P["Prompts + policy versions"]
        R["Retrieval sources"]
        V["Eval results at deploy"]
        K["Revocation paths"]
    end
    C -->|"identifies the artifact"| Q["Incident: agent took harmful action"]
    B -->|"reconstructs the action"| Q
    Q --> A["Only both together answer 'what happened and how do we stop it'"]
```

## From compliance inventory to incident-response graph

The upgrade is to make the AI-BOM a **runtime manifest**, not a static parts list — one that links each component's identity to the behavioral envelope it ran inside. Concretely, alongside model and dataset, the record carries: **tool grants and their scopes** (this agent could call these APIs, with these permissions), **prompt and policy versions** (the exact instruction and guardrail configuration in force), **retrieval sources** (what it was allowed to read), **eval results at deploy** (what was tested and passed before this version went live — the same eval-record discipline that should travel with any AI artifact), **deployment version** (so a specific incident maps to a specific configuration), and **revocation paths** (how to kill this agent's authority, and who can).

Do that and the BOM stops being a compliance checkbox and becomes an incident-response graph. When something goes wrong, the response team traverses it: this deployment ran that prompt with these tool scopes reading that source under this policy — here is the exact blast radius, and here is the revocation lever. That is the difference between "we know which model it was" and "we know what it could touch and how to stop it," and only the second one ends an incident.

At work, the version I keep meeting: an organization proud of its model registry — every checkpoint versioned, lineage tracked, the component BOM immaculate — and, in the same breath, unable to answer which tool permissions their production agent holds or which prompt version is live, because those live in a deployment config nobody treats as inventory. They built a beautiful parts list for a machine whose danger is entirely in its wiring. The first real incident is where they discover the registry was answering a question they didn't have.

## Steal this

Extend your AI-BOM schema with five behavioral fields per deployed agent, today, even by hand: tool grants and scopes, prompt/policy version, retrieval sources, eval-result reference, and revocation path. Then run the tabletop: pick a hypothetical harmful action and time how long it takes your team to reconstruct the full behavioral context and revoke the authority using only the BOM. If the answer involves "we'd have to ask the team that deployed it," your inventory is a museum label, not an incident tool. The component list tells you what you built; the behavioral manifest tells you what it can do to you, and only the second one is worth anything at 3am.

**A model registry names the ingredient — an agent BOM has to record the recipe, the permissions, and the recall number, because the incident lives in what the system was allowed to do, not what it was made of.**

---

## Attack agents test their own tooling

- URL: https://andymental.com/drops/attack-agents-test-their-own-tooling
- Type: post
- Published: 2026-07-09
- Updated: 2026-07-26

> Sysdig watched an autonomous attacker unit-test its own exploit payloads, read the failures, and correct before escaping a container. When the attacker iterates, signature detection chases a target that rewrites itself.

Sysdig's threat team documented an incident from late May that reads like a preview of the next few years of security. An attacker exploited an unauthenticated marimo-notebook WebSocket (CVE-2026-39987), then drove a fully automated kill chain — enumerate the Docker socket, probe a kernel privilege-escalation path, spawn a privileged container, break out to the host, read the shadow file and SSH keys, replay a stolen Kubernetes token to dump the cluster's secret store. No human at the keyboard. An LLM harness ran the whole thing.

The detail I want to isolate — and the one that should reshape detection engineering — is this: Sysdig reports the agent **unit-tested its own payloads.** It fired canary probes, read the errors when they failed, corrected its tooling, and only then advanced. (Separate ransomware figures floating around the coverage — record counts, payload totals, a 31-second recovery — I couldn't trace to a primary report, so set those aside.) The behavioral core is verified enough to build on.

Sit with what self-testing does to signature-based defense. Detection tuned to known payload hashes assumes the attacker's tools are roughly fixed — catalog the bad binaries, match, alert. But an attacker that reads its own failures and rewrites its payloads mutates *faster than you can catalog*. Every canary that trips your rule is not a defeat for the attacker; it is a **free unit test**, telling it precisely what your defense catches so it can route around it on the next iteration. You have accidentally built the attacker a CI pipeline, and you are hosting it.

So the individual payload is the wrong unit of detection. **The loop is the signature.** What an autonomous attacker cannot hide — because it is the essence of what it does — is the *behavior* of iteration: probe, observe the error, interpret, modify the tool, retry, at machine cadence and machine regularity. A human attacker probes irregularly, gets tired, takes breaks, follows hunches. An agent grinds a tight observe-test-correct loop with a metronome's consistency. That rhythm is a fingerprint no payload rewrite erases, because erasing it would mean abandoning the strategy.

Which points detection at a different target. Instead of "does this binary match a known-bad hash," ask "is something in this environment running a mechanical test-and-correct loop against my controls?" Canary payloads being probed in sequence. The same privilege path attempted, failed, tweaked, re-attempted seconds later. Error messages being consumed and immediately acted on. Those are behavioral signals that survive mutation — and catching them needs **correlation across planes**: process, network, identity, and — as Sysdig's orchestration-plane framing stresses — the container and Kubernetes control plane, because that is where the modern kill chain actually lands. A single-plane detector sees fragments; the loop is only visible when you stitch the fragments into a sequence.

At work, the security teams I review are mostly still resourced for the old world: excellent payload libraries, hash feeds, signature tuning — and almost no detection expressed as *behavioral sequences across planes*. That was defensible when attackers were humans reusing tools slowly. Against an attacker that regenerates its tools on every failure, a hash feed is a museum. The re-tool is not optional; it is the same shift defenders made when polymorphic malware killed static AV, arriving now one abstraction level up.

Steal this reframe for your next detection review: for your top three attack paths, write one detection that fires on the *loop* rather than the payload — N failed-then-retried attempts on the same privileged action within a short window, correlated with error-reading behavior, across at least two planes. Test it against a benign automation baseline so it doesn't page on your own CI. The payload signature catches yesterday's attacker; the loop signature catches the one that tests its exploits before it uses them.

**When the attacker debugs its own exploit, your job is to detect the debugging — hunt the loop, not the payload, because the payload is already being rewritten.**

---

## Domain experts are model infrastructure

- URL: https://andymental.com/drops/domain-experts-are-model-infrastructure
- Type: post
- Published: 2026-07-08
- Updated: 2026-07-26

> OpenAI is hiring an investment banker — not to bank, but to build rubrics, reference work, and evals. The labs treat expert judgment, written down and maintained, as infrastructure. Most enterprises treat it as a favor.

OpenAI is hiring an investment banker. Not to raise money — to sit inside the Applied AI team and define the quality bar for AI-assisted banking work. The listing asks for at least two years of live transaction experience and pays $185,000 to $205,000 plus equity, three days a week in San Francisco. (The newsletter framing of "$500K to train AI" I could not verify from the listing itself; the equity presumably does the stretching.) Read the actual job description, because it is a template: design realistic hard tasks, create and assess reference work, build grading criteria, diagnose model failures, and map where AI should automate, where it should support, and where humans must stay in the loop — informed by knowing how judgment actually evolves from analyst to director.

Notice what this role is *not*. It is not an annotator seat — interchangeable, per-task, paid by the label. It is a specification and evaluation role: one expert whose tacit judgment gets converted, deliberately and durably, into assets a model team can reuse — rubrics, reference answers, failure taxonomies. The frontier labs have concluded that **domain expertise is infrastructure**, and they are staffing it like infrastructure: permanent, senior, expensive, and upstream of the product.

Now compare how the average enterprise treats the same resource. At work, the pattern I see in almost every deployment: the AI team asks a business expert to "take a look at" outputs — informally, in a meeting, as a favor squeezed between their actual job. The expert says "this one's wrong, the tone is off on that one," everyone nods, nothing is written down in reusable form. Six weeks later a model upgrade lands and the entire exercise repeats from scratch, with a different expert, producing different opinions, because nothing was captured the first time. The organization is paying for expert judgment repeatedly and banking none of it.

The difference between those two postures is not budget. It is the conversion step: tacit judgment becomes infrastructure only when it is written down as **versioned rubrics** (what does "good" mean for this task, in criteria a non-expert could apply), **reference sets** (worked examples of excellent, acceptable, and subtly-wrong outputs — the subtly-wrong ones are the gold), and a **disagreement process** (when two experts split, who decides, and does the rubric get amended). Do that once and the asset survives model upgrades, expert attrition, and vendor switches. Skip it and every evaluation is a séance.

The strategic point for enterprises: **you need this capability even though you buy the model.** OpenAI's banker will define banking quality for OpenAI's products — generic banking, at the industry's center of gravity. Your firm's definition of a good credit memo, your risk appetite, your client conventions live nowhere in that rubric. Buying the model outsources capability; it cannot outsource your acceptance bar. The enterprises that get durable value are building small internal versions of exactly this role — one respected practitioner per critical workflow, given real hours, producing maintained eval assets instead of meeting-room vibes.

Steal the job listing itself: take the OpenAI posting, replace "investment banking" with your critical domain, and use it as the role charter for your best practitioner — even at 20% allocation. First deliverables in month one: ten hard tasks from real work, a one-page rubric, five reference answers including two subtly-wrong ones, and a standing hour to adjudicate disagreements. That is the whole machine. It fits in a repo next to the eval records it feeds.

**The labs are paying bankers to write down what good looks like — your experts already know it; the only question is whether it dies in meetings or compounds in a repo.**

---

## AI productivity is becoming a team metric

- URL: https://andymental.com/drops/ai-productivity-is-becoming-a-team-metric
- Type: blog
- Published: 2026-07-07
- Updated: 2026-07-26

> Figma's 2026 report: developers doing design jumped 44% to 60%; designers coding nearly doubled. Everyone can create everything now — whether the team ships faster depends on review, reconciliation, and decisions.

Figma published its 2026 AI report — 8,403 survey responses and 639 interviews across ten markets — and the numbers describe a boundary dissolving in real time. Developer participation in design work jumped from 44% to 60% in a year. Designers doing development work nearly doubled, from 21% to 41%. Seventy percent of product builders say they use AI for tasks previously outside their skillset, over half say they've abandoned linear handoffs, and 41% now say AI has changed how their *team* works — up from 7% two years ago. Self-reported, survey-shaped, causality unproven, as Figma would concede. But the direction matches what anyone inside a product org can see from their desk.

The dissolving boundary is genuinely good news. It is also about to break every productivity dashboard built on individual output — because when everyone can create everything, creation stops being the scarce step.

## Permission to create is not a shared standard

Here is what actually happens when a developer can generate screens and a designer can generate components. The developer ships a flow that is plausible and slightly off-system — spacing tokens improvised, a pattern that almost matches the design system's intent. The designer ships a component that works and subtly forks the codebase's conventions. Neither artifact is wrong enough to reject on sight, which is exactly the problem: each one now requires *reconciliation* — a design-system keeper deciding whether to absorb or correct the drift, a senior engineer deciding whether the fork is harmless. Multiply by everyone on the team, every sprint.

The boundary-crossing that AI enables grants permission to create across roles. It does not grant the shared quality standards, the taste, or the system knowledge that the boundary used to enforce by division of labor. Those still exist in a few heads — and the crossing traffic all routes through them.

## The metrics that hide this, and the ones that don't

Individual output metrics — screens generated, PRs opened, tickets closed — are all rising, everywhere, mechanically. They measure the step AI just made free. Meanwhile the costs that determine whether the *team* ships faster are all coordination-shaped and none of them appear on an individual's dashboard: review queues lengthening because generated artifacts arrive faster than seniors can judge them; design-system drift accumulating as a hundred almost-right variants pile up; and decision latency — my quiet favorite — as the newly-empowered wait for someone to tell them which of three generated options is the real one.

```mermaid
flowchart LR
    A["Everyone generates — screens, code, components"] --> Q["Review queues — seniors judging faster than they can"]
    A --> D["Design-system drift — 100 almost-right variants"]
    A --> L["Decision latency — which option is real?"]
    Q & D & L --> T["Team throughput — the number nobody measures"]
    I["Individual output metrics 📈"] -.->|"all rising, all misleading"| T
```

This is the same bottleneck-shift I wrote about for code generation, now spread across the whole product org: accelerate one station and the queues downstream become the system. The scorecard that stays honest has five columns, all team-level: **cycle time** (idea to shipped, the only speed users feel), **rework rate** (how much generated work gets redone after review), **review load** (hours per senior per week — where the cost silently moved), **escaped defects** (whether speed degraded quality), and **decision latency** (median time from options-exist to option-chosen). Individual generation counts appear nowhere, on purpose.

## What the fastest teams in the data are actually doing

The abandoned-linear-handoff finding is the one worth copying carefully. At work, the teams I see genuinely getting faster haven't just let everyone create — they have rebuilt the *acceptance* path to match. Three moves recur. They put the shared standard in the tools, not the review: design tokens enforced in code, lint rules for the system, agents configured with the conventions — so generated work arrives closer to right. They convert seniors from gatekeepers to samplers: trivial cross-boundary work auto-accepts against the encoded standard, and senior judgment concentrates on the cases that deserve it. And they give decisions a service level: someone owns "which option is real" per surface, with a clock on it, because three unblessed variants is negative inventory.

One client team this year did the opposite experiment for me, unintentionally: they enabled cross-role generation with great fanfare and changed nothing else. Individual output doubled inside a month. Cycle time got *worse* — review became the bottleneck, the design system forked in eleven places, and the two people who could arbitrate became the most oversubscribed calendar slots in the building. The fix wasn't limiting generation; it was funding the acceptance path — encoded standards, sampling review, named deciders — after which the same generation volume started converting into shipped work.

## Steal this

Put the five team-level numbers on one page — cycle time, rework, review load, escaped defects, decision latency — and review them monthly next to the individual output charts everyone already celebrates. When output rises and cycle time doesn't, you have found your reconciliation debt; fund the acceptance path, not more generation. And encode one standard into the tools this quarter — tokens, lint, an agent config — because every convention that lives in tooling is a review that never has to happen.

**When everyone can create, creation stops being the metric — the team that ships fastest is the one that accepts fastest.**

---

## Model worldview belongs in procurement evals

- URL: https://andymental.com/drops/model-worldview-belongs-in-procurement-evals
- Type: blog
- Published: 2026-07-06
- Updated: 2026-07-26

> The Economist mapped 25 frontier models onto World Values Survey axes; same-lab models landed far apart. Don't label models politically — test worldview-sensitive behavior on your own use cases, per version.

Tomasz Tunguz surfaced an Economist analysis this week that ran 25 frontier models through the World Values Survey — the questionnaire that has mapped moral attitudes across roughly 100 countries since 1981 — and plotted them on its two classic axes: traditional-to-secular values, and survival-to-self-expression. The headline finding is the title's joke: the models cluster overwhelmingly in one quadrant, the secular, self-expression corner populated by rich Western countries. Tunguz adds the structural explanation candidates: Common Crawl is about 46% English, and alignment work happens where the labs are.

Two details deserve more attention than the headline. First: **models from the same lab landed far apart.** The analysis reportedly places some sibling models as near-strangers on the map while models from rival labs sit as neighbors — which demolishes the lazy heuristic that vendor identity predicts behavioral disposition. Training data and alignment choices, not the logo, set the worldview. Second, the honest caveat: I could not verify the Economist's full methodology, and everything about this genre is sensitive to prompt framing, survey language, and judge choice. A model answering a values questionnaire is performing a task, not confessing a soul.

Which is exactly why the practical conclusion is *not* "label your models politically." It is narrower and more useful: worldview-sensitive behavior is a product property, it varies between models and between versions of the same model, and almost nobody procures for it.

## Where worldview touches revenue

The reflex objection — "we use it for code, who cares" — is fair as far as it goes. Worldview is invisible in a SQL query. But walk the actual enterprise surface area. **Policy and advice:** a model drafting HR guidance or financial recommendations makes assumptions about authority, risk tolerance, and family structure with every paragraph. **Moderation:** what counts as offensive, blasphemous, or harmless banter is precisely a values call, made thousands of times a day at the edge of your brand. **Customer support:** deference versus directness, individual versus family framing — the difference between a reply that lands in Mumbai and one that lands in Munich. **Localized products:** a recommendation engine that quietly optimizes for self-expression values will feel subtly foreign in markets organized around other priorities.

In each case the failure mode is the same: nothing errors, nothing crashes, and the output is fluently, confidently misaligned with the market it serves. The team discovers it after deployment, from complaints, in the most expensive possible classroom.

```mermaid
flowchart TD
    W["Model worldview — set by data + alignment, not vendor logo"] --> P["Policy & advice drafting"]
    W --> M["Moderation calls"]
    W --> S["Support tone across markets"]
    W --> L["Localized recommendations"]
    P & M & S & L --> F["Fluent, confident, market-misaligned output — no error thrown"]
    F --> E["Discovered post-deployment, via complaints"]
    T["Locally-grounded behavior suite, run per model version"] -->|prevents| F
```

## The procurement move: a behavior suite, not a label

At work, the version of this I now push into every regulated deployment: a **locally grounded behavior suite** alongside the capability evals. Not a politics quiz — a set of forty-odd scenarios drawn from the product's real surface, written with people from the markets it serves. The support reply to a customer invoking family obligation. The moderation call on the religiously-inflected complaint. The advice draft where risk appetite matters. Each scenario with an acceptance rubric written by someone who actually knows the market, not inferred from a survey quadrant.

Three rules make the suite work. **Run it per model version** — the same-lab-far-apart finding means every upgrade is a fresh roll of the dice on disposition, so the suite runs on every version bump, exactly like a regression test. **Track drift, not ideology** — the output is "version N handles authority-framing differently than N-1 in these six cases," a diff a product owner can act on, not a political score nobody can defend in a meeting. **Ground it locally** — a suite written entirely in English by the head-office team measures the head office's worldview twice. If the market matters enough to serve, it matters enough to hire three hours of a local reviewer's time.

The client conversation that convinced me: a team shipping an assistant into three markets ran precisely one values-adjacent test — the vendor's own safety demo — and discovered in production that the model's handling of deference and formality read as condescending in one market and evasive in another. Same model, same prompts, two different failures. The fix was two weeks of suite-building that should have been procurement week one; the damage was a quarter of brand repair.

## Steal this

Before the next model selection or upgrade, write ten scenarios where your product touches authority, religion, family, money, or risk — the WVS's own load-bearing themes — in each market you serve. Get acceptance rubrics from someone local to each. Run them against the incumbent and the candidate, diff the behavior, and file the results with the eval record. One day of work, repeated per version, filed with the eval history so the next upgrade decision starts from evidence. The Economist's map is a conversation starter; your suite is the procurement document.

**Every model ships with a worldview whether you tested for it or not — swing the compass before the voyage, and re-swing it every time the needle gets replaced.**

---

## Open models still concentrate infrastructure

- URL: https://andymental.com/drops/open-models-still-concentrate-infrastructure
- Type: post
- Published: 2026-07-05
- Updated: 2026-07-26

> Together AI raised $800M with 500+ MW of compute committed to serving open models. Open weights end software lock-in — and shift the bargaining power to the few providers who can finance inference at industrial scale.

Together AI announced an $800 million Series C this week, alongside commitments for more than 500 megawatts of compute capacity to serve open models. The pitch includes running open models at 6 to 20 times lower cost than closed alternatives, with customer Decagon cited at a sixfold reduction — vendor-reported numbers whose baselines I can't inspect, so treat the multiples as marketing with a direction. The direction, though, is real: open-model serving is now a business that raises hyperscaler-shaped money and signs power contracts measured in megawatts.

Which is exactly the detail that should recalibrate the most common strategic argument for open weights. The argument goes: choose open models, escape vendor concentration. Half right — and the wrong half is expensive.

**Weight portability and serving portability are different properties.** Open weights genuinely kill software lock-in: the checkpoint is downloadable, the architecture documented, no API contract holds your prompts hostage. If your provider misbehaves, the *model* can leave. But your workload doesn't run on a checkpoint. It runs on capacity — and industrial inference is capital, power contracts, hardware allocation in a supply-constrained market, kernel and compiler engineering to hit competitive latency, scheduling at fleet scale, and the operational muscle to keep p95 flat during everyone's simultaneous launch week. That list is precisely what $800 million and 500 megawatts buy, and precisely what almost nobody else can afford to replicate.

So open weights don't eliminate concentration. They *relocate* it — from the model layer, where licenses used to bind you, down to the serving layer, where physics and finance do. The number of parties that can serve a frontier-class open model at enterprise latency, throughput, and regional coverage is small, capital-hungry, and consolidating. Your freedom to leave is bounded not by the license but by whether a second provider can absorb your workload at your SLA — this quarter, in your region, at a price that doesn't erase the reason you left.

At work, the version I keep meeting: a platform team proudly presents an open-weights strategy as their vendor-risk mitigation, and when I ask the operational question — *if your serving provider doubled prices at renewal, where would this workload run next month?* — the answer is a silence with a GPU shortage inside it. Self-hosting turns out to need hardware they can't get and kernel engineering they don't have; the alternative providers turn out to lack capacity in their region or fall over at their throughput. The checkpoint was portable. The workload was not. That gap is the entire negotiation, and their provider knows it to the basis point.

The procurement fix is to test serving portability the way you'd test a backup: by doing it, before you need it. My rule for open-model contracts now: **the exit is real only if it has been rehearsed.** Run a representative slice of the workload — real prompt mix, real latency bar — on a second serving provider for one week per year. Record what broke, what it cost, and how long the migration took. That rehearsal converts "we could leave" from a slide bullet into a negotiating position, and its cost is trivial next to a renewal negotiated without it.

Steal the checklist version: before signing any open-model serving deal, write down your required latency, throughput, and regions; name the two alternative providers that can meet them today; and put a migration-rehearsal clause and a data-egress cost cap in the contract. If you cannot name two alternatives, you have not chosen an open strategy — you have chosen a closed one with extra steps and better vibes.

**Open weights free the model, not the workload — the lock-in moved down the stack, and the only exit that counts is the one you have rehearsed.**

---

## Forward-deployed engineering is distribution

- URL: https://andymental.com/drops/forward-deployed-engineering-is-distribution
- Type: blog
- Published: 2026-07-04
- Updated: 2026-07-26

> Microsoft committed $2.5B and 6,000 engineers to customer-side AI deployment, days after Amazon's $1B version. The hyperscalers concluded the constraint is integration, not model access — services just became a channel.

Microsoft announced a $2.5 billion commitment this week to what it calls its Frontier Company: roughly 6,000 industry experts and engineers placed alongside customers to redesign and deploy AI-enabled work, on-site, inside the enterprise. Days earlier, Amazon reportedly stood up a $1 billion forward-deployed agent organization of its own. Two hyperscalers, one week, three and a half billion dollars — pointed not at models, not at data centers, but at *other people's integration problems*.

I can't verify how Microsoft's number is allocated, or how many of the 6,000 are net-new versus reassigned. Doesn't matter much. The strategic read is unambiguous, and it confirms what everyone doing this work already knows from the inside: **enterprise AI adoption is constrained by integration, not by model access.** Everyone has the models. Almost nobody has the deployment.

## Why the constraint moved

The stall pattern is remarkably consistent across enterprises, and it has nothing to do with capability. A client has model access, budget, executive sponsorship, a signed platform agreement — and progress dies in the same four places every time. **Identity:** whose permissions does the agent exercise, and who signs off on that answer? **Process redesign:** the workflow the AI is supposed to improve turns out to be undocumented, contested, or three workflows wearing one name. **Data ownership:** the knowledge the agent needs is split across systems owned by people with no incentive to share it. **Production integration:** the pilot that worked in a sandbox meets change control, audit, and the on-call rota.

None of those yield to a better model. All of them yield to competent people embedded long enough to learn the org — which is precisely what the hyperscalers just financed at industrial scale.

## Services as distribution, not labor

The accounting view of what Microsoft announced is a services business: billable people, margins thinner than software, analysts sigh. That view misses the design. Forward-deployed engineering at platform scale is three other things wearing a services badge.

**Distribution.** Every embedded engineer resolves blockers *toward their platform*. The identity question gets answered with the vendor's identity stack; the integration gets built on the vendor's primitives. The deployment labor is the sales motion — the tap system installed free, plumbed to one brewery's kegs.

**Feedback.** Six thousand engineers inside real enterprises constitute the largest product-research operation in the industry. Every stalled rollout, every missing permission model, every integration workaround flows home as roadmap. The platform gets better at exactly the seams where deployments die — an advantage compounding invisibly, deal after deal.

**Expansion.** An embedded team sees the adjacent workflow, the next department, the unbudgeted problem. Land-and-expand with engineers instead of account managers, and with far better information.

```mermaid
flowchart LR
    E["Embedded engineers at customer"] --> D["Distribution — blockers resolve toward the platform"]
    E --> F["Feedback — integration failures become roadmap"]
    E --> X["Expansion — adjacent workflows surfaced in place"]
    D --> P["Platform advantage"]
    F --> P
    X --> P
    P -->|"funds more"| E
```

That is a flywheel, and $2.5 billion is what it costs to spin one at hyperscaler scale.

## The question this asks of everyone else

At work, this lands close to home — deploying AI inside enterprises is the business I am in, so read this section as a competitor thinking out loud rather than a neutral observer. The uncomfortable question the hyperscaler move forces on every independent consultancy and services firm: **does your delivery telemetry compound?**

The hyperscalers' embedded engineers make their *platform* better with every engagement. The default consultancy model makes individual *people* better with every engagement — knowledge that walks out the door with attrition and never accrues to anything reusable. If your hundredth deployment is only marginally cheaper and better than your tenth, you are selling labor against competitors who are selling a compounding asset and pricing the labor at zero when it suits them.

The counter-move exists, but it has to be deliberate: treat every engagement as an instrument. Harvest the recurring blockers into playbooks with names. Turn the identity patterns, the process-mapping templates, the integration scaffolds into artifacts that make engagement N+1 structurally faster. Build the eval harnesses and deployment checklists as products, not project files. Independence remains a real advantage — clients know embedded vendor engineers resolve every fork toward the vendor, and a firm that optimizes for the *client's* stack has a trust position no hyperscaler can occupy. But independence plus non-compounding delivery is a shrinking niche. Independence plus a compounding playbook is a business.

## Steal this

If you buy deployment services: use the hyperscaler math as leverage — integration help is now a competitive giveaway, so stop paying rack rates for commodity embedding, and reserve paid engagements for genuinely independent advice, which just became more valuable precisely because so much "free" help now arrives with a platform attached. Ask any embedded team, vendor or independent, one question up front: *when you leave, what stays?*

If you sell deployment services: audit your last ten engagements for artifacts that survived the engagement. If the honest answer is "the invoices", the hyperscalers just told you what your future margin looks like. Start compounding or start specializing.

**The hyperscalers just priced the real bottleneck at $3.5 billion — integration is the product now, and everyone deploying AI is either building a flywheel or feeding someone else's.**

---

## Agent leverage follows context consolidation

- URL: https://andymental.com/drops/agent-leverage-follows-context-consolidation
- Type: blog
- Published: 2026-07-03
- Updated: 2026-07-26

> SaaStr merged ~10 apps into one codebase and its agents got better with every build; Replit says context is now effectively infinite. Consolidating what agents can see beats adding agents that each see a fragment.

SaaStr published notes from its session with Replit's Amjad Masad, and buried under the demo recap is the most practical agent-architecture lesson of the month. SaaStr consolidated roughly ten applications into one codebase — website, a startup-valuation tool, a pitch-deck grader, an API report card, the lot — and reports that the agent building app number ten benefits from everything it learned building the previous nine. Masad's framing from the platform side: context windows have grown from 16K to over a million tokens in two years, effectively infinite with good compaction, so the constraint has moved. The more you put in one place, the more power you get from global context.

First-party claims, as usual — I can't isolate how much of SaaStr's delivery speed came from consolidation versus simply better models this year. But the mechanism is sound, and it inverts how most enterprises are currently spending their agent budgets.

## The fragment problem

The standard enterprise pattern right now: multiple specialized coding agents, multiple repositories, and a platform team proud of both. One agent for the frontend repo, one for the API repo, one for infrastructure — specialization as strategy. And then every substantive change begins the same way: a human reconstructing, in a prompt, the dependencies and history the agent cannot see. *The API you're changing is consumed by the mobile app in that other repo. The schema you're duplicating already exists in the shared service. We tried this approach last year; it broke billing; the write-up is in Slack somewhere.*

That reconstruction step is the tell. **Specialization does not compensate for fragmentation.** An agent that sees only a fragment of the system produces changes that are locally correct and globally wrong — the deprecated pattern faithfully extended, the schema duplicated with a subtle difference, the dependency broken in a repo it never read. Adding a fourth specialized agent to a fragmented estate adds a fourth entity that needs the same briefing. The bottleneck was never agent count. It is what any agent can *see*.

```mermaid
flowchart TD
    subgraph FRAG["Fragmented estate — agents see slivers"]
        A1["Agent A — frontend repo"] --> B1["local fix, breaks consumer it can't see"]
        A2["Agent B — API repo"] --> B2["duplicates schema that exists elsewhere"]
        A3["Agent C — infra repo"] --> B3["extends pattern deprecated last year"]
        H["Human re-briefs every agent, every change"] -.-> A1 & A2 & A3
    end
    subgraph CONS["Consolidated context — one retrievable graph"]
        G["code + schemas + dependencies + decision history"] --> AG["any agent"] --> O["globally-aware change, no briefing ritual"]
    end
    FRAG -->|"consolidate before you specialize"| CONS
```

## What consolidation actually means

The reflex conclusion — "so, monorepo" — is close but not quite it. A monorepo is one implementation, and for a SaaStr-sized estate it is probably the right one; Google and Facebook proved the pattern long before agents made it newly relevant. But the design goal is more general: **one retrievable dependency and decision graph.** Everything an agent needs to make a globally-aware change — code, schemas, who-consumes-what, and crucially the *decision history*, the why behind the architecture — reachable from a single starting point, without a human courier.

That last item is the one nobody versions. Code lives in repos; decisions live in Slack threads, meeting memories, and departed employees. When SaaStr's agent "remembers how it built the last app", that memory is doing the work an architecture-decision record would do — and most organizations have neither. A plain `DECISIONS.md` per system, one line per irreversible choice with its reason, is the cheapest consolidation move available and the one with the highest agent leverage per hour invested.

Masad's infinite-context point matters here precisely because it removes the old excuse. When windows were 16K, fragmentation was partly a technical necessity — nothing could read the whole estate anyway, so boundaries cost little. At a million-plus tokens with good compaction, the model can hold more of your system than most of your engineers can, and every wall you leave standing is now a choice, not a constraint. The estates that were consolidated for human reasons are collecting an agent dividend nobody priced in.

One more honest note from the same session, because it deserves preservation: SaaStr's valuation tool has been used over a million times; its pitch-deck grader, 4,500. Same codebase, same agent leverage, 200x difference in demand. Consolidation makes agents dramatically better at *building* things — it does nothing to guarantee the things deserve building. The scarce input is still a repeated job worth doing.

## Steal this sequencing

At work, when a platform team asks me which agent to add next, my first question is now: *what does a new agent see on day one?* If the answer involves a human explaining the estate, the next investment is not an agent. Run this instead. First, map the fragments — repos, schemas, config stores, and where decisions actually live. Second, consolidate the highest-traffic seam: merge the two repos whose boundary generates the most cross-briefing, or at minimum build the index that lets retrieval cross it. Third, start the decision log, today, backfilled with the five choices that most often need re-explaining. Fourth — only fourth — revisit agent count. My experience matches SaaStr's arc: teams that consolidate first find their *existing* agents got noticeably better, and the specialized-agent shopping list quietly shrinks.

**Ten agents reading fragments lose to one agent reading everything — consolidate the context, and the leverage arrives before the next hire.**

---

## Vertical AI moats live in workflow frequency

- URL: https://andymental.com/drops/vertical-ai-moats-live-in-workflow-frequency
- Type: blog
- Published: 2026-07-02
- Updated: 2026-07-26

> Harvey reportedly added $100M net-new ARR in one quarter — but durability is predicted by the engagement ratio underneath. In vertical AI, the moat is how often the work returns, and what accumulates when it does.

An analysis of Harvey landed this week pairing two numbers that usually don't travel together: roughly $100 million in net-new ARR added in Q2 alone, and a reported 53% DAU-to-MAU ratio — with an $11 billion valuation on top. The figures are company-reported and the analysis is paywalled, so hold them loosely. But the *pairing* is the story, and it teaches something the revenue headline alone cannot.

A few weeks ago I argued that Cursor's spectacular ARR curve proves demand, not defensibility — revenue velocity is a measure of pull. Harvey's quarter invites the same discipline, and then rewards it differently, because the second number is a different kind of evidence. Revenue tells you customers *bought*. An engagement ratio above 50% tells you the product has become **part of how the work is done, most days, for most of the people who have it.** Those are different facts, and only one of them compounds.

## Why frequency is the moat in vertical AI

In horizontal AI, differentiation keeps collapsing toward the models, and the models are rented — that was the Cursor argument. Vertical AI has an escape hatch, and it runs through frequency. Consider what daily use inside one profession's workflow actually produces.

**Feedback velocity.** A product touched daily by 142,000 lawyers learns about its failures at a rate a weekly-use competitor cannot match — every mis-drafted clause surfaces within hours, and the fix ships against live, domain-specific signal. The quality gap this compounds into is invisible in any demo.

**Accumulated context.** Daily use inside legal matters means the product increasingly holds the matter — documents, drafting history, positions taken, precedent preferred. That context is generated by use, is proprietary by construction, and makes the *next* task easier in a way no rival can cold-start. This is the vertical version of the semantic-layer argument: the data the workflow deposits is the asset.

**Switching cost with teeth.** Leaving an occasionally-used assistant costs a login. Leaving a system that holds your matter context, your templates, and your associates' daily habits costs retraining, migration, and a quarter of partner grumbling — real friction, the kind procurement remembers.

```mermaid
flowchart LR
    F["Daily workflow frequency"] --> V["Feedback velocity — failures surface in hours"]
    F --> C["Accumulated matter context — proprietary by construction"]
    F --> H["Habit + switching cost — migration with teeth"]
    V --> M["Compounding quality gap"]
    C --> M
    H --> M
    R["Fast ARR quarter"] -.->|"proves demand, not durability"| M
```

None of this says the ARR is fake — category urgency and large initial contracts are real forces, and legal AI has both in abundance right now. It says the ARR is the *lagging* indicator. The 53% ratio, if it holds, is the leading one: it is what makes this quarter's revenue likely to still be there, larger, in six quarters. A fast quarter can be bought with sales muscle and timing; a daily habit across a hundred and forty thousand practitioners cannot, and that asymmetry is the whole reason the two numbers deserve different weights.

## The diligence question that separates lookalike vendors

At work, the conversation where this matters is rarely about Harvey itself. It is the enterprise AI lead comparing three vertical vendors whose demos look identical — same model underneath, similar UI, comparable feature lists — and whose contracts differ by multiples. The tiebreaker question I now put on the table: **which recurring decisions does this product own?**

Not "what can it do" — everything can do everything in a demo. Which decisions, made how often, by which roles, now happen *inside* the product? A contract-review tool that owns the first-pass markup on every inbound agreement is a daily habit with accumulating context. A tool that gets consulted when someone remembers it exists is a feature subscription with a renewal risk. The vendors know their own DAU/MAU cold; make them share it, by cohort, for accounts a year old. Reference calls should ask one thing above all: *walk me through yesterday* — not the rollout story, yesterday specifically. If the product doesn't appear in yesterday, it will not appear in the renewal.

The same lens applies to builders. If you are building vertical AI, the strategy question is not which model or how many features — it is which recurring workflow you can own end-to-end, at the highest natural frequency the profession offers. Own the daily thing badly rather than the quarterly thing brilliantly; the daily thing gives you the feedback loop to *become* brilliant, and the quarterly thing never will. Pick the workflow first; the feature list is downstream.

## Steal this

For buyers: add two lines to vertical-AI diligence — vendor DAU/MAU by account cohort, and a written answer to "which recurring decisions does the product own?" Weight them above the growth slide, because they predict what the growth slide will look like in two years.

For builders: instrument your own frequency honestly, weekly, from launch. If your users' return rate is drifting down while your ARR is climbing — big contracts, low usage — you are renting category urgency, and the bill arrives at renewal. Frequency first, revenue follows; the reverse ordering has a short shelf life.

**In vertical AI, revenue is what the moat pays out — the moat itself is the worn staircase: the same feet, every day, cutting grooves no competitor can copy.**

---

## Voice agents should not rebuild WebRTC

- URL: https://andymental.com/drops/voice-agents-should-not-rebuild-webrtc
- Type: post
- Published: 2026-07-01
- Updated: 2026-07-26

> OpenAI published the relay-and-transceiver architecture behind its voice products — and the real lesson is the boundary it draws. Media transport is infrastructure; your voice agent's advantage lives entirely above it.

OpenAI published the architecture behind its low-latency voice serving — the plumbing under ChatGPT voice and the Realtime API, infrastructure that operates at the scale of products with more than 900 million weekly active users. (The newsletter headline implies all 900 million are voice users; the primary source describes the wider product infrastructure, so keep the claim where it belongs.) The design is genuinely elegant: a stateless relay layer that only forwards packets, a stateful transceiver service that owns the WebRTC session — ICE, DTLS, SRTP, lifecycle — with the transceiver's address encoded into the connection handshake so the first packet routes without a state lookup, and geographically distributed relays keeping the first hop short.

Enjoy the elegance, then notice the decision underneath it, because that decision is the transferable part: **even OpenAI did not reinvent media transport.** They kept standard WebRTC behavior at the client boundary and innovated in how packets route inside their own infrastructure. The protocol stack that survives hostile hotel Wi-Fi — jitter buffers, congestion control, NAT traversal, reconnection, device switching mid-call — is decades of accumulated scar tissue, and the largest AI company on earth chose to preserve it rather than replace it.

Now look at what voice-agent teams actually spend engineering cycles on. At work I reviewed a voice startup's sprint history this year: roughly a third of their engineering time had gone into connection recovery, network traversal edge cases, and a homegrown relay layer — transport problems, every one of them solved to a higher standard by infrastructure they could rent. Meanwhile the conversation itself — the thing users experience and the thing their customers were evaluating — handled interruptions badly, lost task state on reconnect, and had no evaluation harness at all. They were digging a private well behind a building with municipal plumbing, while the kitchen sent out undercooked food.

The build-versus-buy boundary in voice is unusually clean, and OpenAI's disclosure is evidence for where it sits. Below the line: media transport, relays, traversal, codec plumbing — mature, commoditized, undifferentiating; buy it, rent it, or use the platform's. Above the line: **turn-taking and interruption behavior** (the thing users call "it feels natural"), **task state** that survives a dropped call and resumes mid-transaction, **domain tools** wired to real systems, **evaluation** of conversation quality against your acceptance bar, and **failure recovery** when the model mishears or the tool errs mid-utterance. Every hour spent below the line is an hour your competitor spends above it.

One honest limit: OpenAI's architecture is evidence for the boundary, not an argument that every workload belongs on its Realtime API. Regulated deployments, on-prem constraints, or telephony-heavy stacks may justify owning more of the transport. But "we own our transport" should be a compliance conclusion, arrived at reluctantly — never an engineering ambition. The teams that treat transport as a craft project are spending their scarcest resource on the layer where excellence is invisible.

Steal this audit for your next sprint review: label every voice-related ticket from the last quarter either *below the line* (transport, connection, media) or *above the line* (conversation, state, tools, evals). If below-the-line exceeds a fifth of the total, you are funding infrastructure that exists, at the expense of product that doesn't. Rebalance before your users — or the next OpenAI blog post — do it for you.

**The road is already built and better than yours will ever be — put the engineering into what you deliver on it.**

---

## Voice agents need time-sliced architecture

- URL: https://andymental.com/drops/voice-agents-need-time-sliced-architecture
- Type: post
- Published: 2026-06-30
- Updated: 2026-07-26

> Thinking Machines' interaction models interleave input and output in 200ms micro-turns — deciding each moment whether to listen, interject, or stay silent. The voice-team lesson: the problem isn't latency, it's turns.

Thinking Machines published its interaction-models work this week, and it is the most architecturally interesting release of the month. The core move: instead of the familiar loop — user finishes speaking, model consumes the utterance, model generates a reply — the model processes input and generates output in continuously interleaved **200-millisecond micro-turns**. Every fifth of a second, it is simultaneously perceiving and producing, and deciding on its own clock whether to stay silent, backchannel, interject, or speak over you. The published configuration, TML-Interaction-Small, is a 276-billion-parameter mixture-of-experts with 12 billion active parameters. Nothing in the config needed fact-checking; what does not yet exist is any independent production benchmark or long-session reliability data, so treat the demos as demos.

But the architectural claim stands on its own, and every voice-product team should sit with it: **human conversation is not turn-based, and no amount of latency optimization makes a turn-based system conversational.**

Watch a real conversation. Listening, planning, and speaking overlap constantly. You start formulating a response mid-sentence — theirs. You say "mm-hm" while they talk, and it changes what they say next. You interrupt, get interrupted, and both recover without ceremony. The turn — the clean alternation chat interfaces assume — is a transcript-keeping fiction. Speech-to-text → LLM → text-to-speech pipelines inherit that fiction structurally: whatever their latency, they are walkie-talkies. Push to talk, release, wait.

This is why the standard voice-team roadmap plateaus. At work I watched a team spend a quarter shaving their pipeline from 800ms to under 300 — real engineering, well executed — and user testing still said the same thing: it feels like a machine. Because the failures users actually noticed were never about delay. The agent could not handle **barge-in** — interrupt it mid-reply and it either plows on or dies awkwardly. It could not act on **partial intent** — a human agent starts pulling up your account the moment you say "I'm calling about my bill", not after your full sentence lands. And it could not **perceive while responding** — everything said during its reply fell into a buffer, to be misunderstood later as a fresh utterance. Every one of those is a turn-boundary problem. The boundary was the product defect, and the latency budget could not buy it back.

The deeper point in the Thinking Machines framing is where timing lives. In the pipeline stack, timing is a *transport* concern — the model itself is timeless, a text function wrapped in increasingly frantic plumbing. The interaction-model claim is that timing must enter **training and inference**: the model has to learn when to speak the way it learns what to say, because in conversation, when *is* part of what. Silence for 600ms after a question is an answer. An interjection placed well is competence; the same words placed late are noise.

My rule for voice-agent evaluation, updated this week: before admiring any latency number, run three probes — interrupt it mid-sentence and watch the recovery; give it a long rambling request and see whether anything happens before you finish; talk over its reply and check whether the overlap registered. Systems fail these in the architecture, not in the config.

Steal this for your roadmap review: if your voice agent's backlog is all milliseconds — faster STT, faster first token, faster TTS — add a line called "turn boundaries" and estimate it honestly. If the answer is "requires a different model class", better to know now than after the third latency sprint. That answer just started shipping.

**You cannot optimize a walkie-talkie into a conversation — the turn boundary is the bug, and it lives in the model, not the plumbing.**

---

## AI spend per engineer is not ROI

- URL: https://andymental.com/drops/ai-spend-per-engineer-is-not-roi
- Type: blog
- Published: 2026-06-29
- Updated: 2026-07-26

> Tunguz's estimates — Anthropic at 2.3x payroll on compute, the median firm at $137 per engineer — will anchor a thousand budget debates. Both numbers measure intensity, not value. Spend needs a work denominator.

Tomasz Tunguz published the comparison that will anchor every AI-budget meeting this quarter: by his estimates, Anthropic spends about 2.3 times its engineering payroll on model compute — roughly $515k per engineer against a $224k fully-loaded salary — while the top tier of software firms spends around 0.4 times payroll, and the median company elsewhere spends a nearly comical $137 per engineer per year. Estimates, stacked on assumptions and external forecasts; none of it independently verifiable, as he'd be the first to note. But the spread is so wide that the error bars barely matter. Somewhere between $137 and $515,000 per engineer, your company has a number.

The mistake coming for that number: reading it as a verdict. The bull version — "spend more, the frontier labs do" — and the bear version — "look at the waste" — share the same fallacy. **Spend per engineer is a measure of workload intensity. It says nothing about value until you divide it by work.**

## The same ratio, four different companies

Take a high ratio — an engineer burning thousands a month in tokens. That is consistent with at least four realities. An engineer running productive autonomous loops overnight, shipping accepted changes while asleep — the dream. An agent stuck in retry loops, burning premium tokens on failures nobody audits — the nightmare wearing the dream's invoice. A team defaulting everything to the priciest model because nobody built a router — pure inefficiency. Or genuine research workloads where expensive exploration is the job — legitimate, but a different budget line than delivery. Identical bills. Wildly different businesses.

Now the low ratio. $137 a year is either a company whose engineers barely touch the tools — adoption failure disguised as thrift — or, occasionally, a disciplined shop routing everything through cheap models with sharp evals. The bill alone cannot distinguish frugality from stagnation, any more than the high bill distinguishes leverage from waste.

```mermaid
flowchart TD
    S["AI spend per engineer"] --> H["High ratio"]
    S --> L["Low ratio"]
    H --> H1["autonomous leverage ✓"]
    H --> H2["runaway retry loops ✗"]
    H --> H3["premium-model default ✗"]
    H --> H4["research workloads ~"]
    L --> L1["disciplined routing ✓"]
    L --> L2["superficial adoption ✗"]
    H1 & H2 & H3 & H4 & L1 & L2 --> D["Only a WORK denominator separates these"]
```

A ratio with four interpretations is not a KPI. It is a conversation starter that arrives dressed as a conclusion.

## The denominator that makes it mean something

What turns spend into ROI is the same thing that turns any input into a unit economic: dividing by output you actually accept. Four denominators do the work.

**Accepted changes.** Not tasks attempted, not PRs opened — changes merged and retained. Cost per accepted change is the closest thing to a unit price for AI-assisted engineering, and it is computable from data you already have: the token bill and the git log.

**Review time consumed.** Every AI-generated change bills twice — once in tokens, once in senior-engineer review minutes. A tool that halves token cost while doubling review load raised your real cost; the spend line just hid it in payroll, where it always hides.

**Escaped defects.** Failure cost is part of the price. If accepted changes regress in production at a higher rate than the human baseline, the cheap tokens were expensive. This is the column that most spend dashboards omit and most CFOs never think to request.

**Displaced work.** The honest credit side: what did the spend make unnecessary? Contractor hours not bought, backlog retired, the migration that happened this quarter instead of never. Hard to measure, worth estimating badly rather than ignoring.

At work, I sat in precisely the meeting this piece is warning about: a CFO looking at a rising token bill, asking whether it was "worth it", and a platform team answering with adoption stats — seats, sessions, tokens. Nobody in the room could say what an accepted change cost, whether review load had moved, or what the defect trend looked like. The bill was declared "probably fine" — a decision made by vibes at five figures a month, renewable monthly. We spent a week wiring the token spend to the git log and the incident tracker. Cost per accepted change turned out to be falling — adoption was working — but one team's agent was generating a third of the spend and a tenth of the acceptances. The ratio said nothing; the denominator found the leak in an afternoon.

## What to build before the budget meeting

The instrumentation is deliberately boring: tag AI spend by team and workflow, count accepted changes from the version-control system, sample review time from PR timestamps, and pull escaped defects from the incident tracker. Four joins, one dashboard, no new tools. Then let the ratios argue: cost per accepted change trending down is a program working; spend rising while acceptances flatline is a leak with a growth curve — whatever the per-engineer headline says, in either direction.

Steal the one-line version for the meeting itself. When the spend-per-engineer slide appears, ask: *per what?* If the answer is "per engineer", the metric is intensity. If the answer is "per accepted change, and here is the trend" — fund that team. They are the ones actually measuring.

**A token bill divided by headcount is a feeling; divided by accepted work, it is a business — bring the denominator.**

---

## Inference moats are financed in megawatts

- URL: https://andymental.com/drops/inference-moats-are-financed-in-megawatts
- Type: blog
- Published: 2026-06-28
- Updated: 2026-07-26

> Groq's $650M raise talks like a utility — 13 data centers, a 200 MW target by 2027. The tell for buyers: chip benchmarks decay in quarters; deployed capacity, financing, and placement decide what you can actually buy.

Groq announced $650 million in growth capital this week, and the language of the announcement is the story. The headline numbers: 13 operating data centers across North America, Europe, the Middle East and APAC; more than 5 million developers served; and a target of 200 megawatts of capacity by the end of 2027. All company-reported, and the capacity plan is a forward statement I can't verify — but notice what *kind* of numbers these are. Sites. Power. Global placement. Long-lived capital. Not a single one is a chip benchmark.

Three years ago, every inference-provider pitch led with tokens per second on a bar chart. This raise reads like a utility's expansion filing. That shift in vocabulary is worth more attention than the dollar figure, because it tells you where the durable competition in inference actually lives — and it is not where most enterprise buyers are still looking.

## Why the benchmark decays and the megawatt doesn't

A silicon speed advantage is real but perishable. Competitors iterate, model architectures shift to favor different hardware, and — as Groq's own December licensing arrangement with NVIDIA illustrates — the boundaries between "chip company" and "cloud on someone's chips" blur within a couple of product cycles. Whatever the leaderboard says this quarter, the safe assumption is that it says something different in four.

Deployed capacity obeys a different clock. A data center is permits, land, transformers, grid interconnection queues, and construction crews — years, not sprints. Power purchase agreements and site financing are decade-shaped commitments. And geography is close to permanent: a customer with EU data-residency obligations does not care that your tokens are fast in Texas. The same week's other headline — Meta reportedly contracting Crusoe for major inference capacity — is the same pattern from the demand side: the hyperscale players are locking up *supply*, not admiring benchmarks.

```mermaid
flowchart TD
    B["Chip benchmark advantage"] -->|"decays in quarters"| C["Competitors iterate, architectures shift"]
    M["Megawatts, sites, financing"] -->|"decays in decades"| D["Permits, grid queues, PPAs, construction"]
    M --> G["Geographic placement — residency, latency"]
    M --> S["Supply under contract — who gets capacity in a crunch"]
    B -.->|"what pitch decks lead with"| X["Buyer attention"]
    S -->|"what actually constrains you"| X
```

The economics compound the point. An inference cloud is a high-fixed-cost, utilization-driven business — the unit margin lives or dies on keeping expensive capacity busy. That is a utility's profit model, and it produces utility behavior: sell long contracts, smooth demand, expand where power is cheap and customers are obligated to stay. The 200 MW target is not a technology claim. It is a statement about how much of that business Groq intends to finance into existence, and $650 million is the down payment.

## The question set enterprise buyers keep skipping

At work, the platform-lead conversations I sit in still start — and too often end — with token speed and price per million. Both matter. Neither answers the questions that will actually bite during the contract term.

**Where does the capacity sit?** Residency, latency, and jurisdiction are properties of buildings, not models. If the provider's EU capacity is one site with a waiting list, your residency story is one incident away from an exception memo. **How is it financed?** A provider running on short-term capital with aggressive expansion targets faces a different set of temptations in a downturn than one on long-dated infrastructure financing — and capacity that gets mothballed mid-contract is a risk no benchmark surfaces. **And who holds priority when demand spikes?** This is the one that separates utility thinking from benchmark thinking. When a model launch or a seasonal surge saturates supply, somebody's workloads get throttled, and it is whoever's contract lacks a capacity commitment. Ask directly: is my throughput reserved, or am I buying from the spot pool with a nicer logo on it?

I watched a client discover the third question the hard way during a previous demand spike — their provider's benchmark numbers were unchanged and magnificent, and their batch jobs sat in a queue behind customers with committed-capacity contracts. The remedy cost nothing but negotiation: a reserved-throughput clause at renewal. The lesson cost a quarter's roadmap, and it would have been free if anyone had asked the surge question during procurement.

## Reading providers like utilities

The practical upgrade is to move a third of your inference-vendor diligence from the model card to the infrastructure disclosures. Providers increasingly publish them — site counts, regions, power targets — precisely because the sophisticated buyers now ask. A provider talking megawatts and interconnects is telling you they intend to be constrained by physics and finance, not fashion; that is, on balance, whom you want to be locked in with.

Steal this clause list for your next inference contract: capacity commitment (reserved throughput, not best-effort), placement guarantee (named regions, with remedies), surge policy in writing (who gets throttled, in what order), and a financing question in the diligence call that you actually ask out loud — what is the capital structure behind the capacity I am renting? A provider that answers crisply is a utility. A provider that pivots to the benchmark slide is a bar chart with a burn rate.

**Benchmarks are weather; megawatts are climate — buy inference the way you buy power, because that is what you are buying.**

---

## Version control needs eval history

- URL: https://andymental.com/drops/version-control-needs-eval-history
- Type: link
- Published: 2026-06-27
- Updated: 2026-07-26

> "Put your AI workflows in git" is this month's PM advice. Correct — and incomplete. A diff tells you what changed, not which model, fixtures, and scores made the old version safe. Version the evidence too.

The advice reaching product managers this week is "put everything you build with AI in git", with the Awesome LLM Apps repository as the on-ramp — 100-plus runnable agent and RAG templates, each advertised as three commands from working, Apache-licensed for forking. (I'll skip the newsletter's time-to-restore anecdotes and star counts, which I couldn't verify and which age by the hour.) The advice is correct, and I have given it myself: git brings diffs, authorship, review, and rollback to prompts, skills, and agent configs — artifacts that today mostly live in Notion pages and chat scrollback.

But having pushed teams to do this for a year, I can report where the advice runs out, because I have watched a team follow it perfectly and still get burned. They versioned every prompt. A regression appeared after a change; they rolled back to the previous file, exactly as the playbook says. The regression stayed. Two days of confusion later, the truth emerged: the old prompt had been "safe" *under a different model version and a different retrieval index*, and nobody could reconstruct which combination had actually passed testing, because the repo stored the prompt and nothing else. The rollback restored the text. It could not restore the world the text used to live in.

That is the general failure: **git versions the artifact, not the evidence.** A prompt diff tells you what changed. It does not tell you which model the old version was validated against, which fixture set it passed, what the scores were, what failure modes were known and accepted, who decided it was good enough, or what stored context — memory files, indexes, conversation state — the version assumes. For code, tests-in-CI carry much of that burden automatically. For AI artifacts, where behavior depends on model × prompt × data × context, the burden is heavier and almost nobody carries it.

The fix is small and boring, which is why it works: every versioned AI artifact travels with an **eval record** — a short structured block, in the same commit, stating model and version tested, fixture set and scores, known failures accepted, reviewer, and compatibility notes for any stored context (including migration steps when the shape of memory changes). Rollback then means restoring a *validated combination*, not a text file. The commit message habit that follows is the one I now insist on: no eval record, no merge — the same rule as "no tests, no merge", one abstraction level up.

This also upgrades the templates themselves. Cloning a runnable agent from a repo like this one gets you to working in an afternoon — genuinely valuable. What the clone does not include is any evidence the template behaves acceptably on *your* inputs. The first thing to add to a forked template is not a feature; it is five fixtures and a baseline eval record, so that every subsequent change has something to be compared against.

Steal this today: add an `EVALS.md` next to your most-edited prompt, backfill one honest entry for the current version, and require an entry per change from now on. One file, five fields, no tooling needed to start.

**Git remembers what you changed — only an eval record remembers why it was safe; version both or roll back blind.**

---

## Agent frameworks magnify install risk

- URL: https://andymental.com/drops/agent-frameworks-magnify-install-risk
- Type: post
- Published: 2026-06-26
- Updated: 2026-07-26

> 140+ Mastra npm packages poisoned in under an hour; the payload hunted crypto-wallet extensions. The agent-era lesson: a poisoned dependency lands in your most credentialed environment — pinning alone is not a control.

The Mastra npm compromise is this month's supply-chain story, and the mechanics are worth restating precisely. Attackers took over a maintainer account with publish rights across the framework's package scope and, in well under an hour, pushed poisoned versions of more than 140 packages carrying a typosquatted dependency. The post-install payload collected system information and scanned for over 160 cryptocurrency-wallet browser extensions. Microsoft attributes the campaign to Sapphire Sleet, a North Korean state group — attribution I'm relaying, not verifying, and the full package list circulated in secondary reporting I could not independently confirm.

npm compromises are almost routine now. The reason this one belongs on this site is *where* it landed: an AI agent framework. And agent projects are, structurally, the worst possible place to catch a poisoned package.

Think about what sits in the environment where an agent framework gets installed. Cloud tokens, because the agent calls APIs. Repository credentials, because the agent commits code. Shell access, because the agent runs tools. Browser profiles — wallets included, as this payload knew — because the agent automates the web. MCP configs full of connector secrets. An agent development environment is a deliberate concentration of authority; that is what makes agents useful. A malicious post-install script executes with all of it, at install time, before your agent has run a single prompt.

Add the velocity problem. Agent development in 2026 is install-heavy by culture — new framework this week, three MCP servers to try, a scaffold that pulls four hundred transitive dependencies. Sometimes it is the agent itself doing the installing, mid-task, because a tool it wanted was missing. Rapid dependency churn plus maximum ambient authority is exactly the combination the ehindero account takeover was pointed at. The attackers needed 45 minutes of publish access; every developer who ran an install in the exposure window donated their whole environment.

So: pinning is necessary and insufficient. Pinning defends against *drift* — it does nothing when the poisoned version is the one you pin, and nothing about what a post-install script can reach. The controls that actually fit the agent era attack the blast radius, not just the version number. **Isolate install and build:** dependencies get installed and compiled in a stage — container, sandbox, separate user — that holds no credentials, then the vetted artifact moves into the runtime. **Scripts off by default:** npm installs run with lifecycle scripts disabled (`--ignore-scripts`), with a short allowlist for the packages that genuinely need them. **No ambient credentials:** tokens are injected per-task at runtime, never resident in the dev environment's shell profile where an installer can read them. **Provenance checks:** signed packages and publish-age gates — a dependency published 40 minutes ago has no business in your build. **Egress policy:** the install stage can reach the registry and nothing else, so a payload that fires has nowhere to phone.

At work, the review question I now ask agent teams first: *if a post-install script ran on your laptop right now, what could it steal?* The honest inventory — usually starting with a cloud token in an env file — is the risk register. One client team ran that inventory after the Mastra news, moved installs into a credential-free container stage and flipped scripts off by default; total cost, one afternoon and one broken package that legitimately needed its build script and got allowlisted.

Steal that afternoon. The framework you install next week will have hundreds of transitive dependencies, one maintainer account between any of them and your environment, and — if this campaign is a guide — about 45 minutes of warning.

**Agent environments concentrate authority on purpose — so quarantine the loading dock, because the parcels are aimed at the vault.**

---

## Outcome pricing forces agent scope

- URL: https://andymental.com/drops/outcome-pricing-forces-agent-scope
- Type: post
- Published: 2026-06-25
- Updated: 2026-07-26

> Sierra prices customer-facing agents by outcomes — a sale, a resolution — not seats or tokens. The pricing model is secretly an architecture review: you cannot bill an outcome you cannot define, attribute, and evidence.

LangChain published a conversation with Sierra's head of product, and the part worth stealing has nothing to do with prompts. Sierra — described as serving most of the Fortune 20 with customer-facing agents, a coverage claim I can't independently verify — prices some of its agents by *outcome*: you pay when the agent achieves the valuable thing. And the discussion draws a sharp line inside that model: a high-value outcome like a completed sale is priced as an outcome; a commodity action like a balance lookup is not pretending to be one.

Everyone reads outcome pricing as a commercial innovation. I read it as an architecture review with an invoice attached — because look at what a vendor must be able to do before it dares bill a single outcome.

It must **define success** precisely enough that a customer will pay against the definition — "resolved the ticket" with an explicit standard for resolved, not "had a nice conversation about it". It must **attribute** the outcome to the agent — did the agent close the sale, or did the customer arrive already decided? It must handle **exceptions** — the return, the reopened ticket, the chargeback — because charged outcomes that un-happen become refund disputes. And it must attach **evidence** to every line item, because enterprise finance does not pay invoices that say "trust us, 4,100 resolutions."

Now notice: every one of those four is a thing your agent program should have anyway, and almost certainly doesn't. Pricing just makes them non-optional. That is why the discipline transfers even where no invoice will ever exist. An internal agent whose success cannot be defined separately from its *activity* — conversations held, messages sent, tasks touched — is unbillable, and unbillable is a synonym for unimprovable: if you cannot say what a success is, you cannot count them, compare versions, or decide whether the thing earns its costs.

The scope pressure is the healthy side effect. Ambiguous, sprawling agent mandates — "help customers with whatever they need" — are precisely the ones where success is undefinable, so outcome pricing quietly forbids them. The vendor is pushed toward narrow, measurable, high-value work, which is — not coincidentally — where agents actually succeed today. The same podcast's title thesis, that the best agents are simpler than you think, is the same force viewed from the build side.

At work, the diagnostic I now run on any agent proposal, internal or vendored, is what I've started calling the invoice test: *write the imaginary invoice.* One line per outcome type, a unit price, and the evidence you would attach to defend each line to a hostile CFO. A product owner who cannot draft that invoice — who can only list activity metrics, conversation volumes, or "engagement" — does not yet have an agent product. They have a chatbot with ambitions. On one engagement this year, the exercise took ninety minutes and deleted two-thirds of the proposed agent's mandate; what survived was the narrow slice with definable outcomes, and it shipped, and it worked, and everyone involved would call that a success — measurably, for once.

Steal the Sierra split too: sort your agent's actions into outcomes (rare, valuable, chargeable-in-principle) and commodities (frequent, cheap, table stakes). Fund and evaluate the two differently. The commodity tier is judged on cost and latency; the outcome tier on definition, attribution, exceptions, and evidence. Mixing them is how dashboards end up celebrating ten thousand balance lookups while the sales the agent was hired for go unmeasured.

**If you couldn't bill it, you can't improve it — write the invoice first, even when nobody will ever pay it.**

---

## Model ensembles buy disagreement, not truth

- URL: https://andymental.com/drops/model-ensembles-buy-disagreement-not-truth
- Type: post
- Published: 2026-06-24
- Updated: 2026-07-26

> OpenRouter's Fusion runs a panel of budget models and reportedly beats frontier systems on research at half the cost. The value is the disagreement the panel surfaces — and a smooth synthesis is where that value dies.

OpenRouter has been publishing results for Fusion, its model-ensemble product: send one prompt to a panel of models in parallel — same tools, web search and all — then have a judge model synthesize the outputs. The headline claim from its benchmark runs: a panel of budget models beat GPT-5.5 and Claude Opus 4.8 across 100 deep-research tasks, at roughly half the cost of the frontier configuration. The company was routing 100 trillion tokens a month when it announced the broader release, so this is not a toy pitch. Caveats first: full prompts, judge configurations, and raw outputs were not published, so neither the half-cost claim nor the battle methodology is independently checkable.

But suppose the numbers hold. The interesting question is *why* an ensemble of cheaper models would beat one stronger one on research tasks — and the answer determines how you should use the technique.

An ensemble earns its keep two ways. **Coverage:** different models retrieve differently, chase different threads, and surface facts their peers miss; the union genuinely beats any member. And **disagreement:** when models trained on different data disagree about a claim, that variance is information — it marks exactly the spots where somebody's error, or a genuinely contested fact, is in play. I made the same argument about Benchling's cross-provider checks recently, and it holds here: agreement across families is cheap evidence of truth; disagreement is a flag worth routing to a human.

Here is the trap, and it sits in the last stage of the pipeline. A synthesizer whose job is "produce one good answer" is optimized to *resolve* disagreement, not preserve it. It rewards consensus and fluent presentation — which means the exact signal you paid the panel to generate gets smoothed into confident prose. Worse: where the panel's members share training-data errors — and they share plenty; these models ate much of the same internet — the ensemble votes unanimously for the mistake, and the synthesis launders it into something that *reads* even more authoritative than a single model's output. The blend can hide a bad ingredient precisely because it is smooth.

To OpenRouter's credit, its judge reportedly produces structured analysis — consensus points, contradictions, unique insights, blind spots — which is the right instinct. Whether that structure survives into what teams actually consume is the question that matters more than the benchmark.

Because at work, the research-agent stacks I review keep making the same downstream mistake: they add models, add a synthesizer, and ship the final prose — with no record of which model contributed which claim, which claims were contested, or what the disagreement rate even was. The team has purchased a disagreement detector and configured it as a disagreement shredder. When a claim later proves wrong, nobody can tell whether one model erred, all of them did, or the synthesizer invented the sentence during the merge.

My rule for ensemble research systems: **the contradictions section is the product; the synthesis is the summary.** Configure the pipeline so every material claim carries its provenance — which models asserted it, which dissented — and route contested claims to review instead of letting the judge settle them silently. If your final artifact has no way to look worse when the panel disagreed more, you have built a very expensive way to feel certain.

Steal this one-line instrumentation: log, per research task, the panel's disagreement rate on material claims — even crudely, contested-claims over total-claims. Track it weekly. It is simultaneously your risk surface, your review queue, and your best supply of eval fixtures, and it costs one field in a log schema.

**An ensemble is worth the tokens when it shows you where the models split — pay for the argument, not for the smoothie.**

---

## Security agents need patch acceptance metrics

- URL: https://andymental.com/drops/security-agents-need-patch-acceptance-metrics
- Type: post
- Published: 2026-06-23
- Updated: 2026-07-26

> OpenAI's GPT-5.5-Cyber found 24 kernel privilege-escalation exploits across 30M+ lines. Impressive — and the wrong scoreboard. Judge a security agent by validated fixes accepted upstream, not exploits generated.

OpenAI expanded its Daybreak security program this week with Patch the Planet, and the headline numbers are designed to travel: GPT-5.5-Cyber worked across more than 30 million lines of Linux kernel code and produced 24 local privilege-escalation exploits plus 8 kernel pointer-leak proofs-of-concept. On the ExploitGym benchmark, the specialized model scores 39.5% against 25.95% for base GPT-5.5. The newsletter version of this story is "AI found 24 kernel exploits", and that version will define how a thousand security programs think about agentic security tools.

Which is a problem, because exploit counts are the *supply side* of security, and supply was never the constraint.

Every security leader already lives this. The scanner era taught the lesson thoroughly: tools that produce findings are abundant; what drowns a program is the pipeline after the finding. Raw discovery counts mix real signal with duplicates of known issues, low-impact corner cases, findings that sit in disclosure backlog for months, and — the expensive category — issues that are technically valid but that no maintainer will ever accept a patch for, because the fix breaks something they value more. A wall of exploit trophies tells you the tool can break locks. Your risk went down only if a door somewhere actually got a better lock.

To OpenAI's credit, the program's structure understands this better than its headline does. Patch the Planet pairs the model with human engineers who review findings before maintainers see them, and the early sprint reporting emphasizes *merged patches* across participating projects — with cURL, Python, and Go among those signed on. That is the right pipeline. But the honest gap remains: I could not determine how many of those 24 kernel exploit proofs have become accepted upstream fixes, because disclosure is still in progress. Until that number exists, the 24 is a capability demo, not a security outcome.

So here is the scorecard I would hold any security agent to — vendor-supplied or homegrown. Five stages, each strictly harder than the last: **validated issue** (deduplicated, triaged as real and material), **coordinated disclosure** (reported through the project's actual process), **accepted patch** (a maintainer merged a fix — the stage where most trophy walls go quiet), **time to merge** (because a fix that lands in fourteen months protected nobody for fourteen months), and **post-fix regression rate** (did the patch hold, or did it break something and get reverted). Discovery counts appear nowhere on that card, and that is deliberate: discovery is the cheapest stage and the only one a model can inflate unilaterally.

At work, the version of this I now see in procurement: a security leader shown a vendor deck with a four-digit vulnerability count, and no way to see duplicates, acceptance, remediation time, or regressions behind it. The question I hand them is one line: *of these findings, how many produced merged fixes, and what was the median time from report to merge?* Vendors with a real pipeline answer with two numbers. Vendors with a trophy wall answer with an architecture slide.

Steal this framing for your own program too, internal tools included: pay bounties, bonuses, and renewals on stage-three-and-beyond events — accepted patches — never on finding counts. Incentives aimed at discovery buy you a backlog. Incentives aimed at acceptance buy you a smaller attack surface, which was the point all along.

**An exploit proves the lock can be picked; security is the new lock, installed and holding — score your agents on doors fixed, not doors opened.**

---

## Generated code shifts the bottleneck

- URL: https://andymental.com/drops/generated-code-shifts-the-bottleneck
- Type: blog
- Published: 2026-06-22
- Updated: 2026-07-26

> Research cited by ByteByteGo: PR completion rose 26% — and review burden just moved to teammates. Coding is 20-30% of engineering time; codegen speeds one station and floods the queues that were always the constraint.

ByteByteGo published an organizational playbook for AI-native engineering, and buried in it are two numbers that, held together, explain most of the disappointment in enterprise coding-assistant rollouts. First: research it cites — attributed to Microsoft — found a 26% increase in completed pull requests per week after assistant adoption, with the review burden simply shifting onto other team members. Second: only 20 to 30% of an engineer's time is coding at all; the other 70 to 80% is review, testing, coordination, and governance.

I could not verify every aggregated figure from the article alone, and its company pilots are unnamed. But the two headline numbers don't need to be precise to make the argument, because the argument is arithmetic: **codegen accelerates a station that was never the constraint.**

## The factory you actually run

Model your delivery pipeline honestly and it is a small factory: write → review → test → integrate → security gate → product decision → deploy. Elapsed time — the thing your customers and your CFO experience — is dominated not by how fast any station runs but by how long work sits in the queues between them. This is the oldest lesson in operations: speed up a non-bottleneck and you do not ship faster; you pile up inventory in front of the bottleneck. In software the inventory is invisible — no pallets on the floor — but it is perfectly real: open PRs aging, test environments contended, security reviews stacked, product decisions pending.

Now hand every engineer a code generator. The write station triples its output. Where does the flood arrive? At review — staffed by the same senior engineers, who now face 26% more PRs, each individually cheaper to produce than ever, and therefore, on average, less deliberated. At test environments that were contended before. At the security gate that was already the slowest queue in the building. And at product decisions, which no model accelerates: someone still has to decide whether the thing should exist.

```mermaid
flowchart LR
    W["Write ⚡ 3x faster"] --> Q1["review queue 🔴 flooding"]
    Q1 --> R["Review — same seniors"]
    R --> Q2["test env queue 🔴"]
    Q2 --> T["Test"]
    T --> Q3["security gate 🔴"]
    Q3 --> S["Security"]
    S --> Q4["product decision 🔴"]
    Q4 --> D["Deploy"]
```

The result is the pattern an engineering manager described to me almost verbatim this quarter, at work: PR volume visibly up, dashboards celebrating, and lead time — commit to production — unchanged. Slightly worse, actually, because review latency had grown. The assistants were working exactly as advertised. The system absorbed the acceleration as queue depth.

## The metric that keeps everyone honest

The vanity metric of this era is "percentage of code written by AI". It measures activity at the accelerated station — the one station we now know is not the constraint. The metrics that measure the system are three: **end-to-end lead time** (idea to production, the only speed that counts), **escaped defects** (whether the flood degraded quality downstream), and **review burden** (hours spent reviewing per engineer per week — the number that reveals where the cost silently moved). If lead time is flat while AI-attributed code climbs, you have not transformed delivery. You have moved the waiting room.

This is also the right lens on ByteByteGo's actual recommendation, which is better than its own framing suggests: autonomous cross-functional pods of 3 to 5 people. Strip the transformation language and what a pod does is *internalize the queues.* Review happens inside the pod, same day, by someone with context. Test environments are owned, not contended. The product decision sits at the same table as the code. A pod is a queue-redesign wearing an org-chart diagram — which is precisely why it works when it works, and why bolting assistants onto an unchanged assembly line doesn't.

## Redesigning the stations the flood hits

The honest to-do list after adopting codegen is mostly not about codegen. Review needs new tiers — trivial changes auto-merged on green checks with sampling audits; routine changes reviewed by any engineer with an agent pre-review attached; consequential changes still earning senior human eyes, now freed from the trivial tier. Test environments need to become self-serve and disposable, because contended staging is a queue with a nicer name. Security gates need risk-based lanes, because pushing 26% more volume through a single gate is how gates become the whole story. And product decision-making needs explicit service levels, because "waiting for a decision" is the queue nobody instruments.

None of that ships in a tool rollout. All of it is org design — which is the playbook's real, half-buried point: the tool spend is the cheap, fast part, and the reshaping of work around the new volume is the actual transformation. It is also the part with no vendor, no demo, and no launch date, which is exactly why most programs skip it and then wonder where the productivity went.

Steal this before your next AI-productivity review: put four numbers on one slide — lead time, PRs merged, escaped defects, review hours per engineer. If the second is up and the first is flat, read the other two aloud and watch the room find the bottleneck. Then fund the queue redesign with the money you were about to spend on more licenses.

**Code generation makes one station faster; delivery is the queues — measure lead time, and redesign the stations the flood actually hits.**

---

## Bounded agents beat executive titles

- URL: https://andymental.com/drops/bounded-agents-beat-executive-titles
- Type: post
- Published: 2026-06-21
- Updated: 2026-07-26

> SaaStr's "AI VP of Customer Success" reportedly cut human hours 70% by owning ~40 recurring, checkable deliverables. The result is real; the title is the risk — agents succeed by being bounded, and titles hide bounds.

SaaStr published the operating claims for QBee, its "AI VP of Customer Success": a 70% reduction in human customer-success hours versus 2025, internal and external, described as a 3x multiplier across roughly 40 recurring event-delivery line items. First-party numbers — I can't verify the baseline workload, the saved-hour arithmetic, or whether quality held constant — but the shape of the system is described in enough detail to learn from, and the shape is the story.

Look at what QBee actually does: proactive check-ins on a schedule, daily status updates, asset tracking, answers to recurring questions. Forty-ish known line items, each with an input, a due date, and a definition of done. That is not a vice president. That is a *portfolio of bounded deliverables* — and that is precisely why it works. Every one of those tasks is checkable: the status update either went out or didn't; the asset either arrived or is chased. The system's success is measurable per item, its failures are visible per item, and a human can own the exceptions per item.

So the result I believe. The title is where I want to plant a flag, because the title is about to become an industry habit, and it teaches the wrong lesson.

**"VP" describes authority; QBee has a checklist.** A vice president's defining property is discretion over an open-ended problem space — reading situations, making judgment calls where no playbook exists, owning outcomes rather than tasks. Nothing in the disclosed system does that, and — this is the point — *nothing in it needs to.* The 70% came from the bounded work: the mind-numbing, recurring, perfectly-specifiable coordination that nobody wanted to do. Calling that a VP is marketing on SaaStr's part, and fair enough, it is their brand. The damage happens downstream, when buyers internalize the title instead of the architecture.

At work I now get the downstream version as a request: "we want an AI manager for client operations." When I ask which recurring artifacts it will own, what the acceptance check is per artifact, which exceptions route to which human, and who approves what before it leaves the building — the room goes quiet, because the title did the thinking. An executive title on an agent invites over-broad autonomy ("a VP wouldn't need approval for that") and makes failure boundaries invisible ("what exactly is it allowed to get wrong?"). The same system named honestly — "the renewal-checklist agent" — triggers exactly the right questions by reflex.

My rule for scoping any agent role: **name it after the deliverables, size it by the checklist, and let the title be boring.** The unit that works is a known deliverable with five properties — input, due date, acceptance check, escalation path, accountable human. Forty of those is a spectacular system, as SaaStr apparently proves. Zero of those plus a C-suite title is a demo with a business card.

Steal this when the "AI manager" conversation reaches you: skip the role discussion entirely and run the line-item exercise first. List every recurring artifact in the workflow, mark which are fully specifiable, attach the five properties to each. The specifiable list is your agent — however long it is. Whatever remains is the human's actual job, and it usually turns out to be the part that deserved the title all along.

**SaaStr's agent wins because it is forty checklists in a trench coat — copy the checklists, and leave the trench coat on the hook.**

---

## AI research needs evidence lineage

- URL: https://andymental.com/drops/ai-research-needs-evidence-lineage
- Type: post
- Published: 2026-06-20
- Updated: 2026-07-26

> KPMG pulled an agentic-AI report after UBS, the NHS, Swiss Federal Railways, and TfL disputed its case studies. The lesson isn't "AI hallucinates" — it's that review without claim-to-source lineage is not a control.

KPMG has withdrawn a report on agentic AI — a report about the benefits of AI, which is the detail the internet will enjoy — after the organizations named in its case studies started disputing them. Per TechCrunch: UBS called claims about AI agents in its investment advice "factually incorrect". Swiss Federal Railways said the journey-planning agents KPMG described are not accurate. NHS Greater Manchester and Transport for London disputed their examples too. The report had been live since October 2025; outside researchers inferred hallucination patterns in the citations, and KPMG says it is investigating. Whether an LLM actually produced the errors is unconfirmed — and for the lesson worth taking, it doesn't matter.

Because here is the uncomfortable part: that report almost certainly passed review. A firm like KPMG does not publish a flagship thought-leadership piece without partners reading it. Competent, motivated humans read fluent, plausible, well-structured prose about named organizations — and approved it. The failure wasn't that nobody looked. It is that **looking is not a control when the artifact gives the reviewer nothing to check.**

A reviewer confronting the sentence "Swiss Federal Railways uses agents to plan and book journeys" has exactly two options: recognize it as false from personal knowledge, or find it plausible. Plausible is what hallucinations *are*. Without a link from the claim to its source — where it came from, the quotation in context, the date it was retrieved — the reviewer is not verifying; they are vibing with extra steps. Eight months of a false report on the website is what vibing signs off.

The pre-AI version of this control existed and worked: fact-checking against footnotes, painful and slow. AI-assisted research quietly dropped it, because generation became free while lineage stayed expensive — the model produces the claim but not the receipt. So organizations bolted the old review step onto the new pipeline and called it governance. Review inherited a job it cannot do.

The control that works is structural: **quarantine consequential claims until lineage is attached.** Every claim that names an organization, cites a number, or describes a deployment enters the draft flagged, and the flag clears only when four things are attached — the source, the quotation in its context, the retrieval date, and the named reviewer who opened that source and cleared it. No lineage, no publication; the sentence gets cut or rewritten as explicitly illustrative. This is boring, checkable, and delegable — the properties a control needs and "a partner read it" lacks.

At work I now put one question to every AI-assisted research deliverable before it goes near a client: *pick any three material claims and show me the source behind each within two minutes.* Teams with lineage pass instantly — the links are in the draft. Teams without it produce the distinctive silence of people realizing that what they reviewed was prose, not evidence. That silence, at a Big Four firm, is now a global news story with four blue-chip organizations issuing corrections.

Steal this threshold for your own shop: a claim is "material" if it names a real organization, states a figure, or would embarrass you in a correction. Material claims travel with receipts or they don't travel. Your review meeting stops asking "does this read well?" and starts asking "is every flag cleared?" — a question a junior person can answer accurately, which is precisely what makes it a control.

**Hallucination is a model behavior; publishing one is a process failure — reviewers can only catch what the artifact lets them check, so ship the receipts with the prose.**

---

## ARR velocity does not prove a product moat

- URL: https://andymental.com/drops/arr-velocity-does-not-prove-a-product-moat
- Type: post
- Published: 2026-06-19
- Updated: 2026-07-26

> Cursor reportedly ran from $2B ARR in February to $4B by June. That curve proves demand like almost nothing in software history — and proves nothing about switching costs once models and interfaces converge.

A newsletter this week assembled nineteen details of Cursor's rise, and the headline pair is genuinely historic: reportedly $2 billion ARR in February, $4 billion within weeks of April. Company-reported figures, no audited filings, and the piece is paywalled — but even with generous error bars, this is among the fastest revenue scaling in the history of business software. My favorite detail is the small one: the product's early feedback loop traces to its first 20 testers. Twenty careful users, then a curve that bends like that.

Now the discipline: state precisely what that curve proves, and what it doesn't.

It proves **demand** — beyond any argument. Developers want agentic coding so badly they will adopt, pay, and expense it faster than procurement can spell it. It probably also proves learning speed: the 20-tester story suggests a team that metabolizes feedback unusually fast, and the enterprise mix reportedly growing underneath suggests real motion beyond prosumers.

What it does not prove is a **moat.** Revenue velocity is a measurement of pull, not of switching cost — and in this category the distinction is unusually sharp, because the core capability lives in the models, and the models are rented. The same frontier models power every competitor; the interface patterns converge within months of anyone's good idea; and the customer's data — their code — famously lives in git, portable by design. A curve like Cursor's is what it looks like when a team executes brilliantly *into* open water. It is silent on what happens when the water gets crowded.

The moat questions are the boring ones the nineteen details don't cover. **Cohort retention:** do February's customers still pay in August, or does each doubling mask churn under acquisition? **Collaboration lock-in:** does the product get harder to leave as more of the team uses it — shared context, review workflows — or is it N individual seats that can each defect alone? **Proprietary context:** does accumulated usage produce something — indexed org knowledge, tuned behaviors — that a rival cannot cold-start? **Migration cost:** what actually breaks if a team switches next sprint? And the brutal one: **model-commoditization resilience** — if the underlying models converge in capability, what premium survives?

At work this shows up as a buying question, not an investing one. A client asked me this spring whether committing to a multi-year enterprise agreement with a fast-growing AI tool was safe "because look at the growth". We wrote the moat questions into the procurement review instead: retention references from year-old cohorts, an exit-cost estimate, a data-portability clause. The growth number never appeared in the final decision matrix — pull tells you the category is real; it tells you nothing about whether *this* vendor is where the category settles. You can be a genuine phenomenon and still be the category's Netscape.

To be fair to Cursor: none of this says the moat is absent. Distribution at this scale can itself become one — default status compounds, and enterprise contracts have their own gravity. The point is narrower and it is about evidence: the ARR curve is consistent with both a durable franchise and a spectacular way station, and buyers, investors, and imitators keep reading it as proof of the first.

Steal this for the next hypergrowth deck you evaluate, as buyer or builder: cross out every growth number and see what evidence remains. Whatever is left — cohort retention, lock-in mechanics, proprietary context, migration cost — is the moat case. If nothing is left, you are looking at demand, which is wonderful, rentable, and shared with every competitor who can call the same model APIs.

**Revenue proves the gold rush is real — the moat question is who owns the river when everyone has a pan.**

---

## Deployment expertise is a full-stack role

- URL: https://andymental.com/drops/deployment-expertise-is-a-full-stack-role
- Type: post
- Published: 2026-06-18
- Updated: 2026-07-26

> SaaStr says the scarce role now is the "agentic deployment expert" — and it's right, if deployment means owning data, evals, workflow, cost, and change end to end. Installing tools fast is the junior version of the job.

SaaStr published a piece arguing that the scarce skill of this phase isn't building AI or prompting it — it is *deploying* it. The "agentic deployment expert": someone who can look at your team, name the eight ways a tool would improve it and the three ways it wouldn't, get it in, train it, and measure the output. Their framing of the eras rings true — 2023 needed engineers because raw models were wild, 2024 was the convoluted-prompt period, and now capable generalists can make agents do real work. Their proposed hiring test is pleasingly concrete: what commercial AI tool did you deploy in the last 30 days, and what measured ROI did it produce?

No labor-market data backs the scarcity claim — it is an experienced investor's opinion, and Craft Ventures was pointing 80% of its investments at AI back in 2023, so the author is talking their book. I'll co-sign anyway, with one large amendment, because the phrase will otherwise get cheapened within a quarter.

**"Deployment" cannot mean installing tools quickly.** That version of the role already exists in every enterprise — the person who has trialed forty tools, demos beautifully, and leaves behind forty licenses and no changed workflow. If "deployment expert" comes to mean that person, the title will be worthless by Diwali.

The version worth hiring — the version that is genuinely scarce — owns a full stack that has almost nothing to do with software installation. **Workflow discovery:** finding the recurring work where an agent actually pays, which requires sitting with the team, not reading the tool's website. **Data readiness:** knowing whether the knowledge base, permissions, and integrations can feed the tool before promising anything. **Evaluation:** defining what "working" means for this workflow and building the small eval that proves it — before rollout, not after complaints. **Security and policy review:** what the tool may read, write, and retain, cleared with the people who own those answers. **Change adoption:** managers enrolled, training mapped to real tasks, the first cohort coached through week three, where usage goes to die. **Operating cost:** the run-rate at real volume, not the pilot invoice. And **sustained outcomes:** still measuring at month six, when the honeymoon metrics have worn off.

That is a full-stack role — the stack just isn't technical. It is data, evaluation, workflow, risk, money, and organizational change, held by one accountable person.

At work, the client pattern that proves the scarcity: plenty of AI-aware staff, real budget, tools everywhere — and when I ask who is accountable for integration, adoption, quality evidence, operating cost, and post-launch support on their flagship deployment, five different people answer, which means nobody. The tools were all deployed, in the installation sense. Nothing was deployed in the ownership sense. The gap between those two sentences is the entire role SaaStr is naming.

My rule for hiring it: extend SaaStr's 30-day test by two questions. What did you deploy — and *what did you decline to deploy, and why?* The expert has a graveyard of tools they evaluated and rejected; the installer has only launches. Then: *what broke in month two and what did you change?* Real deployments always produce this answer. Demos never do.

Steal this scorecard for the job description: workflow discovery, data readiness, evals, security review, change adoption, cost ownership, sustained outcomes — seven lines, each needing a named example from the candidate's last two quarters. Anyone who clears five is worth a premium. Anyone who clears all seven, hire before your competitor's procurement cycle finishes.

**The scarce skill isn't knowing the tools — it's owning everything around the tool that decides whether it worked.**

---

## AI-native teams still need eval ownership

- URL: https://andymental.com/drops/ai-native-teams-still-need-eval-ownership
- Type: blog
- Published: 2026-06-17
- Updated: 2026-07-26

> Codex ships 10-12 surfaces with 2 PMs; Cursor runs 40 engineers on 1. The deleted layers were carrying decisions — acceptance, risk, launch evidence, rollback. Delete the layer, keep the decision, name its owner.

Product Growth published the org charts everyone in enterprise AI will be asked about by Friday. At OpenAI, the Codex group reportedly runs 10 to 12 product surfaces with 2 PMs, 1 designer, and about 40 engineers — surfaces that at a traditional company would each get a squad of fifteen. Cursor reportedly operates 40 engineers with a single PM. The numbers are unverified — reported headcounts from an inside-look piece, not filings — but the shape rings true, and the shape is what your executive team will want to copy.

The article's causal story is right, as far as it goes. When an agent builds a working version of a feature in under an hour, the sequence inverts: build first, evaluate second. Codex reportedly ships about two of every ten things it builds and discards the rest — being wrong got cheap, so the coordination machinery that existed to prevent expensive wrongness (sprints, PRDs, handoffs, dedicated QA) stops paying rent. Fewer translators between intent and code. Smaller teams. Faster loops.

Here is the part the copy-paste version misses, and it is the part that will hurt: **the layers were carrying decisions, not just coordination.** Delete the layer and the decision doesn't disappear — it goes unowned.

## What the handoffs were quietly deciding

Walk through what a "bloated" product process actually adjudicated. The PRD forced someone to write down *what counts as working* — acceptance criteria. The QA gate forced someone to ask *how does this break, and whom does it hurt* — the harmful edge cases. The launch review forced someone to assemble *evidence that it works* before customers found out otherwise. And the release process meant somebody could answer *how do we un-ship this* at 2 a.m. — rollback.

Four decisions: acceptance, risk, evidence, reversal. In the fifteen-person squad they were smeared across roles so thoroughly that nobody noticed they were being made. That smearing was waste, mostly — three meetings to decide what one competent person could decide alone. The AI-native insight is that the *smearing* was the waste. The copy-paste failure is concluding that the *decisions* were.

```mermaid
flowchart TD
    subgraph OLD["Traditional squad — decisions smeared across layers"]
        PRD["PRD ritual"] --> A1["acceptance"]
        QA["QA gate"] --> R1["risk"]
        LR["launch review"] --> E1["evidence"]
        REL["release process"] --> B1["rollback"]
    end
    subgraph NEW["Lean AI-native team — same decisions, named owners"]
        O["explicit eval owner(s)"] --> A2["acceptance = evals"]
        O --> R2["risk = red-team fixtures"]
        O --> E2["evidence = eval results at launch"]
        O --> B2["rollback = rehearsed path"]
    end
    OLD -->|"delete the ritual, keep the decision"| NEW
```

Look at what the lean teams in the article actually have, on the reported description. Two of every ten prototypes ship — which means *somebody is evaluating ten and choosing two*, against some bar. That bar is eval ownership, operating at high cadence with no ritual around it. Cursor's one PM is not doing one-fifteenth of the old PM job; they are doing the concentrated decision core of it while agents and engineers absorb the rest. The layers are gone. The ownership is not — it is denser.

## The flattening I keep getting called after

At work, the call I now get quarterly: a leader read a piece like this one, flattened the AI product team, velocity went up, everyone was thrilled — and then a launch went sideways and the post-mortem could not answer four questions. Who signed off that this was good enough to ship? Nobody; the demo looked great. Who owned the harmful edge cases? QA, except QA was deleted. What evidence existed at launch? A screen recording. Who could roll it back? Eventually, someone, after four hours of figuring out how.

None of that is an argument against the lean shape. It is the punch list for adopting it honestly. The fix in that engagement was one document and one habit: a decision-rights page naming a human owner for acceptance, risk, evidence, and reversal on each surface — and evals as the standing artifact that carries the first three. Prototypes stayed fast and disposable. The two-in-ten that shipped now crossed a bar someone owned. Velocity survived; the next incident had a name attached within minutes, in the good sense.

My rule for the flattening conversation: **you may delete any layer whose decisions you can name and reassign — and no layer whose decisions you cannot name.** If nobody can articulate what the launch review used to decide, you are not ready to delete the launch review, because you will be deleting it blind.

## Steal this

Before copying anyone's org chart, run the four-question audit on each product surface, today, at current headcount: who owns *what counts as working* (and where are those criteria written — "the evals" is the right answer), who owns *harmful edge cases*, who owns *launch evidence*, who can *reverse a ship* and how fast. Write four names per surface on one page. Every blank is a decision currently being made by default — which is to say, by luck. Fill the blanks first; flatten second. The lean teams you are copying did it in that order, even if the article about them leads with the headcount.

**The AI-native teams didn't delete the decisions — they concentrated them; copy the concentration, not just the deletion.**

---

## Bug reports are untrusted agent input

- URL: https://andymental.com/drops/bug-reports-are-untrusted-agent-input
- Type: post
- Published: 2026-06-16
- Updated: 2026-07-26

> Tenet Security injected fake Sentry errors and watched coding agents execute the "resolution steps" with the developer's shell and credentials. The broken boundary: data retrieved for diagnosis became instructions.

Tenet Security published an attack this week they call agentjacking, and it deserves to reset how every team wires telemetry into coding agents. The recipe: inject a fake error event into a project's Sentry using its public DSN — the write-only credential that ships in client-side code by design. The poisoned event contains helpful-looking "resolution steps". A developer later asks their coding agent to check Sentry; the MCP server dutifully feeds the event in; the agent reads the resolution steps and executes them — on the developer's machine, with the developer's shell, holding the developer's cloud credentials.

Tenet reports finding 2,388 organizations with injectable DSNs reachable this way, and says more than 100 agents acted on injected errors in its controlled testing. I could not verify the experiment or the counts beyond Tenet's own disclosure, so hold the numbers loosely. The mechanism, though, needs no audit — any developer who has watched an agent eagerly follow a stack trace knows exactly how this works, and worse: Tenet says explicit system-prompt instructions to distrust external data did not reliably stop it.

## The boundary that broke

Notice what actually failed. Not authentication — the DSN is supposed to be public. Not the MCP server — it faithfully retrieved what Sentry held. What broke is a boundary most agent setups have never drawn: **the line between data retrieved for diagnosis and instructions the agent is permitted to follow.**

A bug report is a hostile document. It always was — support inboxes have carried phishing for decades, and every experienced engineer reads "just run this command to fix it" from a stranger with narrowed eyes. But a human triaging Sentry brings that suspicion for free. An agent brings the opposite: it is optimized to be helpful, the poisoned event is formatted exactly like the help it wants to give, and the moment tools are attached, *reading* becomes *doing*. Telemetry, tickets, logs, code comments, README files from dependencies — anything an attacker can write and your agent will read is a command channel until proven otherwise.

The equation to internalize: **retrieval plus tool authority equals execution.** Whoever can write into what your agent reads holds a share of that authority.

## The pattern that survives this

At work, the setup I now flag in every review is the developer who asks one agent session to "check Sentry and fix what you find" — a single context holding untrusted telemetry, shell access, and live AWS credentials. That sentence is the vulnerability. The fix is structural, not prompt-level, and Tenet's finding that distrust instructions failed is the proof: you cannot ask the model to be the boundary. The boundary has to be architecture.

Concretely, four moves. **Isolate retrieval:** the session that reads telemetry gets no shell and no write access — it produces a summary, nothing else. **Label provenance:** everything crossing into the working session arrives marked as untrusted content, quoted, never inlined as narrative. **Strip active instructions:** anything shaped like a command, a URL to fetch, or "steps to resolve" gets neutralized in the summary — described, not transcribed. **Require a fresh trust decision:** execution happens only after a human (or a separately-privileged step) approves the plan, so the leap from diagnosis to action is a decision, never a default.

That is inconvenient. It is also just the confused-deputy lesson every mature security team already knows, restated for a world where the deputy reads at machine speed.

Steal this today: list every read source your coding agents touch — error trackers, ticket systems, CI logs, dependency docs. For each, one question: *who can write into this?* If the answer includes anyone outside your trust boundary — and for Sentry-style public DSNs it does by design — that source never shares a session with tools that can execute. Two sessions, one seam, human in the middle.

**An error report is a stranger handing your agent a note that says "run this" — sandbox the note before the agent holds the keyboard.**

---

## AI training needs production telemetry

- URL: https://andymental.com/drops/ai-training-needs-production-telemetry
- Type: blog
- Published: 2026-06-15
- Updated: 2026-07-26

> 1,720 Wall Street employees, 53 workshops, one uncomfortable number — 3-5% of power users generate over 70% of usage. Workshops without telemetry don't spread capability; they concentrate it in people who already had it.

GAI Insights published seven months of data from running AI workshops inside financial services: 1,720 employees, 53 sessions across 8 firms, one-to-four-hour formats at introductory and intermediate levels. First-party consultancy data, unaudited, with obvious selection bias — the firms that buy workshops are not a random sample — so hold the exact figures loosely. I will anyway, because one of them matches everything I have seen from the inside of enterprise AI programs:

**3% to 5% of power users generated more than 70% of the usage.**

Sit with what that distribution means for the workshop model. You train a thousand seats. You report a thousand seats trained. And then the actual work — the measurable pull on the tools — comes from thirty or forty people, most of whom, in my experience, were already the ones experimenting before the training budget existed. The detail from the same dataset that seals it: most employees who *described themselves as daily AI users* had never set up custom instructions, projects, or reusable skills. Self-reported fluency, structurally shallow usage.

A workshop without follow-through doesn't spread capability across an organization. It concentrates value in the people who already had the disposition — and issues everyone else a certificate of awareness.

## Why the workshop model produces this curve

Nothing is wrong with the workshops themselves. Awareness is real and necessary: people cannot adopt what they cannot imagine. The failure is treating awareness as the deliverable, when three harder problems remain untouched the moment the session ends.

**Workflow design.** "Use AI" is not a workflow. The analyst who leaves the session inspired still returns to a desk where the actual task — the Monday report, the client memo — has no designed AI-shaped path through it. Power users invent their own paths; that is what makes them power users. Everyone else needs the path drawn, for their specific work, with their firm's approved tools.

**Manager reinforcement.** Usage survives where a manager expects it, asks about it, and treats it as how work is done — the same finding, incidentally, that GAI reports at firm level: adoption tracks how much leadership itself uses the tools daily. A workshop the manager didn't attend produces a behavior the manager doesn't reinforce, and unreinforced behavior decays in weeks.

**Safe practice.** In a regulated firm, the median non-user is not lazy — they are unsure what they are allowed to paste, and the safest interpretation of ambiguity is abstinence. No amount of inspiration fixes a permission question. Only explicit, work-specific guardrails do.

## The operating loop that replaces the calendar

The alternative is to run fluency the way you run production software — instrumented, iterated, coached:

```mermaid
flowchart LR
    I["Instrument — usage telemetry by team and workflow"] --> W["Identify — repeatable wins AND quiet failures"]
    W --> C["Coach — teams at their real work, not in a hall"]
    C --> E["Update — internal examples, guardrails, playbooks"]
    E --> M["Measure — work outcomes, not seats trained"]
    M --> I
```

The load-bearing box is the first one. Without telemetry, a transformation lead can count trained seats but cannot answer the three questions that matter: which workflows actually changed, who stopped using the tools after week two, and where an hour of coaching would move a whole team. With telemetry, the 70/5 curve stops being a verdict and becomes a work list — every team stuck at zero is a coaching target, every power user is a source of playbooks worth harvesting, every drop-off is a signal that a workflow or a permission is broken.

At work I sat with a transformation lead this year whose dashboard was, in its entirety, seats trained and a satisfaction score — 4.6 out of 5, genuinely. When we pulled actual usage from the platform logs, two-thirds of trained users had not touched the tools in the thirty days after their session, and the distribution among the rest was exactly the shape GAI describes. Nobody had lied. The program was simply measuring attendance at the theatre and calling it fitness. The uncomfortable meeting where the real curve went on the wall was also the most productive one of the engagement: the budget moved from more workshops to telemetry, team-level coaching, and a library of harvested power-user workflows — the same money, pointed at the gap instead of the calendar.

My rule for enablement budgets now: **no telemetry, no training spend.** Instrument first, even crudely — platform logs and a monthly pull are enough to start. Awareness sessions are the cheapest part of the program and the only part most programs fund. The expensive parts — workflow design per team, manager enablement, guardrail clarity — are exactly the parts that never fit in a hall of two hundred people, which is why they get skipped, which is why the curve stays at 70/5.

## Steal this

Three numbers on one page, monthly, per team: active users in the last 30 days (from logs, not surveys), the usage share of the top 5% (your concentration index), and workflows-changed (count only cases where a recurring deliverable is now produced differently — named, verifiable). If the concentration index isn't falling quarter over quarter, you are not running an adoption program. You are running a fan club for your power users — and paying workshop rates for the privilege.

**Seats trained is theatre attendance; telemetry is fitness — instrument the gym before you book more seminars.**

---

## Premium models need failure-cost routing

- URL: https://andymental.com/drops/premium-models-need-failure-cost-routing
- Type: post
- Published: 2026-06-14
- Updated: 2026-07-26

> Fable 5 reopened the price gap between frontier and routine inference — $10/$50 per million tokens. The router that earns that money asks one question the leaderboard never does: what does a wrong answer cost here?

Claude Fable 5 has reopened a gap the market had been quietly closing: the distance between frontier pricing and routine pricing. Anthropic lists it at $10 per million input tokens and $50 per million output — several multiples of the workhorse tier — plus a 1.1x multiplier if you need US-only inference. The newsletters are already producing when-to-use-it guides for GTM teams; one I read this week claims the model is twice as capable as Opus 4.8, a ratio I could not verify and which I suspect means nothing measurable anyway.

Because "how capable" is the wrong axis for the buying decision. The gap that matters is not between the models' benchmark ranks. It is between what their *mistakes* cost you — and that number is a property of your workflow, not of the model.

## The question the leaderboard never asks

Take a GTM stack, since that is where the guides are aimed. Under one roof: lead classification (thousands a day, a mistake costs a mis-tagged row someone fixes next week), draft email copy (a human reads it before send, so failures are caught free), and the recommendation that goes in front of a client with your name on it (a bad one costs the deal, or worse, the relationship). I keep meeting versions of this stack where all three route to the same expensive model — usually because the team upgraded everything the day a launch impressed them, and nothing ever got downgraded.

The router that actually earns Fable-tier money asks three questions per task class, none of which appear on a leaderboard. **What does a wrong outcome cost, at the margin?** A mis-tag is pennies; a wrong client recommendation is the whole engagement. **Is failure detectable before it does damage?** Reviewed-by-a-human tasks can fail cheap, so the premium buys little; fire-and-forget tasks fail invisible, which is where quality is worth real money. **Can a cheaper model refuse?** If the workhorse can flag its own uncertain cases and escalate them upward, the premium model becomes the specialist you consult, not the generalist you salary.

Run those three and the routing table mostly writes itself: high failure cost + low detectability + rare volume routes premium; everything else routes cheap with an escalation path. Model selection stops being a leaderboard exercise and becomes error economics — which is what it always was, we just let the launches distract us.

At work I ran this exact exercise with a delivery team last quarter. Their pipeline sent every task to the priciest model available "to be safe". We tagged each task class with failure cost and detectability; roughly 80% of volume was cheap-to-fail work that a mid-tier model handled indistinguishably, 15% was human-reviewed anyway, and about 5% — the client-facing synthesis — genuinely justified the premium. The bill dropped by more than half, and the interesting part: quality complaints *fell*, because the escalation rule caught uncertain cases that the old everything-premium setup had been waving through unreviewed on the assumption the expensive model is always right.

My rule, stated once: **pay for the model by the price of its mistakes, not the price of its tokens.** Cheap failures deserve cheap models; expensive failures deserve the frontier plus a detection net.

Steal this: three columns on your task inventory — failure cost (money, honestly estimated), detectability (caught before damage: yes/no), monthly volume. One hour of tagging. Route premium only where column one is high AND column two is no. Everything else gets the workhorse and an escalation rule.

**The frontier model is insurance, and nobody insures postcards at diamond rates — price the loss, then pick the courier.**

---

## Agent stack diagrams hide the last mile

- URL: https://andymental.com/drops/agent-stack-diagrams-hide-the-last-mile
- Type: link
- Published: 2026-06-13
- Updated: 2026-07-26

> ByteByteGo's four-layer agent stack is good vocabulary and a fine checklist — but not a reference architecture. Identity, channel failure, and ownership of half-done actions all live outside the neat layers.

ByteByteGo published the kind of diagram that will be pasted into a thousand strategy decks by Friday: "the typical AI agent stack", four clean layers around an agent runtime. The model layer is the brain, tools are the hands, memory is the notebook — working, semantic, transactional — and an observability-and-safety layer wraps the lot. The runtime loops: think, pick a tool, observe, reflect, repeat.

As vocabulary, it is genuinely good. If your organization's first architecture conversation about agents happens over this diagram, everyone will at least mean the same thing by "memory" — no small win. Nothing in the taxonomy is wrong, and to ByteByteGo's credit, nothing in it needed fact-checking. My caveat is about what the word "typical" smuggles in: no production evidence is offered for typicality, and no deployment topology is defined at all. It is a map of concepts, drawn from other maps.

The trouble starts when the slide is treated as a reference architecture — because everything that makes an agent deployable in an enterprise lives in the white space between those four layers.

Run the architect's exercise: take the diagram and try to build the thing your CIO thinks it depicts. Immediately you meet the questions the layers don't answer. **Whose identity** does a tool call carry — the user's session, or a service account that outlives it? Where is the **network boundary** between the runtime and a tool that touches customer data, and who audits crossings? What is the **latency budget** per loop iteration, and what does the user see during a nine-second think? What happens when a **channel is down** — the CRM mid-migration, the API rate-limited — does the agent wait, degrade, or hallucinate around the hole? And the question I have learned to ask first: when the loop half-completes an action — the payment initiated, the confirmation never sent — **who owns the incomplete action?** Support? Engineering? The agent itself, on retry, possibly doubling the payment?

None of that fits in a layer, because layers describe components and these are all *seams*. Identity, boundaries, budgets, fallbacks, and ownership are relationships between components and the organization around them — which is exactly why the generic diagram can travel the internet unchanged while every real deployment is bespoke.

At work my rule for this genre is now: **read every stack diagram as a checklist, never as a blueprint.** As a checklist it earns its keep — "do we have an answer at each layer?" is a fine opening hour. The failure mode is the team that returns from that hour believing the architecture is chosen, when the six seam questions above — the ones that will consume 80% of the engineering — have not yet been asked.

Steal this: next time a stack slide appears in review, add one slide after it titled "the white space" and force written answers to six prompts — user identity, tool identity, network boundary, latency budget, channel fallback, incomplete-action owner. If any box is empty, that is the project plan. The four layers you can buy; the white space is the part you have to build.

**A stack diagram names the parts anyone can assemble — the last mile is the seams, and the seams are never on the slide.**

---

## MCP needs revocation before reach

- URL: https://andymental.com/drops/mcp-needs-revocation-before-reach
- Type: link
- Published: 2026-06-12
- Updated: 2026-07-26

> ReversingLabs maps MCP adoption onto the API era's timeline, where security arrived years after the sprawl. The do-over only counts if identity, least privilege, audit, and a kill switch ship before the connectors do.

ReversingLabs published a piece this week arguing that MCP adoption is replaying the API industry's timeline — and the historical detail it anchors on is worth sitting with. APIs became the backbone of everything years before their security discipline existed; OWASP didn't publish its first API Security Top 10 until 2019, long after API sprawl was a fact of every enterprise. The damage pattern of that gap is well documented.

The hopeful half of the comparison: this time the guidance is early. By now — barely eighteen months after MCP's introduction — OWASP has already shipped both an MCP Security Cheat Sheet and a secure-server development guide. The paper trail that took the API era most of a decade exists for MCP before most enterprises have deployed their tenth connector. That is genuinely a do-over.

The caveat the piece deserves, and the reason I am linking it with an argument attached: **a cheat sheet is not a control.** Guidance documents provide no runtime enforcement, no credential rotation, no proof that any deployed server follows them. And I could not verify any incident-rate comparison between the API era and MCP — the historical analogy is an argument, not a dataset. What the analogy actually earns is a claim about *sequencing*: the API era proves that connectivity adopted before governance becomes ungovernable retroactively, because by the time you inventory the sprawl, the sprawl is load-bearing.

Which is why my test for MCP readiness is deliberately unglamorous. Before a platform team approves connector number ten, four things exist or they don't. An **identity** answer: when a tool call fires, whose permissions is it exercising — the user's, or a server account with god-scope? A **least-privilege** answer: does each server get the narrowest token that does the job, or one broad OAuth grant because scoping was annoying? An **audit** answer: is there one place that lists every server, its scopes, and its last activity? And the one I weight most: a **revocation** answer — when a server misbehaves at 5 p.m. on a Friday, who can kill it, how fast, and has anyone ever actually pulled that lever?

At work I now ask the revocation question first in every MCP review, because it is the one that exposes the real state of the program. Teams can usually produce an identity story and gesture at scopes. The kill switch is where the silence happens: no central inventory, so nothing to revoke against; connectors approved one by one, each reasonable, collectively unmapped — servers that can read customer data and trigger tools, with no list and no lever. That is the API-sprawl movie again, running at agent speed.

Steal this ordering: inventory first, revocation lever second, scope audit third, new connectors last. The reason is arithmetic — inventory and revocation cost days at three connectors and quarters at forty, so sequencing them late is strictly more expensive. The API era got the order backwards and spent a decade paying retroactively.

**MCP really is a do-over — but only for whoever builds the off switch before the on switches multiply.**

---

## Domain agents need disagreement signals

- URL: https://andymental.com/drops/domain-agents-need-disagreement-signals
- Type: link
- Published: 2026-06-11
- Updated: 2026-07-26

> Benchling runs the same scientific task across different model providers — not for redundancy, but because disagreement between model families is a routing signal that sends the risky cases to a domain expert.

LangChain published a conversation with Benchling's AI lead about building agents for life-science work, and one architectural choice in it deserves more attention than it will get: when a scientific answer matters, Benchling runs the task across **different model providers** and cross-checks the answers — deliberately not the same model sampled twice.

The distinction is the whole trick. Re-sampling one model gives you that model's opinion with error bars; its blind spots vote in every sample, so it agrees with itself right through its own systematic mistakes. Different model families were trained differently and fail differently — so when they *agree*, that consensus is worth something, and when they *disagree*, you have found exactly the kind of case where somebody's systematic error is in play. Agreement and disagreement across families carry a signal that repeated samples from one family cannot.

What I like most is what Benchling does with the signal. Disagreement is not resolved by a third model or a tie-break vote — it routes the case to a **domain expert**. The disagreement is a triage bell, not a jury. Confident-and-agreeing flows through; conflicting answers land on a scientist's desk, which is precisely where the risky work should land. Compare that with the single confidence score most agent stacks show: one model grading its own certainty, reassuring right up until the case where it is confidently wrong — and in a domain where errors are material, "reassuring" is the failure mode.

The surrounding operational scaffolding matters too, and matches a pattern I now look for in serious deployments: a weekly rotating "fire chief" who owns production-trace review and brings flagged issues to a standing ops meeting. Notable also that this AI layer launched in October 2025 on top of a data platform Benchling has run since 2012 — the agents sit on thirteen years of structured domain data, which is a large part of why verification is even tractable.

The honest caveat: this is practitioner experience, not a controlled benchmark. Nobody in the piece offers a measured error-rate improvement from multi-provider checking, so what transfers is the architecture, not a number.

At work, the domain agents I review mostly ship the opposite design — one model, one confidence score, and an escalation rule keyed to that score. After reading this, my rule for high-stakes domain agents: **if an error is expensive, buy a second opinion from a different family, and treat disagreement as a routing event, not a bug.** The second provider costs real money; a systematic error reaching production in a scientific or financial workflow costs more.

Steal this in one afternoon: pick your agent's ten highest-stakes task types, run them through a second provider for a week, and log only agree/disagree. The disagreement rate tells you your hidden-risk surface — and every disagreeing case is a free, pre-triaged eval fixture with an expert's answer attached.

**One model's confidence is self-report — two families' disagreement is evidence, and evidence is what should page the expert.**

---

## Launch buzz measures demoability

- URL: https://andymental.com/drops/launch-buzz-measures-demoability
- Type: post
- Published: 2026-06-10
- Updated: 2026-07-26

> 400 builder posts from Claude Fable 5's first 48 hours tell you what the model makes easy to show off. They tell you almost nothing about what survives production review — the sample is selected for shareability.

Claude Fable 5 is days old and the launch-reaction industry is already at work: one newsletter says it classified 400 builder posts from the model's first 48 hours into a map of what people are building. I could not verify the sampling method or reproduce the categories — the piece is paywalled — but the exercise itself is the interesting artifact, because every major model launch now produces one, and every one of them gets read the same wrong way.

A pile of launch posts is not a capability survey. It is a **demoability survey.** The sample is selected — ruthlessly, structurally — for what performs well in public: visual output, one-shot wins, surprising behavior, anything that fits in a clip. What never appears in the sample: the boring extraction pipeline that got 2% better, the confidential legal workload nobody may post about, the task that failed quietly after six attempts, and the job that worked but cost too much to repeat. The absent categories are precisely the ones an enterprise runs.

So launch telemetry answers one question with real authority: *what does this model make easy to demonstrate?* That is genuinely useful — early posts are how interaction patterns and surprising capabilities get discovered, and I read them for exactly that. The failure is using them to answer a different question: *should we build on this?*

## Where the two questions diverge

Fable 5 lists at $10 per million input tokens and $50 per million output. A launch-week demo pays that price once, for a clip. Your workload pays it at volume, every day, against an alternative that may be five times cheaper and good enough. No launch post carries that arithmetic, because the person posting ran the task once and was selecting for wow, not for unit economics. Demoability is measured in screenshots; production fitness is measured in acceptance rate at a price, and the two numbers are not even correlated in the cases that matter — the strongest demo categories are often the ones with the weakest tolerance for a 4% error rate.

At work I watched a product team run this movie earlier this year with a different model: the eval that chose their foundation model was, functionally, a highlight reel — a dozen viral examples reproduced in-house, all of which worked. Impressive demos, signed contract. The first month of production surfaced what the reel never could: their real documents were longer than anything in the demos, their acceptance criteria were stricter than a screenshot's, and recovery from failure — the thing no viral post ever shows — was the majority of the engineering. They re-ran the selection with their own fixtures and a cost curve; a less glamorous model won.

My rule: **launch buzz is a discovery feed, never a decision input.** Read the 400 posts to learn what is newly possible. Then test what *you* need on your own twenty fixtures, at your prompt lengths, with your acceptance bar and your failure-recovery path, and let that table pick the model. The demo economy is optimized to show you the best 48 hours of a model's life. You are buying the other 8,712.

Steal this for the next launch week: keep two lists. List one — "patterns worth stealing" — fill it from the viral posts freely. List two — "reasons to switch models" — may only be fed by results from your own eval fixtures. The discipline is refusing to let anything cross from list one to list two without passing through a test you own.

**A launch tells you what the model shows well; only your fixtures tell you what it does daily — never let the first list make the second list's decision.**

---

## Agent go-live starts the expensive phase

- URL: https://andymental.com/drops/agent-go-live-starts-the-expensive-phase
- Type: blog
- Published: 2026-06-09
- Updated: 2026-07-26

> Salesforce's lessons from 12,000+ Agentforce deployments say the quiet part: test before launch, monitor after. Budget most of an agent program for what comes after go-live — that is when the real debt surfaces.

Salesforce has been publishing what it learned shipping Agentforce at scale — the newsletter version circulating this week says 20,000 enterprise deployments; the accessible Salesforce account says more than 12,000 over a year. I could not reconcile the two numbers, so take the smaller one. Either way, it is the largest public corpus of production agent experience anyone has described, and the shape of the advice is more interesting than the case studies.

Two claims stand out. Salesforce reports a 30% engineering cycle-time improvement from its own internal use, and — the number that should reframe your budget — automatic remediation of 87% of detected incidents within 20 minutes. Both are first-party and unaudited, so hold them loosely. But notice what the second number implies: a mature agent operation is one that expects incidents continuously, detects them fast, and has invested so heavily in remediation machinery that most fixes need no human. That is not a pilot capability. That is a running cost, staffed and tooled, forever.

Which is the quiet message underneath all the operational guidance: **test before deployment, monitor after it.** Translated out of vendor-speak: go-live is not the finish line. It is the start of the expensive phase.

## Why the debt only surfaces after launch

Enterprise agent programs fund themselves like theatre productions — months of rehearsal, a big opening night, and then the assumption that the show runs itself. The reason this fails is structural, not motivational: three kinds of debt are invisible until real users arrive, because only real users push the system off its designed paths.

**Data debt.** The pilot ran on the datasets someone curated for it. Production runs on the knowledge base as it actually is — the stale article, the two systems that disagree about a customer's status, the sync job that quietly broke. An agent surfaces every one of these as a confident wrong answer, at retail volume.

**Policy debt.** The pilot's permissions were whatever made the demo work. In production, an agent discovers every gap between what your policies say and what your systems enforce — the refund limit that lives in a manager's head, the escalation rule nobody wrote down. Each gap becomes either an incident or a new rule you now maintain.

**Exception debt.** Designed paths cover the common cases. Users bring the rest: the order that is half-cancelled, the account with two owners, the request that is reasonable but unanticipated. Every exception is a decision — handle, escalate, or refuse — and the backlog of those decisions is the real work of the first six months.

```mermaid
flowchart LR
    P["Pilot — curated data, demo permissions, designed paths"] --> L["Go-live"]
    L --> D1["Data debt — stale articles, disagreeing systems"]
    L --> D2["Policy debt — rules that lived in heads"]
    L --> D3["Exception debt — the cases nobody designed"]
    D1 --> O["Operations loop: trace review → fix → re-eval"]
    D2 --> O
    D3 --> O
    O --> A["Agent that stays launched"]
```

None of this debt is visible in the pilot, because the pilot was designed not to hit it. The launch does not create the debt. It reveals it — and it reveals it on a schedule you don't control, at whatever volume your users happen to bring on the day the sync job breaks.

There is also a fourth cost that isn't debt at all: drift. The business changes under a healthy agent — a new product line, a renamed policy, a reorganized team — and answers that were right in March are wrong in August with no incident, no error, and no alert. Only scheduled review catches it.

## The plan I keep rejecting

At work, the client plan I see most often funds a polished pilot, a launch date, a communications push — and then assigns the agent to "the platform team" as one more thing they own. No rotation for reading traces. No owner for policy fixes. No budget line for keeping evals current as the business changes. No rehearsed rollback. The program's org chart simply stops at go-live.

When I ask who reads the traces in week three, the answer is usually "the dashboard will alert us". But dashboards alert on failures the builders anticipated; the debt above is by definition what they didn't. The only instrument that finds it is a human reading real transcripts on a schedule and filing what they find — wrong answers to the data team, permission surprises to the policy owner, new exception classes to the workflow backlog.

My rule for sizing this now, after watching a few programs through their first year: **plan the pilot-to-launch effort as the smaller half.** Whatever you spent getting to go-live, reserve at least as much again for the twelve months after — a trace-review rotation with real time carved out, an owner for policy repair with authority to change rules, eval maintenance wired to every model and prompt change, and a rollback path you have actually exercised. If the budget cannot fund the second half, ship a narrower agent. A small agent with a funded operations phase beats a broad one abandoned at launch — the broad one does not stay launched.

Salesforce's own 87%-in-20-minutes figure, whatever its audit status, is the strongest version of this argument: the most experienced agent operator on record responded to production reality by building an incident-remediation machine. They did not get to skip the expensive phase. They industrialized it.

## Steal this

Before your next agent go-live, write the week-three rota on one page: who reads twenty traces a day, who owns policy fixes with what authority, who re-runs evals when anything upstream changes, who can roll back and how fast — with names, not team labels. If any line says "TBD", the launch date is fiction; you are scheduling an incident, not a release. And when the go-live retro happens, hold the applause for month six — that is when you will know whether you shipped an agent or an announcement.

**The pilot proves the agent can work; the year after go-live decides whether it does — fund the year, not the launch party.**

---

## Agent loops are release artifacts

- URL: https://andymental.com/drops/agent-loops-are-release-artifacts
- Type: post
- Published: 2026-06-08
- Updated: 2026-07-26

> The head of Claude Code says he doesn't prompt anymore — he writes loops. The quiet part: a loop that runs unattended is software, and it belongs in version control with tests, budgets, and stop conditions.

A guide making the rounds today is built on a striking admission from Boris Cherny, the creator of Claude Code: he doesn't really prompt anymore. He writes loops — recurring, persistent jobs that prompt the model, check the result, and decide what to do next, while he does something else. Review loops, maintenance loops, nightly cleanup loops.

The guide sells this as the next productivity unlock, and it is. But the framing everyone will take away — "stop prompting, start looping" — hides the part that will actually determine who wins with it. The moment your instruction runs without you watching, it stops being a prompt and becomes software. And almost nobody is treating it like software.

## Disposable versus maintained

The sharp line is not prompt versus agent. It is **disposable instruction versus maintained operational artifact.** A prompt you type, read the answer to, and discard is disposable — quality control is you, in the moment. A loop that wakes up at 2 a.m., reads your codebase, and files changes has no you in it. Everything you were silently providing — judgment, context, the decision to stop — has to be made explicit, or it is simply absent.

Count what a real loop contains: a schedule, state that survives between runs, retry behavior, a termination condition, a review step, and some evidence trail of what it did. That is a release checklist wearing a trench coat. We have known for decades where things with those properties live: in version control, with tests, behind code review.

The prompt cookbook is where this goes wrong quietly. I have watched a team demo a beautiful library of prompts — every one of them worked, in the demo, with a human at the wheel. Promoted to scheduled loops, the same prompts had no fixture inputs to regression-test against, no failure policy beyond "retry", no budget ceiling, and no history of which revision worked last month. When a model update shifted behavior — and the guide itself warns a workflow edge can expire within three to six months as models change, a claim I can't independently verify but my own experience roughly matches — nobody could say which loops were degraded, because nothing was pinned, versioned, or measured. A cookbook cannot carry operational reliability. It was never designed to.

## What a released loop looks like

My rule: **a loop earns its schedule the way code earns a deploy.** Concretely, the loop file lives in the repo next to the code it touches. It declares its budget — tokens, dollars, wall-clock — and its stop conditions: what "done" is, what "give up" is, and what triggers a human. It has at least three fixture inputs with expected-shape outputs, run on every model or prompt change, so drift shows up in CI instead of in production. And every run writes a one-line ledger entry — started, cost, outcome — because a loop with no evidence trail is unauditable by construction.

None of that is heavyweight. It is perhaps an hour per loop, once. Which is the point: the discipline is cheap, and the absence of it is only expensive *later*, which is exactly the trade software engineering exists to manage.

Steal this: take your three most-used prompts — the ones already informally load-bearing — and promote them properly. Commit each one, add fixtures, a budget line, a stop condition, and a ledger. That's your first release. Cherny's job title shift is real, and it is coming for the rest of us: less prompting, more loop-writing. The teams that treat those loops as release artifacts will compound. The teams running a folder of hot prompts on a scheduler will spend next year debugging ghosts.

**A prompt is a conversation; a loop is a deployment — version the things that run without you.**

---

## AI budgets should follow workloads, not employees

- URL: https://andymental.com/drops/ai-budgets-should-follow-workloads
- Type: post
- Published: 2026-06-07
- Updated: 2026-07-26

> Uber burned a 12-month AI budget in 4 months and answered with a $1,500/month cap per employee per coding tool. The cap treats spend as a person problem — but spend belongs to workloads, and workloads are what to budget.

Uber's CTO disclosed this week that the company burned through its entire annual AI budget in four months. The response, per Bloomberg and TechCrunch: a $1,500 monthly cap per employee, per agentic coding tool — Claude Code, Cursor and the like — trackable on an internal dashboard, exceedable with permission.

The backstory makes it better. Uber had been encouraging staff to use AI "as much as possible", complete with internal leaderboards ranking usage. Roughly 95% of engineers now use the tools, and about 10% of the company's code is agent-generated. They built the incentive, the incentive worked, and the bill arrived. Fair enough — every platform team is living some version of this year.

It is the fix I want to argue with. A per-user cap is the payroll department's mental model applied to inference: spend attaches to a person, so ration the person. But almost nothing interesting about AI spend is true at the level of a person.

## The unit of spend is the workload

Under one $1,500 cap sit three completely different things. An engineer autocompleting through the day — cheap, latency-sensitive, low stakes. An overnight agentic loop refactoring a service — expensive, async, and the whole reason you bought the tools. And somewhere, a production-critical pipeline that happens to run under a human's account because that was easiest at setup time. The cap prices all three identically, which means it rations the second and third — the highest-value work — to protect against overuse of the first.

And people respond to caps the way people always respond to caps. The practitioner version of this story, which I have now watched at more than one client: the moment individual limits land, teams start sharing accounts, routing expensive jobs through whoever has headroom this month, or burying inference inside an unrelated project's cloud budget where nobody itemizes it. The spend does not shrink — it goes dark. A platform owner who sees sudden per-user uniformity right under the cap is not looking at compliance. They are looking at shadow routing, and they have lost the telemetry that would have told them which work was worth the money.

My rule: **budget the workload, not the identity.** A workload has an owner, an outcome, a quality threshold, a monthly budget, and an escalation rule for the month it wants more. "Nightly refactor loop on the payments service, owned by K., $4,000/month, escalate past that with a one-line justification" is governable. "Every human gets $1,500" is not governance — it is a speed bump with reporting.

To be fair to Uber: the exceed-with-permission valve and the visible dashboard are the right instincts, and a blunt cap is a defensible tourniquet while you build the real thing. The mistake would be mistaking the tourniquet for the circulatory system.

Steal this if the cap memo is heading for your desk: before capping anyone, tag every AI-consuming job with a workload label — even a crude one. Run thirty days of spend-by-workload instead of spend-by-employee. My bet, having seen the exercise run: the top ten workloads explain most of the bill, at most a couple of them are questionable, and the per-user distribution underneath turns out to be noise. Then fund the workloads that earn it, kill the ones that don't, and let humans autocomplete in peace.

**Uber capped the people because people are what the dashboard could see — build the dashboard that sees workloads, and the budget conversation becomes an investment conversation.**

---

## Local model benchmarks are not capacity plans

- URL: https://andymental.com/drops/local-model-benchmarks-are-not-capacity-plans
- Type: post
- Published: 2026-06-06
- Updated: 2026-07-26

> Ollama 0.30 is up to 20% faster on NVIDIA — measured on one model, one GPU, one quantization. A single-GPU speedup proves an optimization exists. It does not tell you whether local inference can carry your workload.

Ollama shipped 0.30 this week: broader GGUF compatibility through llama.cpp, Vulkan on by default so AMD and Intel GPUs work out of the box, tool-calling preserved where the model exposes it, and a headline number — up to 20% faster on NVIDIA hardware.

Read the fine print on that number. The disclosed test is Gemma 4 26B, Q4_K_M quantization, on a single RTX 5090. One model, one quant, one GPU. Ollama published no broader benchmark matrix, so performance anywhere outside that exact configuration is extrapolation — mine and yours.

None of this is a criticism of the release. It is a good release. The criticism is reserved for what happens next in a dozen planning meetings: someone pastes the 20% into a slide and a local-first deployment gets approved on the strength of a number that answers a different question.

## A speedup is not a capacity plan

A single-GPU benchmark tells you an optimization exists. A capacity plan answers questions the benchmark never asked:

How many **concurrent sessions** before tokens-per-second per user collapses? Vendor numbers are almost always single-stream; production is never single-stream. What happens at your real **prompt lengths** — the 40-page policy document, not the benchmark's tidy prompt? Where does **tail latency** land when the KV cache fills and requests start queuing? What does **memory pressure** do when two long contexts collide on a 32GB card? How much **quality** did Q4_K_M cost on your tasks — not on MMLU, on your extraction schema? And when the process falls over at 2 a.m., what is the **recovery behavior** — who reloads the model, how long does the warm-up take, and where do in-flight requests go?

Six budgets: concurrency, prompt length, tail latency, memory, quality, recovery. A vendor benchmark funds none of them.

## The meeting where this bites

At work this spring, an engineering lead showed me a local-inference proposal justified by exactly this genre of number — a vendor's single-GPU tokens-per-second, multiplied by GPU count, divided by expected users. On paper it cleared the workload with headroom. We ran a load test with the client's actual prompt distribution instead: p95 latency crossed their SLA at roughly a third of the concurrency the slide promised, because their real prompts were four times longer than anything the vendor measured and the cache thrashed. The fix wasn't a faster runtime. It was buying to the measured number and keeping a cloud overflow lane for the spikes.

My rule since: a vendor benchmark is admissible as evidence that the software improved, never as evidence that your deployment will hold. The only benchmark that counts as a capacity plan is the one that ran your prompts, your quant, your concurrency, on the hardware you will actually buy.

Steal this for the next local-model proposal that crosses your desk: require one table before approval — target concurrent sessions, real p50/p95 prompt lengths, measured p95 latency at that concurrency, memory headroom at peak, task-level quality delta versus the cloud baseline, and time-to-recover from a process kill. If any cell says "from vendor blog", the table is not done. The test costs a day on rented hardware. The slide-number deployment costs you the quarter in which you discover the difference.

**A benchmark proves the engine got faster — it says nothing about the traffic you are about to drive into it.**

---

## Route agents by queue age, not token price

- URL: https://andymental.com/drops/route-agents-by-queue-age-not-token-price
- Type: blog
- Published: 2026-06-05
- Updated: 2026-07-26

> A laptop handled 78% of one investor's AI work last week — but the number that matters is queue age falling from 73 seconds to 4. Hybrid local-cloud routing is a scheduling problem first and a pricing problem second.

Tomasz Tunguz published a week of measurements from his own desk this morning: a local classifier on his Mac now routes his AI tasks, keeps the straightforward ones on-device, and sends only the hard ones to cloud models. Over seven days, the laptop handled 78% of the work. Throughput rose about 25%. Average task duration fell from 47 seconds to 19.

Everyone will quote the 78%. It is a great number for the local-models-are-winning narrative, and that narrative will run all year.

The number I would frame and hang on the wall is a different one: **queue age fell from 73 seconds to 4.**

## The metric hiding in the fourth sentence

Queue age is how long a task sits waiting before anything starts working on it. Before the router, Tunguz's average task waited 73 seconds — longer than most of the tasks themselves took to run. After, it waited 4. That is not a cost improvement. That is a scheduling improvement, and it is where nearly all of the felt speed-up lives.

This is queueing theory doing what it always does. When every task — the two-line draft reply and the forty-minute research run — goes into the same line for the same expensive resource, small jobs get stuck behind big ones. Head-of-line blocking. Your work-in-progress ages on the shelf, and the system feels slow even when the workers are fast. Give the small jobs their own fast lane and the whole system's responsiveness transforms, even if the big jobs finish no faster than before.

Read the architecture again with that lens. The local model isn't valuable because it is free. It is valuable because it is *idle and nearby* — an always-available worker for the small stuff, which means the queue for the serious cloud work stops being contaminated by trivia. Cloud avoidance is the mechanism. Waiting time is the outcome. Tunguz's own framing gets this right: most of his AI work, it turns out, can wait a bit or run somewhere cheaper — what it must not do is stand in line.

## What the router market wants you to optimize instead

There is a growing shelf of model-routing products, and almost all of them pitch the same metric: price per token. Route the easy prompts to the cheap model, save 60% on inference, here is the dashboard. The pitch works because token spend is legible — finance can see it, and nobody's finance team has ever seen a queue-age chart.

At work I sat in exactly this evaluation this quarter: a platform lead choosing between a cheaper model router and a workflow scheduler. The router demo showed cost falling. But when we pulled the traces from their existing agent fleet, the picture was Tunguz's picture: small classification and drafting tasks were queuing behind long document-analysis runs on the same worker pool, and the median task spent more time waiting than executing. A cheaper model would have made the *same wait* cost less. Only the scheduler made the wait go away. We chose the scheduler, and the token bill barely moved — but the backlog cleared and the team stopped babysitting the queue, which was the actual complaint.

My rule from that engagement: **instrument queue age before you buy a router.** If tasks wait longer than they run, you have a scheduling problem, and no per-token discount fixes a scheduling problem.

## The shape of the fix

```mermaid
flowchart LR
    T["Incoming tasks"] --> C["Local classifier"]
    C -->|"small, latency-tolerant"| L["Local model — always idle, always nearby"]
    C -->|"hard, quality-critical"| Q["Cloud queue"]
    Q --> F["Frontier model"]
    L --> D["Done — seconds"]
    F --> D2["Done — uncontaminated by trivia"]
```

Two lanes, one classifier, and the discipline to route by *latency class* — what the task needs and when it is due — rather than by which model is cheapest this week. The classifier does not need to be clever. It needs to be fast and local, so classification itself never joins the queue.

## The honest caveats

This is one person's desk, one week, self-reported. I could not verify the workload mix, whether quality was measured at parity between local and cloud outputs, or whether tasks that failed locally and were re-sent to the cloud got counted once or twice. A 78% local share for an investor's reading-and-writing workload will not transfer to a coding-heavy or compliance-heavy one. Treat the numbers as an existence proof, not a benchmark.

But the existence proof is enough, because the queueing argument does not depend on his exact numbers. It depends on arithmetic: any fleet where task sizes vary by two orders of magnitude and everything shares one line will spend most of its life waiting.

## Steal this

Add three timestamps to every agent task in your system this week: `enqueued_at`, `started_at`, `finished_at`. Report two numbers per task class: p50 and p95 queue age. That is the whole instrument. If p95 queue age exceeds p50 execution time, your next purchase is a scheduler or a second lane — local model, dedicated worker pool, either works — and not a cheaper model. Re-measure after. The day queue age drops is the day the system starts feeling fast, whatever happens to the invoice.

**The laptop didn't win because it was cheap — it won because the small work stopped standing in line behind the big work.**

---

## Pre-release model review is procurement leverage, not a safety standard

- URL: https://andymental.com/drops/pre-release-model-review-is-procurement-leverage
- Type: post
- Published: 2026-06-04
- Updated: 2026-07-26

> The new executive order buys the government up to 30 days of confidential pre-release access to frontier models. Without published pass criteria, that is leverage dressed as assurance — buyers should read it that way.

This week's executive order on advanced AI directs agencies to stand up a voluntary framework under which frontier-model developers can grant the federal government up to 30 days of confidential pre-release access, with the framework itself due by August 1. The order is explicit about what it is not: no mandatory licensing, no preclearance, no permitting for new models.

The newsletter framing doing the rounds — "the NSA now grades your AI model" — is ahead of the text. I could not verify any provision for an actual grade, and the order publishes no pass criteria, no evaluation suite, and no remediation triggers. What exists is an access mechanism and a promise of collaboration. Everything else is inference.

So what is 30 days of quiet access actually worth? Read it as a procurement instrument and it makes complete sense.

## Access without criteria is leverage

A review with published pass criteria and consequences is a standard. A review with neither is a relationship — and in government, structured relationships are how leverage works. The state gets threat intelligence on frontier capabilities before the public does, which is genuinely valuable for defence planning. The lab gets something too: the ability to say, truthfully, that the government saw the model before release and didn't object.

That sentence is where the danger lives for the rest of us. "Reviewed under the federal framework" will start appearing in sales decks within a quarter of the framework going live — my bet, and I will happily be wrong. It will sound like certification. It will be an attendance record. Voluntary access with confidential findings and no published bar is an informal market gate that looks much stronger than it is, and the vendors who lean on it hardest will be the ones who benefit most from the ambiguity.

There is a real tension here, and it is worth being fair about it: early access probably does improve national threat intelligence, and the no-licensing language keeps the door open for small labs. The order is not a bad instrument. It is just not the instrument your risk register thinks it is.

## What this changes for a regulated buyer: nothing

At work, the question landed within days of the order: a client in a regulated industry asked whether the federal review would "count" toward their own model-risk obligations. The answer I gave them is the whole post: a review whose methods, findings, and thresholds you cannot see transfers zero obligation away from you. Your regulator will not accept "the government looked at it" any more than it accepts "the vendor tested it". Context is the whole game — the government did not test the model on your data, your workflows, or your failure costs.

My rule for any external review badge, government or otherwise: it reduces your testing burden only to the extent that its methods and pass criteria are public and its scope covers your use. Score it zero on both today. That may change — if the August framework ships with published evaluation suites and consequence triggers, I will upgrade it happily. The order as signed commits to neither.

Steal this for your vendor file: add a line item called "external reviews claimed", and next to each one record two fields — methods public? consequences defined? Anything with two nos is marketing, and gets weighted accordingly in the risk assessment.

**Thirty days of confidential access buys the government intelligence and the lab a talking point — your own eval on your own risks is still the only review that transfers.**

---

## Data agents need semantic infrastructure, not smarter models

- URL: https://andymental.com/drops/data-agents-need-semantic-infrastructure
- Type: blog
- Published: 2026-06-03
- Updated: 2026-07-26

> OpenAI's data agent serves 3,500+ users over 600 petabytes — and the architecture is mostly lineage, definitions, and permissions. Natural-language analytics is a semantic-layer problem wearing an agent costume.

OpenAI published the architecture behind its in-house data agent — the internal tool that answers analytics questions for more than 3,500 employees over roughly 600 petabytes and 70,000 datasets. If anyone on earth could brute-force "text to SQL" with raw model intelligence, it is the company that makes the models.

They didn't. Read the writeup and count what the architecture is actually made of: schema metadata, table lineage, historical queries, curated human annotations about what tables mean, and access control passed through from the underlying platform. The agent grounds itself in explicit context layers — starting with table usage and human-written domain notes — before it ever touches broader organizational context.

The model is in there somewhere. It is the least interesting part of the diagram.

## The failure mode nobody benchmarks

Every data team that has piloted natural-language analytics knows the specific way it fails, and it is not syntax errors. The SQL comes back clean, runnable, and plausible. It just queried `orders_v2_legacy` instead of `orders_current`, because both exist and nothing told the model which one the finance team actually trusts. Or it computed "revenue" — a word your company defines one way in the billing warehouse and a subtly different way in the board deck — and produced a number that is correct by one definition and wrong in the meeting.

I have debugged exactly this at work: a client's pilot agent writing beautiful SQL against the wrong grain of a table, monthly figures pulled from a daily snapshot, everything plausible and everything off by design. No amount of model upgrade fixes that, because the failure isn't reasoning. The failure is that the knowledge of which table is real and what the metric means lived in two senior analysts' heads, and the agent was never given a way to read it.

That is the quiet message of OpenAI's disclosure. Their answer to "which table is real" is not a bigger model — it is human annotations, usage statistics, and lineage wired into context. The organisation wrote its tribal knowledge down and made it machine-readable. That is the product.

## What the stack actually is

Strip the branding and OpenAI's data agent is a layered grounding system:

```mermaid
flowchart TD
    Q["Employee question"] --> A["Agent"]
    A --> L1["Schema metadata — columns, types"]
    A --> L2["Table lineage — upstream, downstream"]
    A --> L3["Historical queries — what people actually run"]
    A --> L4["Human annotations — what tables mean, caveats"]
    A --> L5["Pass-through permissions — what THIS user may see"]
    L1 --> S["Grounded SQL + answer"]
    L2 --> S
    L3 --> S
    L4 --> S
    L5 --> S
```

Four of those five layers existed as a category before agents. We used to call them a data catalog, a lineage graph, a query log, and row-level security. The agent interface is new; the infrastructure it depends on is the semantic layer your data team has been asking to fund since 2019.

My claim, stated plainly: natural-language analytics is mostly a semantic-layer and permissions problem presented through an agent interface. The companies that get magical-seeming data agents in the next two years will be the ones that did unglamorous metadata work in the last two. The model is becoming a commodity component inside somebody else's information architecture — which also means a "data agent" bought off the shelf inherits none of this, because the semantic layer is, by definition, yours.

Note the honest gap too: OpenAI published the architecture but no independent accuracy benchmark for the agent. Even the reference implementation is asking to be trusted on outcomes. If they had a number they loved, I suspect we would have seen it.

## The permissions layer is not optional garnish

The detail I would underline twice: access control passes through to the agent. The agent can only read what the asking user could read. That single decision kills the most common enterprise data-agent disaster — the friendly chatbot that happily aggregates tables its user was never allowed to see, because the service account it runs on is God. If your data agent pilot runs on a privileged service account "just for the demo", you do not have a pilot. You have an incident with a delay timer.

## Steal this sequencing

If a data-agent initiative is on your roadmap, reorder it. First: pick the fifty tables that answer 80% of real questions, and write one paragraph per table — what it is, what grain, what to never use it for. Second: wire lineage and the query log into retrieval. Third: pass through user permissions, not service-account permissions. Fourth — only fourth — pick the model, and pick it by eval on your own questions, not by leaderboard.

Run the pilot on the annotated fifty tables only, and publish the coverage number to the business — "the agent currently speaks for 50 tables, verified" sets expectations the way a chatbot demo never will. An agent that says "I don't have context for that table yet" is trustworthy infrastructure with a roadmap. An agent that answers everything is a liability with good manners.

The budget conversation gets easier too. Semantic-layer work used to be a hard internal sell because its beneficiaries were dashboards nobody loved. Now the same investment is the difference between a data agent that works and one that embarrasses you in a QBR — same metadata, suddenly fundable.

**A data agent is your semantic layer wearing a conversational interface — fund the layer, and the agent gets smart for free.**

---

## Agent count is not a production metric

- URL: https://andymental.com/drops/agent-count-is-not-a-production-metric
- Type: post
- Published: 2026-06-02
- Updated: 2026-07-26

> SaaStr says it runs on 3 humans and 21+ AI agents. The useful part of the disclosure is the job map, not the ratio — and the scorecard your agent program needs has five columns, none of which is headcount.

On June 2, SaaStr published the operating setup behind the headline it has been running for months: 3 humans, 21+ AI agents. Not a thought experiment — a named list, agent by agent, with marketing agents, customer-success agents, each assigned an actual job. Caveat up front: the operating and revenue figures are first-party claims with no audit trail, so treat the numbers as directional.

The headline ratio is what everyone will repost. Three humans! Twenty-one agents! It reads like a productivity miracle, and every automation programme in every enterprise will now get asked some version of "how many agents do we have?"

That question is a trap, and I want to name it before it lands in your quarterly review.

## The job map is the disclosure; the ratio is the marketing

The genuinely useful part of SaaStr's post is not the count. It is that every agent has a name, an owned workflow, and a specific job. An agent that owns "reply to sponsor enquiries end to end" is an operational fact you can inspect. "We have 21 agents" is an inventory line — and inventory is what you count when you don't yet know what the assets produce.

We have been here before. Nobody serious reports "number of microservices" as an engineering KPI, because a service count tells you nothing about uptime, latency, or cost. It took the industry years to stop bragging about cluster sizes and start reporting SLOs. Agent programmes are speed-running the same mistake with a shinier noun.

My rule: an agent only counts when you can answer five questions about it. What workflow does it own? How many completions did it deliver this month? How often did a human have to step in? What did its worst error cost? And what would make you retire it? That last one is the tell — a programme that cannot name retirement criteria is collecting agents, not running them.

## The dashboard I keep seeing

At work I reviewed a client dashboard this quarter that celebrated "agents deployed" as the headline metric — a big, confident number, trending up and to the right. Nowhere on the page: accepted outcomes, escalation load, or rework. When we pulled the escalation queue, one "deployed" agent had been silently routing 40% of its tasks to a human for weeks. It was being counted as automation while functioning as a form with extra steps.

We rebuilt the dashboard around the five columns above. Two agents got retired the same week — not because they were broken, but because nobody could say what they owned. The deployed count went down. The programme got healthier. Those two sentences should be allowed to coexist in more board decks than they currently are.

That is also the honest way to read SaaStr's own disclosure. The impressive part is not that they have 21 agents; it is that they can apparently name what each one does. Most enterprises publishing agent counts cannot — and the count is doing the work the job map should be doing.

## Steal this before your next review

Take your agent inventory and add five columns: owned workflow, completions, interventions, worst-error cost, retirement criteria. Fill it in before Friday. Any row you cannot complete is not an agent in production — it is a demo with a budget line. Report the rows you completed, and only those, as your programme.

The count will be smaller. It will also, for the first time, be a production metric.

**Agents are headcount only in the sense that headcount was never the point — measure owned work, not owned bots.**

---

## Voluntary AI frameworks are regulatory signals

- URL: https://andymental.com/drops/voluntary-ai-frameworks-are-regulatory-signals
- Type: blog
- Published: 2026-06-01
- Updated: 2026-07-20

> OpenAI's Frontier Governance Framework is a well-built compliance document nine weeks before EU enforcement bites. Read it for what it concedes, not what it promises — and ask your vendor for the judgment record instead.

The most useful sentence in OpenAI's new Frontier Governance Framework is not a commitment. It's an admission: on harmful manipulation — one of the four risk categories the document itself names — they are "still in the early stages of developing an approach for assessing" it, and handle it with post-deployment monitoring rather than pre-deployment evaluation.

That sentence is worth more to a buyer than the other nineteen pages, and almost nobody will quote it.

## What was published, and what it is for

On May 28, 2026, OpenAI published the Frontier Governance Framework — a document that does double duty by design. Under California's Transparency in Frontier AI Act it is their Frontier AI Framework; under the EU's General-Purpose AI Code of Practice it is the public summary of their Safety & Security Framework for models covered by Regulation (EU) 2024/1689. One artifact, two regimes.

The content is more specific than most governance prose. Four systemic risk categories: cyber offense, CBRN, harmful manipulation, loss of control. A definition of systemic risk with actual numbers in it — foreseeable and material risks of severe harm, including a model materially contributing to more than 50 fatalities or a billion dollars of damage from a single incident. Alignment claimed to ISO 42001, the NIST AI Risk Management Framework, and METR's Responsible Scaling Policy proposal.

The timing is not subtle, and doesn't need to be. GPAI obligations under the EU AI Act have applied since August 2, 2025, but the first year was a good-faith period — the AI Office working alongside signatories to the Code of Practice rather than penalising them. From August 2, 2026, the Commission enforces full compliance, fines included. That is nine weeks from today. Models already on the market before August 2025 get until August 2027.

The obvious read is "OpenAI complied because the deadline is coming". Fine, and true. It also doesn't tell a working engineer anything actionable.

## What a framework can and cannot carry

Here's the distinction that matters if you have to sign off on a vendor. A framework describes gates. Evidence proves gates bind. Publishing the first has now become a legal requirement in two jurisdictions; publishing the second has not.

Read the document closely and it says so itself, honestly. Threshold determinations are "informed by" evaluation results and also "reflect a holistic judgment based on the totality of available evidence". One-time capability elicitations are treated as a lower bound, not a ceiling. Out of caution they have counted a threshold as crossed even without direct evidence that it was. Every one of those sentences describes a **judgment call by a named group of people**, made repeatedly, under commercial pressure, and never published.

That is not a criticism of the document. It's the nature of the artifact. No public framework can carry the thing a risk review actually needs, which is the record of the specific calls: who decided, on what evidence, and which releases went ahead anyway.

## The bet

So here's my claim, and I'll take the other side of the consensus on it. Frameworks are about to become worthless as a differentiator, precisely because they're becoming mandatory.

By the time enforcement starts in August, every large frontier developer will have published one of these. They are all mapping to the same two regimes and citing the same three standards — ISO 42001, NIST AI RMF, RSP-style scaling commitments. Converged inputs produce converged outputs. Within a year these documents will be structurally interchangeable, and a procurement team scoring vendors on framework quality will be scoring on prose style.

My bet: the vendors who actually differentiate themselves will be the ones willing to show a customer the judgment record — the threshold calls and the exceptions — under NDA, in a room, with a named owner present. Not published. Shown. And the first serious enterprise deal won or lost on that basis happens well before the August 2027 deadline for legacy models.

![Diagram contrasting the published framework, which describes risk categories, thresholds and gates, with the unpublished judgment record — who decided, on what evidence, and which releases shipped anyway — which is what a model risk review actually needs.](/api/media/file/voluntary-ai-frameworks-are-regulatory-signals-gap.png)

## What I do in a model risk review now

At Trigent I sit in vendor reviews where a client needs a model decision defended to their own risk committee. The pattern is consistent: the vendor sends the framework, the certifications, and a security questionnaire, and everyone treats the package as the answer. It isn't the answer. It's the cover sheet.

The three questions I ask instead, in this order:

- **Who owns the release gate by name, and what happens to them if they're wrong?** A framework with no named owner is a description of a process, not a control.
- **Show me one evidence artifact from the most recent release** — an eval report, a red-team summary, a sign-off record. Not the policy that says such artifacts exist.
- **Has a release ever proceeded over an unresolved objection, and what was the disposition?** The answer "never" is not reassuring; it usually means nobody is tracking it.

I've had vendors answer all three well and I've had that conversation end the evaluation. Both outcomes were worth far more than the framework PDF, and neither could have been reached by reading it.

There's a corollary I've had to accept about my own work: if I'm asking vendors for evidence artifacts and named owners, my team has to be able to produce ours on the same day's notice. We couldn't, the first time someone asked. That was a fair hit, and fixing it took longer than writing any policy did.

## Read it for the concessions

So read these documents — genuinely, read them — but invert how you read them. Skip the commitments; they are written to be unfalsifiable. Go hunting for the sentences where the vendor admits a gap, because those are the only lines that carry information their competitors' documents won't also contain by year end. The harmful-manipulation admission is the tell in this one. It's specific, it's unflattering, and it tells you exactly which risk category to interrogate in the room.

**A published framework tells you what a vendor intends; only the exception record tells you what they enforce — ask for the second one before August.**