Agent stack diagrams hide the last mile

ByteByteGo published the kind of diagram that will be pasted into a thousand strategy decks by Friday: "the typical AI agent stack", four clean layers around an agent runtime. The model layer is the brain, tools are the hands, memory is the notebook — working, semantic, transactional — and an observability-and-safety layer wraps the lot. The runtime loops: think, pick a tool, observe, reflect, repeat.

As vocabulary, it is genuinely good. If your organization's first architecture conversation about agents happens over this diagram, everyone will at least mean the same thing by "memory" — no small win. Nothing in the taxonomy is wrong, and to ByteByteGo's credit, nothing in it needed fact-checking. My caveat is about what the word "typical" smuggles in: no production evidence is offered for typicality, and no deployment topology is defined at all. It is a map of concepts, drawn from other maps.

The trouble starts when the slide is treated as a reference architecture — because everything that makes an agent deployable in an enterprise lives in the white space between those four layers.

Run the architect's exercise: take the diagram and try to build the thing your CIO thinks it depicts. Immediately you meet the questions the layers don't answer. Whose identity does a tool call carry — the user's session, or a service account that outlives it? Where is the network boundary between the runtime and a tool that touches customer data, and who audits crossings? What is the latency budget per loop iteration, and what does the user see during a nine-second think? What happens when a channel is down — the CRM mid-migration, the API rate-limited — does the agent wait, degrade, or hallucinate around the hole? And the question I have learned to ask first: when the loop half-completes an action — the payment initiated, the confirmation never sent — who owns the incomplete action? Support? Engineering? The agent itself, on retry, possibly doubling the payment?

None of that fits in a layer, because layers describe components and these are all seams. Identity, boundaries, budgets, fallbacks, and ownership are relationships between components and the organization around them — which is exactly why the generic diagram can travel the internet unchanged while every real deployment is bespoke.

At work my rule for this genre is now: read every stack diagram as a checklist, never as a blueprint. As a checklist it earns its keep — "do we have an answer at each layer?" is a fine opening hour. The failure mode is the team that returns from that hour believing the architecture is chosen, when the six seam questions above — the ones that will consume 80% of the engineering — have not yet been asked.

Steal this: next time a stack slide appears in review, add one slide after it titled "the white space" and force written answers to six prompts — user identity, tool identity, network boundary, latency budget, channel fallback, incomplete-action owner. If any box is empty, that is the project plan. The four layers you can buy; the white space is the part you have to build.

A stack diagram names the parts anyone can assemble — the last mile is the seams, and the seams are never on the slide.