Google’s TPU split makes the workload question harder to skip

A hardware comparison that begins and ends with peak compute can miss the reason the hardware was specialised. Google's eighth-generation TPU announcement gives me a concrete reason to ask how the intended workload spends its time.

Google describes TPU 8t and TPU 8i as distinct architectures for training and inference. It emphasises compute throughput and scale-up bandwidth for 8t, and memory bandwidth for latency-sensitive serving on 8i. The 8i specification includes 288 GB of high-bandwidth memory and 384 MB of on-chip SRAM, with the latter sized around reasoning-model KV-cache demands.

These are vendor-described design choices. They do not establish that every agent workload is memory-bound, that compute no longer matters or that one chip will be cheapest for a particular application.

I would take the specialisation as a prompt to characterise the work before choosing the machine. The label “agentic” is too broad to do that job.

The loop can wait in several places

Consider a hypothetical assistant that reads a large context, generates a short tool call, waits for an external service and then reasons over the result. Its elapsed time includes several different activities, only some of which run on the accelerator.

A second application might process a large batch of independent inputs with little external waiting. Even if both use the same model, their useful throughput and latency constraints can differ substantially.

The first system may benefit from a faster serving step without seeing the same proportional improvement in the complete task. The second may be able to trade individual response time for higher aggregate throughput. Neither outcome can be inferred from a chip's peak arithmetic alone.

I would measure where time and resources go before projecting a hardware improvement onto the user experience. That includes context processing, generation, communication, queueing and external tool waits. The purpose is to identify the part of the system a proposed purchase can actually improve.

Specialisation has a fleet cost too

A chip optimised for one workload can create a useful efficiency gain. A mixed fleet also has to place work, manage capacity and handle changes in demand. The best local result is not automatically the best operating arrangement.

For a proposed deployment, I would test a representative mix at the quality and latency thresholds the service needs. Include the slow cases and the periods when demand is uneven. Ask whether spare capacity can usefully serve other work rather than assume specialisation is free to operate.

Google says both platforms support familiar frameworks, including JAX and PyTorch, alongside serving tools. Shared framework support is helpful, but it is not proof that a particular application runs unchanged or performs identically on both. That needs a compatibility and performance check.

I have not run those checks on these chips and would not claim Google's published gains as savings for my own workload. The announcement is a specification and design rationale to evaluate, not a completed procurement comparison.

What I would change immediately is the review question. Instead of asking which accelerator has the most impressive headline, ask which measured waiting point it removes and what operating constraint it introduces.

Choose specialised AI hardware by the work it can unblock in your actual loop, then include the fleet trade-offs in the same decision.