4GB VRAM is not a 4GB capacity plan
In short: AirLLM runs Llama 3.x 70B on about 4GB of VRAM by streaming layers from disk, which moves the constraint to host RAM, disk, latency and token rate.
On 29 October 2026, a reel from Aman Arya put a familiar number back into circulation: Llama 3.x 70B, running inference on roughly 4GB of GPU VRAM. Gavin Li's AirLLM documents full-precision 70B inference on about 4GB of VRAM by keeping one decoder layer on the card and streaming the rest from disk.
A 4GB VRAM floor is not a 4GB capacity plan. Layer streaming does not shrink a 70B model; it relocates the constraint into host RAM, checkpoint-plus-shard disk, time to first token and steady token rate. The reel reports none of those four.
What Aman Arya's reel shows, and what it leaves out
I played the full 41.73-second reel and read the complete caption. Near 13 seconds it shows AirLLM's layer-by-layer streaming diagram, near 24 seconds the official GitHub repository, near 40 seconds the presenter closing the point.
What never reaches the screen is the rest of the sheet: no tokens per second, no first-token time, no GPU model, host RAM figure or disk type.
Why 4GB VRAM is not a 4GB capacity plan
AirLLM's README is straight about the trade. It splits the checkpoint into per-layer shards, warns that the split is disk intensive, and names disk loading as the main bottleneck. Hugging Face's big model inference guide describes the same ladder—GPU, then CPU, then disk—warns that disk offload can be very slow, and holds one hard condition: the largest layer must still fit on the execution device.
So “about 4GB” is a peak resident-weight budget on one card, not the machine's bill. The repository's GPU streaming test caps per-process CUDA allocation and reports peak allocated VRAM: the right instrument for exactly one line of five. I found no published 70B receipt carrying the other four.
The five-line receipt I would require before calling this deployable
This is labelled editorial judgment, not a Trigent engagement or a client result. I did not install AirLLM, and I did not reproduce the 70B figure on any hardware.
Before “runs on 4GB” reaches an approval memo, I want five measurements from the exact model revision and proposed machine.
| Line | Measure | What it catches |
|---|---|---|
| 1 | Peak GPU VRAM | the claim and the largest-layer floor |
| 2 | Peak host RAM | cache growth while layers stream |
| 3 | Checkpoint plus shard disk | the split copy, not the download |
| 4 | Time to first token | cold-start cost of disk loading |
| 5 | Steady tokens per second | whether a human can wait |
Fields for a local-inference acceptance receipt; every value comes from the proposed hardware, never a README.
Each line fails closed: a VRAM figure cannot approve a latency budget, and an NVMe workstation demo cannot approve a laptop with a slow disk and 16GB of RAM.
Where layer streaming holds and where it breaks
It holds where throughput is not the product: an offline extraction pass, or a one-off job on a card that could never hold the weights. Four gigabytes standing in for seventy billion parameters is useful engineering, and Gavin Li documents the cost instead of hiding it.
It breaks the moment you attach a human to the other end, because interactive sessions and agent loops are priced in tokens per second and first-token delay. One bound on my reading: I read the README and test at commit c5875cd on 30 October 2026, so later commits may move the behaviour.
What's in it for you
- Treat a VRAM number as one line of five, not a verdict on the machine.
- Ask for first-token latency and steady token rate before approving any offload-based deployment.
- Re-run the receipt when the model revision, quantisation, disk or host RAM changes.
A VRAM floor tells you the model will load; only disk, host RAM and the clock tell you whether anyone can use it.
