
On-device AI: models that run on your phone or laptop
In short: On-device AI runs compressed models on a phone, laptop or edge device instead of the cloud, trading model size for privacy, speed and offline use.
On-device AI means running AI models directly on a user's device, such as a phone, laptop, car or industrial sensor, instead of sending data to cloud servers. It relies on small, compressed models: Apple's June 2025 research update describes an on-device model of about 3 billion parameters quantised to 2 bits per weight. The benefits are privacy, low latency and offline use; the limits are model size, battery and memory.
On-device AI in plain words
It is keeping a pocket dictionary instead of phoning a library every time you need a word. The dictionary is less complete, but it is instant, private and works without signal.
How it works
Models are compressed, typically through distillation and quantisation, to fit device memory, then run on the device's own hardware. On Android, Gemini Nano runs inside a system service called AICore, which handles model distribution, updates and hardware acceleration, so the app does not ship its own copy of the model. Apple opens its on-device model to developers through the Foundation Models framework.
Both vendors describe a hybrid. Google's documentation, read in November 2026, notes that the local model has token limits and device hardware constraints, and points to hybrid inference that falls back to a cloud model when the device cannot serve the request. Apple pairs its on-device model with a larger server model running on Private Cloud Compute.
Example
As an illustration, a field-sales app summarises a voice note into CRM fields on the phone, with no network in a basement office. The rep reviews it and syncs when back online. No audio ever leaves the device.
Where you'll meet it
- Phone and laptop AI features from Apple and Google
- Developer frameworks for local models on Android, iOS and desktop
- Local model runners on laptops and workstations
- Edge deployments in vehicles, factories and clinics
What it changes for you
For product teams, on-device AI makes features possible where data cannot leave the device or connectivity is poor, as it often is in field operations. For developers, it adds model packaging, device fragmentation and update management to the work. For privacy and compliance teams, it can simplify consent and data-transfer questions.
On-device AI vs cloud AI
Cloud AI offers the largest models and easy updates but needs connectivity and sends data off-device. On-device AI is private and instant but limited in model size.
Common mistake
My view: shipping one large on-device model and finding it drains batteries and fails on mid-range phones. Test on the cheapest device your users actually own, and leave the developer's flagship for last.
Related terms
Small language model · Quantization · Distillation · Model routing · Inference
From the archive: On-device AI needs a feature resource ladder · Local models do not make local agents private
Test an on-device model on the cheapest phone your users actually own before you ship it.


