Running inference on the device removes the network, the per-call cost and the data-residency question in a single move. It replaces them with a memory budget you cannot exceed, a thermal ceiling you cannot argue with, and a model that now ships and updates at firmware speed.
Quantisation, with the baseline stated
Quantisation stores weights at lower numeric precision. The memory saving is real and the throughput gain is often real. The figure quoted for it is frequently wrong, because it omits what it is measured from.
| From | To 8-bit | Note |
|---|---|---|
| 32-bit float | 4× smaller | The number everyone quotes. |
| 16-bit float | 2× smaller | Where most serving already sits, so the honest number. |
Quality loss is task-dependent and has to be measured on your own evaluation set, not assumed from a published average. Some tasks are almost unaffected. Some degrade in ways that only show on the inputs you care about most.
Memory is the binding constraint
The model does not have the device to itself. It shares with the operating system, the application it exists to serve, and whatever else is resident. And the peak matters more than the average: loading frequently needs more than running.
- Budget against the peak, on the weakest device you intend to support.
- Test with the application running, not with the model alone.
- Decide what happens when allocation fails — degrade to a remote call, or refuse.
The constraint nobody benchmarks
A phone or an embedded board will run fast for a few seconds and then throttle. A benchmark on a cool device measures the best case you will see once, not the behaviour of a workload that runs continuously.
Sustained throughput on a warm device is the number to design against. It is usually well below the headline, and it is the one your users experience.
The model is now firmware
A model shipped on-device updates through the firmware path. That brings the whole firmware reality with it:
- Staged rollout and rollback, because a bad model on a fleet is not a redeploy.
- Version skew. Several model versions will be live at once, for a long time.
- Devices that never update — switched off, out of coverage, or simply ignored.
- Size limits. Update payloads over constrained links have practical ceilings.
If your deployment cannot tolerate three versions behaving slightly differently in the field simultaneously, the model may not belong on the device.
When it is the right answer
On-device earns its complexity when the network is unreliable or absent, when latency must be predictable rather than merely low, when per-call cost at volume is prohibitive, or when the data must not leave the device at all.
Where none of those holds, device inference tends to mean a weaker model on weaker hardware, chosen for a reason that did not survive contact with the requirement.
Scope the task, not the model
The recurring failure is expecting a small local model to behave like a large hosted one. It will not, and no amount of tuning changes that.
What works is narrowing what the device has to decide on its own: a classification, an extraction, a constrained command vocabulary, a first-pass filter that escalates anything uncertain. On that shape of problem a small model is not a compromise — it is the correct tool, and it is faster than the round trip would have been.