Queueing under concurrency
A model that answers in 400ms alone answers in four seconds when sixty requests arrive together and batching was never configured. The model did not get slower; the queue got deeper.
Inference · SLMs · Quantisation · Edge
Your demo ran in 400ms. Production runs at p99, and p99 is where revenue is.
Average latency is a vanity metric. The transaction you lose is the one in the tail — the slow call during peak, on the worst connection, when the queue is deepest. Inference engineering is the work of making the tail hold.
- The number that decides whether users stay
- p99
- Memory reduction from quantisation, depending on baseline precision
- 2–4×
- Per-call cost once a workload runs on-device
- 0
The cost
Why the tail breaks
Latency problems are rarely about the model. They are about everything around it.
Cold starts on the critical path
Scale-to-zero saves money until a user waits through a container start. The saving and the abandonment sit with different teams.
Round trips that did not need to happen
A chain of three sequential model calls where one structured call would do, because the prompt was never designed for the task.
Network distance to the model
A Gulf user routed to a US region adds hundreds of milliseconds before inference even begins. That is a placement decision, not a model decision.
The build
What we build
Inference that holds its budget when the system is busy, not when it is idle.
Latency budget per user action
A stated ceiling for each interaction, allocated across network, queue, inference and post-processing, so a regression is attributable instead of arguable.
Quantised deployment
INT8 and 4-bit variants evaluated against a task-specific quality floor, not a leaderboard. From an FP16 baseline that is roughly 2× at INT8 and 4× at 4-bit — the baseline matters, and most serving is already FP16.
On-device and edge SLMs
Small models running on the handset or at the edge for the high-frequency tail. No network, no per-call cost, no residency question.
Continuous batching
Throughput under concurrency without paying for idle capacity between peaks.
Speculative decoding where it pays
A draft model proposing tokens a larger one verifies. Real gains on structured output; less on open generation, and we say which you have.
Regional placement
Inference near the user. The cheapest hundred milliseconds you will ever recover.
The economics
Where the money moves
Two separate economies: the cost of serving, and the revenue lost to latency. The second is larger and almost never measured.
- 01On-device for high-frequency workPer-call cost goes to zero and stays there. The lever compounds with volume instead of decaying.
- 02Quantisation2–4× less memory depending on where you start, which means more concurrent requests per GPU and a smaller fleet for the same load.
- 03BatchingRaises utilisation under concurrency. You stop paying for idle silicon between peaks.
- 04Tail latency reductionShows up as conversion and completion, not as an infrastructure saving — which is why it stays invisible to the team that could fix it.
Why this firm
We built this control for ledgers first
A latency budget is a gate. It is stated before the work starts, measured against, and it either holds or the change does not ship. Same shape as a reconciliation gate, different unit.
Questions
Answered straight
It can, which is why the quality floor is established per task before anything is quantised. For classification, extraction and routing the floor usually holds comfortably at INT8. For open-ended reasoning it often does not, and we will say so before shipping the smaller number.
For the right task, routinely. Structured extraction, intent classification and tool selection are well within reach of a small model that has been evaluated on your data. For genuinely open reasoning, no — and mixing the two up is the most expensive mistake in this space.
Three things at once: no per-call cost, no network latency, and no data leaving the device — which removes the residency conversation entirely. Vantura exists because this is worth engineering for.
p95 and p99 under representative concurrency, not averages on an idle system. If a benchmark was run without load it tells you nothing about the request that will actually be lost.
Next
Tell us the interaction that feels slow.
We will instrument it end to end, show you where the milliseconds go, and tell you which ones are recoverable. Most teams are surprised by the split.