What latency is achievable for on-device computer vision?

Sub-10 ms per frame is achievable for vision-transformer inference on current mobile and embedded accelerators. That is comfortably inside a real-time interaction loop and roughly two orders of magnitude below what a cloud round trip can guarantee at the tail.

Hitting it depends on keeping the whole pipeline on the accelerator. Pre- and post-processing that bounces to the CPU often costs more than the inference itself, and shows up as jitter rather than as a higher average.

Measure the 99th percentile, not the mean. A vision loop that averages 6 ms but spikes to 40 ms under thermal load will feel broken even though the average looks excellent.

Written by Binary AI Labs · Reviewed 2026-11-01