How small does a model need to be to run on an NPU?

Small enough to fit the accelerator's on-chip memory after quantisation. On microcontroller-class NPUs — the Cortex-M55 plus NPU class of part used in wearables — that means single-digit megabytes. This is an architecture constraint rather than a compression one: models are chosen for the budget, not squeezed into it afterwards.

The practical path is to pick a model family that quantises cleanly to 8-bit or 4-bit integers, then verify accuracy at the target precision before committing to the architecture. Post-hoc compression of a model that was never intended to be small usually costs more accuracy than choosing a smaller model to begin with.

Memory bandwidth, not raw compute, is usually the binding constraint. A model that fits but streams weights from external flash on every inference will miss its latency budget and drain the battery.

Brilliant Labs Halo runs its on-device workload on an Alif B1 with a colour microOLED and bone-conduction audio, and holds a 14-hour battery — a figure that only works because the model was sized to the part.

Written by Binary AI Labs · Reviewed 2026-11-01