Edge export, measured latency, and what INT8 actually cost

The detector is exported to ONNX and served through ONNX Runtime's CPU execution provider — the same artefact and the same runtime a line-side box would use. Every number below was measured on this machine, and this machine is named, because a latency figure without hardware attached is decoration.

FP32 median latency

10.6 ms

p95 13.1 ms, batch 1

INT8 static median

14.1 ms

0.75× the FP32 speed

Model size

3.9× smaller

11.5 MB → 3.0 MB

Accuracy cost of INT8

-0.0004 AP

0.3443 → 0.3447

The headline result here is not the one we expected The literature quotes two to three times faster for INT8 on CPU. On this hardware static INT8 came out slower — 0.75× — while being 3.9 times smaller and, within noise, no less accurate. That is a measurement, not a mistake, and the explanation is on this page. Quoting the literature's number here would have been the easy thing and the wrong thing.

All three artefacts, measured on the same machine in the same session

Artefact	MB	median ms	best ms	p95 ms	COCO AP	note
FP32	11.5	10.56	8.91	13.09	0.3443	the reference
INT8 static QDQ, per-channel, percentile	3.0	14.10	7.08	20.47	0.3447	the configuration the research prescribes
INT8 dynamic	2.9	would not load `NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name '/stem/stem.0/Conv_quant'`				the configuration most teams reach for first

100 timed runs per artefact after 10 warm-up runs, batch 1, 4 intra-op threads, one inter-op thread. Accuracy is COCO AP@50:95 over the labelled defects in the test split, decoded by the same numpy code that produced every other number in this demonstration.

Why INT8 did not speed this up

This CPU is an Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz — Coffee Lake. It has AVX2 but no AVX-512 and no VNNI. The 2–3× figures in the literature are measured on server parts with VNNI, whose vpdpbusd instruction does an INT8 multiply-accumulate in one go. Without it, ONNX Runtime's INT8 convolution kernels have to widen and accumulate with more AVX2 instructions than the FP32 path needs, and the quantise/dequantise nodes around each convolution are pure overhead.

Note the best column: INT8's fastest run (7.08 ms) beats FP32's fastest (8.91 ms). The median and p95 do not, because this is a laptop with thermal headroom that varies. On a VNNI-capable server or an NPU the picture would very likely reverse — which is exactly why this belongs on a slide as “measure it on the target” rather than as a number to quote.

What INT8 unambiguously bought: 3.9× smaller, at an accuracy change of -0.0004 AP. On a device chosen for cost rather than speed, that size reduction is often the point.

The dynamic-quantisation trap, worse than advertised

The research pack warns that quantize_dynamic on a convolutional stack quantises weights only, leaves activations in float, and is frequently slower than FP32. On this graph the outcome is more decisive than that.

The dynamically quantised model does not run at all.

NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name '/stem/stem.0/Conv_quant'

It converts convolutions to ConvInteger, for which the ONNX Runtime CPU provider has no kernel, so the artefact fails at session creation. It exported without complaint and produced a plausible 2.9 MB file. A team that quantised this way would find out on the line.

The working recipe, unchanged from the research: static, QDQ format, per-channel weights, percentile calibration over 150 real images drawn from the training split only — calibrating on the test split would leak.

Against the cycle-time budget

Set a budget and see what fits. The default is 250 ms, a four-parts-per-second inspection station. It is an assumption, not a measurement, and it is yours to change.

Cycle-time budget, ms

What was measured on what

CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (x86_64), 12 logical cores
OS: macOS 14.7.6
Runtime: ONNX Runtime 1.23.2, CPUExecutionProvider
Threads: intra-op 4, inter-op 1
Input: 1×1×256×256 greyscale, batch 1
Runtime licence: MIT — onnxruntime; Apache-2.0 — onnx

Latency measured on this machine's CPU. A plant edge box will differ; the ratio between FP32 and INT8 travels better than the absolute numbers.

Quantisation was never validated on the GPU. Training used the Metal backend; every number on this page came from the CPU execution provider, because low-bit paths on Metal are emulated and upcast, and a model can look correct there and behave differently on the target.

precomputed Built by scripts/export_onnx.py at 2026-08-10T02:42:54+00:00. Measured on Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz. The budget arithmetic above is recomputed in the browser as you change the budget. Re-run that script to regenerate it; nothing on this page is hardcoded.