A model is mostly numbers

A trained detector is a few million numbers, its weights, plus the arithmetic that combines them with the image. During training those numbers are stored as 32-bit floats: very precise, and four bytes each. An 11-million-weight model like a small YOLO is about 45 MB in that form.

Inference doesn't need all that precision. Quantization stores the same weights, and does the same arithmetic, in fewer bits.

FP16: the easy win

Half precision keeps the floating-point format but uses 16 bits instead of 32. The model halves in size, there is half as much data to move through memory, and modern GPUs have dedicated hardware for 16-bit maths. In practice it's often close to twice as fast, and for most vision models the accuracy doesn't change in any way you can measure.

On a GPU deployment, FP16 is close to a default. I'd need a reason not to use it.

INT8: a quarter of the size, with a catch

INT8 goes further: each weight and activation becomes an 8-bit integer, one of only 256 values. To do that you have to decide which range of real values those 256 steps cover. Cover too wide a range and every step is coarse, so small values lose precision. Cover too narrow a range and big values get clipped.

The simplest choice, the minimum and maximum seen, is usually too wide, because a few outliers stretch the range. In the figure that's the INT8 min-max result: a quarter of the size, nearly three times faster, but a visible drop in accuracy.

Calibration fixes most of the loss

Calibration runs a few hundred real images through the model and picks a tighter range for each layer, one that fits where the values actually are. The rare outliers get clipped and everything else gets finer steps. Press calibrate in the figure and the clip range closes in while most of the lost accuracy comes back.

The calibration images matter. They should look like production: the same cameras, lighting, and scenes. Calibrating on a public dataset and deploying on a dark warehouse is a reliable way to lose accuracy you never noticed losing.

What about FP8?

FP8 is the newest step: floating point in 8 bits, so the same size as INT8 but without the separate value range. It comes in two flavours, E4M3 with more precision and E5M2 with more range, and it runs natively on recent NVIDIA data-centre and workstation GPUs, from the Hopper and Ada generations on.

Today FP8 matters most for large language models, where it halves memory against FP16 and speeds up both training and inference with little accuracy loss. For small vision detectors on edge devices, INT8 with calibration is still the more common choice, because that is what most edge hardware accelerates. If your target GPU supports FP8, it is worth measuring against INT8 on your own test set, the same way as any other precision.

How I decide

  • Start with FP16 on any GPU. It's almost free.
  • Move to INT8 when the hardware is the limit: an edge box, many cameras per GPU, or a hard latency budget.
  • Calibrate on images from the real site, and keep a held-out test set from the same place.
  • Compare accuracy per class, not just the overall number. Small or rare objects tend to suffer first.
  • If one class drops too far, keep the sensitive layers in FP16 and quantize the rest.