Write down the target first

Optimization without a target usually becomes a collection of impressive microbenchmarks. Decide whether the system needs lower end-to-end latency, more camera streams per GPU, lower cost, or better battery life.

Keep the quality requirement beside the performance target. A faster pipeline that misses small objects is not an improvement if small objects drive the application.

Profile before changing the model

Video decoding and memory copies often consume more time than expected. Instrument the complete pipeline and look at the distribution, not only the average.

P95 latency and frame drops tell a more useful story than a clean benchmark produced from a warmed-up tensor.

  • Benchmark with representative resolutions and codecs.
  • Separate cold start from steady-state performance.
  • Record CPU, GPU, memory, and queue metrics together.

Change one variable at a time

Test precision, input size, batch size, runtime, and model architecture independently. Keep the dataset and measurement window fixed so the result means something.

baseline = benchmark(fp32, batch=1, size=1280)
fp16    = benchmark(fp16, batch=1, size=1280)
batched = benchmark(fp16, batch=4, size=1280)
compare(latency, throughput, recall)

Validate on difficult examples

Optimization can change numerical behavior, resizing artifacts, and post-processing output. Re-run a focused evaluation on small objects, motion blur, occlusion, and low light after each major change.

The final choice should be a measured tradeoff that matches the operating environment, not the configuration with the highest synthetic throughput.