Start with the decision, not the model

The model is one component in a much larger decision system. Define the operational outcome, latency budget, failure cost, and human fallback before choosing an architecture.

A warehouse alert, a batch inspection report, and an assistive camera may use similar models, but their reliability requirements are completely different.

  • Define the decision produced by each inference.
  • Measure acceptable false positives and false negatives separately.
  • Document latency, throughput, privacy, and hardware constraints.

Treat data as a production dependency

Training data needs versioning, validation, and ownership. A model score is not meaningful if the dataset silently changes or no longer represents the cameras and environments in production.

I version images, labels, class maps, split assignments, and preprocessing settings together. That makes an experiment reproducible and a production regression traceable.

  • Reject corrupt images and invalid coordinates before training.
  • Split video data by source sequence to prevent leakage.
  • Keep a frozen evaluation set that reflects real operating conditions.

Measure the complete path

End-to-end latency is not the same as inference latency. Decoding, network transfer, queueing, tracking, database writes, and UI updates can each become the bottleneck.

Measure timestamps at each boundary. A single aggregate timer hides the part of the system that needs work.

capture_ms + decode_ms + queue_ms
+ inference_ms + tracking_ms
+ decision_ms + delivery_ms
= end_to_end_latency

Design for observability

Monitor the system around the model as carefully as the model itself. Queue depth, decoding failures, inference time, confidence distribution, and downstream decision rates often reveal problems before an accuracy metric does.

Keep a small review stream of difficult or low-confidence examples. Those samples give the next dataset version a direct connection to production behavior.

  • Record model and dataset versions with every deployment.
  • Sample predictions for human review.
  • Create explicit rollback and degraded-operation paths.