Detection has no memory

A detector looks at one frame and draws boxes. It has no idea that the person in this frame is the same person as in the last one. For counting, dwell time, near-miss alerts or anything that happens over time, you need a tracker on top: the part that gives each object an ID and keeps it.

Most production trackers, from SORT to ByteTrack, follow the same simple loop. The figure shows it on six frames of a short clip.

Predict, then match by overlap

For each track, the tracker first predicts where the object should be in the new frame, here by assuming it keeps moving at the same speed. That's the dashed ghost box. Then it compares every prediction with every new detection by overlap, intersection over union, and picks the pairing with the highest total overlap.

A match only counts if the overlap is above a gate. In the figure the gate is 0.3: person A's prediction and detection overlap 0.642 at frame 4, person B's 0.366, so both match and both keep their IDs.

What happens when people cross

Crossings are where trackers fail. At frame 4 in the figure, A and B pass each other. Lower the gate to 0.1 and a weak 0.230 pair is now allowed, and the wrong pairing suddenly scores higher in total: 0.840 plus 0.230 beats 0.642 plus 0.366. A gets B's ID and B gets A's. That's an ID switch, and the figure counts two of them.

Set the gate too high and the opposite happens: at 0.5, the same clip breaks into 11 IDs for 4 people, because the first predictions don't overlap enough. The right gate depends on frame rate, how fast things move, and how crowded the scene is.

Misses and new arrivals

Detectors miss things, especially behind other objects. A good tracker doesn't drop an ID on the first miss. In the figure, person C is missed at frame 3, the track waits, and C is found again at frame 4 with the same ID. When C leaves for good, the track is dropped after two misses. Person D walks in at frame 6 and gets a new ID.

What I tune in production

  • The match gate and how long a lost track waits, on footage from the real site.
  • Frame rate: at higher frame rates boxes overlap more between frames, and matching gets easier.
  • Appearance features for crowded scenes, so the tracker can tell two similar boxes apart.
  • ID switches and fragmented tracks as metrics, not just detection accuracy.
  • Counting lines and zones on the tracks, not on raw detections, so one person is counted once.