Two ways to say where something is
A detection model draws a rectangle around each object: four numbers. A segmentation model goes further and marks every pixel that belongs to the object. Both are useful, and the choice changes how much labelling you need, how fast the model runs, and what you can measure.
How much a box gets wrong
In the figure, the person's box covers 119 pixels but the person is only 83 of them, so 30% of the box is background. The pallet's box is 156 pixels for a 78-pixel pallet: half of it is background, because the person is standing in front of it.
The difference gets bigger when objects overlap. The two boxes share 84 pixels, an IoU of 0.44, as if the objects overlapped heavily. The masks share zero pixels, because no pixel belongs to both. Anything that reasons about overlap, area, or what's in front of what is working with the wrong numbers when it uses boxes.
When a box is enough
For counting, tracking, and most zone alerts, a box is plenty. You want to know that a person is there and roughly where, quickly, on modest hardware. Boxes are also much cheaper to label, which matters more than people expect when you build a real dataset.
When masks are worth it
- Measuring size or area: fill level, spill size, the share of a surface that's damaged.
- Defects with irregular shapes, like cracks, stains, or scratches, where a box is mostly background.
- Crowded or overlapping objects, where boxes overlap but the objects don't.
- Robot picking and anything that needs the exact edge.
- Cutting objects out for further processing, like reading a label on a specific parcel.