Object Detection Algorithms: How a Model Draws a Box

A detector does not decide where an object is. It scores several thousand candidate rectangles that were fixed before it saw the image, then deletes most of them. Understanding detection is mostly understanding that deletion step, because that is where the boxes you are unhappy about went.

What a detector actually returns

The output is a variable-length list. Each entry is a box, a class label and a score in [0, 1]. Nothing guarantees the list has one entry per object: it can hold three boxes for one car and none for the car behind it, and both of those are ordinary outcomes rather than bugs. The list is also unordered with respect to the image — there is no “first” object — so any downstream logic that depends on ordering has to impose its own.

Two numbers control the length of that list and they are usually confused. The score threshold drops boxes the model is not confident about. The NMS threshold drops boxes that overlap a higher-scoring box too much. They are applied in that order, they fail in different directions, and tuning the wrong one is the most common reason a detector that looked fine on a validation set produces nonsense on a real feed.

Anchor boxes: turning detection into classification

Regressing four coordinates directly from an image is a badly conditioned problem: the target has no fixed scale and no fixed position, and the loss has no idea which of the several objects present it should be pointing at. The anchor-box trick, introduced in the region proposal network of Faster R-CNN and adopted by SSD and the middle YOLO generations, converts it into something a convolutional network is good at.

Tile the image with a feature grid — say 40×40 cells for a 640×640 input at stride 16. At each cell, place a fixed set of rectangles: several scales crossed with several aspect ratios, commonly nine. Those are the anchors, and they are the same for every image the model will ever see. Now the network only has to answer two easy questions per anchor: does this anchor contain an object of class k, and what small offset would move this anchor onto the object exactly. The second is a regression on four numbers — typically (dx, dy, dw, dh) parameterised so that the width and height offsets are in log space, which keeps a doubling and a halving symmetric in the loss.

The consequence is arithmetic you can do in your head. 40×40 cells × 9 anchors is 14,400 candidates from one feature level, and a detector with three feature levels for three object scales has tens of thousands. Almost all of them are background. That extreme imbalance — often 1,000 negatives per positive — is the problem focal loss was designed for, and it is also why anchor design matters: if your objects are long and thin and your anchor set has no aspect ratio past 2:1, no anchor ever reaches a high enough IoU with a ground-truth box to be assigned as a positive, and the class never trains.

IoU, exactly

Intersection over union of two axis-aligned boxes is the area they share divided by the area they jointly cover. For boxes given as (x1, y1, x2, y2):

inter_w = max(0, min(a.x2, b.x2) - max(a.x1, b.x1))
inter_h = max(0, min(a.y2, b.y2) - max(a.y1, b.y1))
inter   = inter_w * inter_h
union   = area(a) + area(b) - inter
iou     = inter / union

The two max(0, ...) clamps are the whole implementation: without them, two disjoint boxes produce negative widths whose product is positive, and you get a confident IoU for a pair that does not touch. It is worth noting what IoU is insensitive to. It is scale-free, so a 10-pixel error on a 30-pixel box scores far worse than the same error on a 300-pixel box. And it is zero for any two boxes that miss each other, however narrowly, which is why the gradient from a pure IoU loss vanishes for a badly placed prediction — the motivation for GIoU and its successors.

The 0.5 threshold everyone quotes is the Pascal VOC criterion: a detection counts as correct if its overlap with a ground-truth box exceeds 50%, as set out by Everingham and colleagues in the IJCV Pascal VOC papers. COCO does something stricter: its headline AP averages precision over ten IoU thresholds from 0.50 to 0.95 in steps of 0.05, which you can read directly in pycocotools’ cocoeval.py as iouThrs. A COCO AP of 0.45 and a VOC mAP of 0.45 are not comparable quantities, and papers that put them in one table are comparing nothing.

A non-max suppression pass, worked

Greedy NMS runs per class. Sort the surviving boxes by score, take the highest, emit it, delete every remaining box whose IoU with it exceeds the threshold, and repeat on what is left. Here are four boxes from one street scene, coordinates in pixels:

A  (100, 100, 200, 260)  score 0.92   area 100 x 160 = 16000
B  (108, 112, 204, 256)  score 0.85   area  96 x 144 = 13824
C  (150, 100, 250, 260)  score 0.71   area 100 x 160 = 16000
D  (400, 120, 470, 300)  score 0.66   area  70 x 180 = 12600

A vs B   inter = (200-108) x (256-112) = 92 x 144 = 13248
         union = 16000 + 13824 - 13248  = 16576
         IoU   = 13248 / 16576          = 0.799

A vs C   inter = (200-150) x (260-100) = 50 x 160 =  8000
         union = 16000 + 16000 -  8000  = 24000
         IoU   =  8000 / 24000          = 0.333

A vs D   no overlap in x                = 0.000

At the common threshold of 0.5: A is emitted, B is suppressed at 0.799, C survives at 0.333, D survives at 0.000. The output is three boxes. If A and C are in fact two boxes on the same car — one hugging the body, one that has drifted right onto the wing mirror — you have a duplicate, and the obvious fix is to lower the NMS threshold to 0.3 so that C is suppressed too.

Why the NMS threshold has no safe value

Now suppose A and C were two cars parked bumper to bumper. Their true boxes overlap at IoU 0.333 because the near car occludes part of the far one, and the threshold you just lowered to 0.3 deletes the second car entirely. Nothing in the score, the class or the geometry distinguishes “two boxes on one object” from “two objects that overlap”, because greedy NMS sees only the IoU.

That is the trade in full: raise the threshold and you keep duplicates, lower it and you delete real objects in crowds. The recall ceiling this imposes is fixed before the model runs — no amount of extra training capacity recovers an object whose only surviving box was suppressed. Soft-NMS, from Bodla and colleagues in 2017, changes the delete into a score decay, so an overlapping box is demoted rather than erased and can still be recovered by a lower score threshold; the set-prediction detectors in the DETR line remove the step entirely by training a bipartite matching loss that penalises the model for emitting two boxes for one object in the first place. Both are answers to the same structural problem, and both are the reason to reach for when your failure mode is objects hidden behind other objects rather than a badly trained backbone.

One more trap: NMS is per class by default, so a box labelled car never suppresses a box labelled truck at the same coordinates. If your consumer expects one detection per physical object, you need class-agnostic NMS as a second pass, and you have to decide which label wins.

What anchor-free detectors changed

Anchor-free designs — FCOS, CenterNet, and the later YOLO generations — drop the fixed rectangle set. Instead, each feature location predicts whether it is inside an object and how far it is to the object’s four edges, or predicts a heatmap peak at the object’s centre. The practical gain is that the aspect-ratio failure above disappears: there is no anchor to mismatch, so an unusually shaped class trains without you hand-designing for it. What does not disappear is duplicate suppression, in any design that emits one prediction per location.

Whichever family you use, the coordinate convention in your labels has to match the one your training code expects, and it usually does not by default — see how COCO, YOLO and Pascal VOC each write the same box. And if what you need is not a rectangle at all but the exact pixels of each object, the task is instance segmentation, which changes both the output shape and the metric.