Perception · Lesson 7 of 12

Deep learning for robot perception: CNNs and detection

How CNNs find objects for a robot arm: convolution and pooling, detector outputs, IoU and NMS, precision, recall and mAP, and a 3D target from a 2D box.

intermediate55 minUpdated Fact-checked Free

What you will learn

  • Compute a 2D convolution, a max-pool and the output size of a convolution layer
  • Calculate IoU between two boxes and run non-maximum suppression by hand and in NumPy
  • Count true positives, false positives and false negatives and turn them into precision, recall and average precision
  • Plan the fine-tuning of a pretrained detector on a small lab dataset
  • Convert a 2D detection into a 3D position in the robot base frame with the pinhole camera model

Before you start

  • Camera geometry (lesson 6)
  • Spatial maths (lesson 2)
  • Python and NumPy

After this lesson you can explain what each layer of a convolutional network computes, read the output of an object detector, and score a detector with IoU, precision, recall and average precision. You will also turn a detected box into a 3D target for the SO-101 arm using the camera model from camera geometry, and plan a small labelled dataset for your own lab objects.

From hand-crafted features to learned features

A colour threshold finds a red cube on a white table until the light changes or a second red object appears. Classical vision added hand-designed features, such as gradient histograms and keypoint descriptors, with a trained classifier on top; the features stayed fixed. A convolutional neural network (CNN) learns the features as well, from labelled images. You collect and label data instead of tuning thresholds, and the model only knows what the data showed it.

Convolution, the building block

A convolution layer slides a small kernel ww of size kh×kwk_h \times k_w over the image xx and takes a weighted sum at every position:

y[i,j]=∑m=0kh−1∑n=0kw−1x[i⋅s+m,  j⋅s+n] w[m,n]+by[i, j] = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} x[i \cdot s + m,\; j \cdot s + n]\, w[m, n] + b

Here ss is the stride and bb a bias. Strictly this is cross-correlation: a mathematical convolution flips the kernel first. CNN libraries do not flip, which is harmless for a learned kernel, so in SciPy correlate2d matches a CNN layer and convolve2d does not.

With zero padding pp on each side, the output length along one axis is

nout=⌊n+2p−ks⌋+1n_{out} = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1

A 3×33 \times 3 kernel with s=1s = 1, p=1p = 1 keeps the size. With s=2s = 2 it halves it: a 640-pixel-wide image becomes 320 wide.

Real layers map CC input channels to KK output channels with a K×C×kh×kwK \times C \times k_h \times k_w weight array; each output map sums the correlations of all input channels with its own kernels. A 3×33 \times 3 layer from 64 to 128 channels has 3⋅3⋅64⋅128+128=73 8563 \cdot 3 \cdot 64 \cdot 128 + 128 = 73\,856 parameters, whatever the image size. Compare a fully connected layer from a 640×480×3640 \times 480 \times 3 image to only 1000 units: 921.6 million weights. Weight sharing across positions is what makes CNNs trainable on images.

The download implements this with plain loops and checks it against SciPy:

def conv2d(image, kernel, stride=1, padding=0):
    x = np.asarray(image, dtype=float)
    w = np.asarray(kernel, dtype=float)
    if padding > 0:
        x = np.pad(x, padding, mode="constant")
    kh, kw = w.shape
    oh = (x.shape[0] - kh) // stride + 1
    ow = (x.shape[1] - kw) // stride + 1
    out = np.empty((oh, ow))
    for i in range(oh):
        for j in range(ow):
            r, c = i * stride, j * stride
            out[i, j] = np.sum(x[r : r + kh, c : c + kw] * w)
    return out

On a 6×66 \times 6 image with a vertical step from 0 to 1, a Sobel kernel gives 4 in the two output columns whose window straddles the step and 0 elsewhere: an edge detector, much like the kernels a trained first layer learns.

Download conv.py Download test_conv.py
cd public/code/robotics-ai/deep-perception
/Volumes/M2-Dev/conda/envs/toptech/bin/python -m pytest -q test_conv.py
# 10 passed

The tests compare conv2d with scipy.signal.correlate2d, including padding and stride, and confirm that convolve2d gives the flipped-kernel answer.

Nonlinearity and pooling

Stacked linear layers are still one linear map, so each convolution is followed by a nonlinearity, usually the ReLU, max⁡(0,x)\max(0, x). A 2×22 \times 2 max pool with stride 2 keeps the largest value in each window, halves the width and height, and tolerates shifts of a pixel or two. Many networks use stride-2 convolutions for the same downsampling. max_pool2d in the download implements pooling.

Feature hierarchies

Each output value depends on a patch of the input called its receptive field. One 3×33 \times 3 layer sees 3 pixels across; LL stacked 3×33 \times 3 layers with stride 1 see 2L+12L + 1. Strides and pooling multiply the growth, so after a few downsampling stages a single feature covers a large part of the image. Early layers respond to edges and colour blobs, middle layers to textures and parts, and late layers to whole objects.

The network that turns an image into these features is the backbone. The ACT paper, used in lesson 9, reports that its ResNet-18 backbone maps a 480×640×3480 \times 640 \times 3 image to a 15×20×51215 \times 20 \times 512 feature map: 32 times smaller in each direction, with 512 channels instead of 3.

What a detector outputs

A detector says what is in the image and where. For each object it returns:

  • a box, here (x1,y1,x2,y2)(x_1, y_1, x_2, y_2) in pixels with yy pointing down the image (other tools use centre, width and height);
  • a class, such as cube or cup;
  • a score between 0 and 1, the model’s confidence.

Internally a detector scores many candidate boxes over positions and sizes, from the backbone’s feature maps, either in one pass or by first proposing regions and then classifying them. One object produces several overlapping high-scoring candidates. Post-processing drops boxes below a score threshold, then runs non-maximum suppression.

IoU and non-maximum suppression, with numbers

Intersection over union compares two boxes:

IoU(A,B)=area(A∩B)area(A)+area(B)−area(A∩B)\text{IoU}(A, B) = \frac{\text{area}(A \cap B)}{\text{area}(A) + \text{area}(B) - \text{area}(A \cap B)}

It is 1 for identical boxes and 0 for disjoint ones. Take A=(100,100,200,200)A = (100, 100, 200, 200) and B=(120,110,220,210)B = (120, 110, 220, 210). Both are 100×100100 \times 100, so each has area 10 000. The intersection runs from x=120x = 120 to 200 and y=110y = 110 to 200, which is 80×90=720080 \times 90 = 7200. The union is 10 000+10 000−7200=12 80010\,000 + 10\,000 - 7200 = 12\,800, and IoU =7200/12 800=0.5625= 7200 / 12\,800 = 0.5625.

Greedy non-maximum suppression (NMS) works per class:

  1. Sort the boxes by score.
  2. Keep the highest-scoring box.
  3. Remove every remaining box whose IoU with it exceeds a threshold, typically 0.5.
  4. Repeat with the boxes that are left.

In image processing the same name is used for a pixel-level operation that keeps a pixel only if it is greater than all its neighbours. Neubeck and Van Gool (2006) give fast algorithms for that version, not for box NMS.

Add two boxes to the example: C=(300,120,380,200)C = (300, 120, 380, 200) on a second cube, and D=(105,95,195,205)D = (105, 95, 195, 205), another hit on the first cube. The scores are A 0.92, B 0.85, C 0.60, D 0.40. The pairwise IoUs are A−B=0.563A{-}B = 0.563, A−D=0.826A{-}D = 0.826, B−D=0.558B{-}D = 0.558, and 0 for every pair with C. NMS keeps A, removes B and D because both overlap A by more than 0.5, then keeps C. Two boxes remain for two cubes.

def nms(boxes, scores, iou_threshold=0.5):
    b = np.asarray(boxes, dtype=float).reshape(-1, 4)
    order = np.argsort(-np.asarray(scores, dtype=float), kind="stable")
    keep = []
    while order.size > 0:
        best = int(order[0])
        keep.append(best)
        rest = order[1:]
        overlaps = iou_matrix(b[best], b[rest])[0]
        order = rest[overlaps <= iou_threshold]
    return np.array(keep, dtype=np.int64)

The threshold is a trade-off. Too low, and two real objects that touch (two cubes side by side) suppress each other. Too high, and duplicates survive.

Two panels showing a 640 by 480 pixel image with three dashed ground-truth boxes. On the left, 17 blue detection boxes cluster in groups of five around each object, plus two isolated boxes. On the right, after non-maximum suppression, one box remains on each object with scores 0.90, 0.89 and 0.89, plus the two isolated boxes with scores 0.35 and 0.22.
Synthetic detections before and after non-maximum suppression at an IoU threshold of 0.5. The clusters collapse to one box per object; the two low-scoring false positives survive NMS and must be removed by a score threshold.

Precision, recall and average precision

To score a detector, go through its detections from highest to lowest score. A detection is a true positive (TP) if its IoU with a not-yet-matched ground-truth box of the same class reaches a threshold, often 0.5. Otherwise it is a false positive (FP); a second box on an already matched object is one too. Unmatched ground-truth objects are false negatives (FN).

precision=TPTP+FPrecall=TPTP+FN\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN}

Precision answers “when the robot reaches for a detection, how often is something there?” Recall answers “of the objects on the table, how many did it find?”

Continue the example with three labelled objects: the two cubes and a third object at (450,300,520,380)(450, 300, 520, 380) that the detector missed. Keep detections with a score of at least 0.5 and skip NMS:

SettingTPFPFNPrecisionRecall
Score at least 0.5, no NMS2110.6670.667
Score at least 0.5, after NMS2011.0000.667
Score at least 0.9, no NMS1021.0000.333

Box B is the false positive in the first row: a duplicate on cube A. NMS removes it and precision goes to 1. The missed object caps recall at 2/3 regardless of thresholds.

Sweeping the score threshold from high to low traces a precision-recall curve. Average precision (AP) is the area under it, after replacing each precision value by the best precision at the same or any higher recall (the envelope). Mean average precision (mAP) is the mean of AP over classes. Benchmarks differ in the IoU threshold they use, and some average AP over several thresholds, so check the definition before comparing numbers between papers or tools.

Precision on the vertical axis against recall on the horizontal axis for a simulated detector. The blue curve for IoU threshold 0.5 stays above precision 0.93 up to recall 0.6, then falls to about 0.51 at recall 0.85; its AP is 0.81. The amber curve for IoU threshold 0.75 stays between precision 0.69 and 0.77 from recall 0.1 to 0.6, then falls to about 0.41 at recall 0.69; its AP is 0.52.
Precision-recall curves for a simulated detector on 60 images with 150 objects. A stricter IoU threshold of 0.75 turns loosely placed boxes into false positives, so the same detections score an AP of 0.52 instead of 0.81. Neither curve reaches recall 1 because some objects were never detected.

For a robot, pick the operating point from the cost of each error. A false positive sends the gripper to an empty spot; a false negative leaves an object behind. A system that can retry usually favours recall.

Download iou.py Download test_iou.py

iou.py contains iou, iou_matrix, nms, match_detections, precision_recall, pr_curve and average_precision. Running it prints the numbers above, and test_iou.py checks them.

Pretrained backbones and fine-tuning

You rarely train a detector from scratch for a lab task. Backbones pretrained on large labelled image collections already compute general features (edges, textures, object parts). You keep those and train the task-specific layers on your own images. This is fine-tuning. It needs far fewer labelled images than training from scratch, because the network only has to adapt features it already has.

A typical plan:

  1. Start from a detector pretrained on a large public dataset, with its class head replaced by your classes.
  2. Freeze the backbone and train only the head. This is fast and hard to overfit.
  3. Unfreeze the backbone and train everything for a few more epochs with a smaller learning rate (for example ten times smaller), so the pretrained features change slowly.
  4. Watch validation AP after each epoch and keep the best checkpoint.

Start small, for example 100 labelled images, measure, then add data where the errors are. Split training and validation by recording session, not by random frame. Consecutive video frames are near copies, and a random split puts copies on both sides, which inflates validation scores. Use augmentation: brightness and colour jitter, small scale changes and crops. Flip images only if the task does not depend on left and right.

Segmentation and keypoints

A box around a mug includes table pixels and says nothing about the handle. Two finer outputs are common in manipulation:

  • Segmentation labels every pixel. Instance segmentation also separates objects, so two touching cubes get two masks. A mask gives a better centroid and, with depth, a point cloud of just the object. See fully convolutional networks and Mask R-CNN in the further reading.
  • Keypoints are named points, such as a mug’s handle. A keypoint network predicts one heatmap per point. Keypoints give orientation, which a box does not, and map directly to grasp points.

Both cost more labelling time per image than boxes.

From a 2D detection to a 3D position

The arm needs a target in metres in its base frame, not a box in pixels. The pinhole model from camera geometry links the two. With intrinsics fx,fy,cx,cyf_x, f_y, c_x, c_y, a point at depth ZZ along the optical axis that projects to pixel (u,v)(u, v) lies at

X=(u−cx) Zfx,Y=(v−cy) ZfyX = \frac{(u - c_x)\, Z}{f_x}, \qquad Y = \frac{(v - c_y)\, Z}{f_y}

in the camera frame. One pixel gives a ray; the depth ZZ has to come from somewhere else. Three common sources:

  1. A depth camera gives ZZ at the pixel directly. Take the median depth over the box or mask, not one pixel, to reject holes and edge noise.
  2. Known object size. An object of width WW that appears ww pixels wide is at Z=fxW/wZ = f_x W / w, if it faces the camera.
  3. Known support plane. Objects on a table lie on a plane you can calibrate once. Intersect the pixel ray K−1(u,v,1)TK^{-1}(u, v, 1)^T with the plane n⋅X=d\mathbf{n} \cdot \mathbf{X} = d.

Worked example with example intrinsics for a 640×480640 \times 480 image: fx=fy=600f_x = f_y = 600 px, cx=320c_x = 320, cy=240c_y = 240. The detector returns a box (376,276,424,324)(376, 276, 424, 324) on a 4 cm cube. Its centre is (400,300)(400, 300) and its width 48 px.

From the size: Z=600⋅0.04/48=0.50Z = 600 \cdot 0.04 / 48 = 0.50 m, then X=80⋅0.5/600=0.0667X = 80 \cdot 0.5 / 600 = 0.0667 m and Y=60⋅0.5/600=0.050Y = 60 \cdot 0.5 / 600 = 0.050 m. This is a point on the cube’s top face, which is what the camera sees.

From the plane: the camera looks straight down from 0.54 m above the table. The cube centre is 2 cm above the table, so intersect the ray with the plane Z=0.52Z = 0.52 m: (0.0693,0.052,0.52)(0.0693, 0.052, 0.52) m. The two methods agree: a cube on a table 0.54 m away has its top face at 0.50 m. The XX and YY values differ slightly because the size method finds the top face and the plane method the centre.

Finally map the point to the base frame with the extrinsic transform Tbase,camT_{base,cam} from spatial maths: pbase=R pcam+t\mathbf{p}_{base} = R\, \mathbf{p}_{cam} + \mathbf{t}. With the camera at (0.30,0,0.54)(0.30, 0, 0.54) m looking down, the cube centre is at (0.248,−0.069,0.020)(0.248, -0.069, 0.020) m in the base frame, ready for inverse kinematics.

Download lift.py Download test_lift.py

Data collection and labelling in the lab

  • Record what the robot will see: the deployment camera, mount, resolution and lighting. A detector trained on phone photos fails on a wrist camera.
  • Vary what will vary: object pose, distractors, daylight at different times, occlusion by the gripper.
  • Hold back a test set from a separate session and never tune on it.
  • Write a labelling guide first: how tight a box is, whether half-hidden objects count. Label 20 images twice and compute the IoU between the two sets to check consistency.
  • Label the model’s failures (missed objects, false positives on shadows) and add them to the training set.
  • Keep the provenance: session, camera and date with each image.
Exercise

IoU, NMS and scoring by hand

A detector returns four boxes for class cube:

Box(x1,y1,x2,y2)(x_1, y_1, x_2, y_2)Score
P1(50, 60, 150, 140)0.9
P2(60, 70, 160, 150)0.8
P3(200, 60, 280, 140)0.7
P4(55, 55, 145, 145)0.6

The ground truth is G1=(52,58,152,142)G_1 = (52, 58, 152, 142) and G2=(205,62,285,138)G_2 = (205, 62, 285, 138).

  1. Compute IoU(P1, P2) and IoU(P1, P4).
  2. Run NMS with a threshold of 0.5. Which boxes are kept?
  3. Compute precision and recall at IoU 0.5 before and after NMS (all scores are above 0.5).
Show solution
  1. P1 and P2 are both 100×80=8000100 \times 80 = 8000. The intersection is xx 60 to 150 and yy 70 to 140, so 90×70=630090 \times 70 = 6300. IoU =6300/(8000+8000−6300)=6300/9700=0.650= 6300 / (8000 + 8000 - 6300) = 6300 / 9700 = 0.650. P4 is 90×90=810090 \times 90 = 8100; the intersection with P1 is 90×80=720090 \times 80 = 7200, so IoU =7200/(8000+8100−7200)=7200/8900=0.809= 7200 / (8000 + 8100 - 7200) = 7200 / 8900 = 0.809.
  2. P1 is kept first. P2 (0.650) and P4 (0.809) exceed 0.5 and are removed. P3 does not overlap P1 and is kept. Result: P1 and P3.
  3. Before NMS: P1 matches G1G_1 (IoU 0.916), P3 matches G2G_2 (IoU 0.841), and P2 and P4 are duplicates on G1G_1, so TP = 2, FP = 2, FN = 0. Precision =0.5= 0.5, recall =1.0= 1.0. After NMS: TP = 2, FP = 0, FN = 0, so precision and recall are both 1.0.

Check your answers with iou_matrix, nms and precision_recall from iou.py.

Exercise

How far away is the bottle?

With the same intrinsics as above (fx=fy=600f_x = f_y = 600, cx=320c_x = 320, cy=240c_y = 240), a 6 cm wide bottle is detected with a box 30 px wide centred at (260,330)(260, 330).

  1. Estimate its depth and its position in the camera frame.
  2. How much does the depth change if the box is one pixel wider or narrower?
Show solution
  1. Z=600⋅0.06/30=1.20Z = 600 \cdot 0.06 / 30 = 1.20 m. X=(260−320)⋅1.2/600=−0.12X = (260 - 320) \cdot 1.2 / 600 = -0.12 m and Y=(330−240)⋅1.2/600=0.18Y = (330 - 240) \cdot 1.2 / 600 = 0.18 m.
  2. At 31 px, Z=1.161Z = 1.161 m; at 29 px, Z=1.241Z = 1.241 m. One pixel moves the estimate by about 4 cm, about four times the error for the cube at 48 px. Small, distant objects need a depth camera or the table-plane method.

Summary

  • A convolution layer slides shared kernels over the input; CNN libraries compute cross-correlation, and the output size is ⌊(n+2p−k)/s⌋+1\lfloor (n + 2p - k)/s \rfloor + 1. ReLU and pooling or strided convolutions follow, and stacked layers build features from edges up to objects.
  • A detector outputs boxes, classes and scores. A score threshold and per-class non-maximum suppression turn many overlapping candidates into one box per object.
  • IoU measures box overlap. With an IoU threshold, detections become true or false positives, and missed objects false negatives. Precision and recall follow, and AP is the area under the precision-recall envelope.
  • Fine-tune a pretrained detector instead of training from scratch, split data by session, and add the model’s own failures to the training set.
  • A box plus the pinhole model gives a ray; depth from a depth camera, a known object size or a known table plane fixes the point, and the extrinsic transform puts it in the base frame.

Check your understanding

Pick an answer for each question, then check. Answers stay hidden until you submit.

1. A 3 x 3 convolution with stride 2 and padding 1 is applied to a 640 x 480 image. What is the output size?
2. Box A is (0, 0, 10, 10) and box B is (5, 0, 15, 10). What is their IoU?
3. Before NMS, a detector outputs two boxes on the same cube, both with IoU 0.9 against the ground truth. How are they counted at an IoU threshold of 0.5?
4. You raise the detector's score threshold from 0.3 to 0.8. What usually happens?
5. With fx = 600 px, a 5 cm wide object appears 40 px wide. Roughly how far is it from the camera?

Further reading

Found a mistake or something unclear?Report a correction.