difficulty 3/5
Semantic segmentation
A class label for every pixel. The output is the same size as the input, which forces an encoder-decoder shape and makes pixel accuracy a useless way to score it.
Intuition
Object detection answers "what is here, roughly where". Semantic segmentation answers it per pixel. d2l draws the contrast directly: "different from object detection, semantic segmentation recognizes and understands what are in images in pixel level: its labeling and prediction of semantic regions are in pixel level. Compared with in object detection, the pixel-level borders labeled in semantic segmentation are obviously more fine-grained" [1, §14.9].
Two consequences follow, and they organise the rest of this page. The output is the same shape as the input, so the architecture has to shrink and then grow again. And the classes are wildly unbalanced by construction — most pixels in most images are background — so the obvious metric is worthless.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Vision tasks →
Sources
- Dive into Deep Learning — Semantic Segmentationbook · retrieved 2026-09-14https://d2l.ai/chapter_computer-vision/semantic-segmentation-and-dataset.html
- Dive into Deep Learning — Fully Convolutional Networksbook · retrieved 2026-09-14https://d2l.ai/chapter_computer-vision/fcn.html
- U-Net: Convolutional Networks for Biomedical Image Segmentation (2015)paperhttps://ar5iv.labs.arxiv.org/html/1505.04597