difficulty 4/5
Instance segmentation
A mask per object, not a class per pixel. It is detection and segmentation at once, and the thing that made it work was removing two roundings.
Intuition
Detection tells you there are three sheep and roughly where. Semantic segmentation tells you which pixels are sheep. Neither tells you which pixels are the middle sheep.
Instance segmentation does both at once, and He et al. are precise about why that is hard: it "requires the correct detection of all objects in an image while also precisely segmenting each instance. It therefore combines elements from the classical computer vision tasks of object detection, where the goal is to classify individual objects and localize each using a bounding box, and semantic segmentation, where the goal is to classify each pixel into a fixed set of categories without differentiating object instances" [1, §1].
The surprise of Mask R-CNN is how little machinery this needed. Take a detector, bolt a small segmentation head onto each region it proposes, and stop two roundings from happening. That is essentially the whole paper, and it beat the heavily-engineered COCO 2016 winner [1, Abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Vision tasks →
Sources
- Mask R-CNN (2017)paperhttps://ar5iv.labs.arxiv.org/html/1703.06870
- Microsoft COCO: Common Objects in Context (2014)paperhttps://ar5iv.labs.arxiv.org/html/1405.0312