difficulty 4/5
PointNet
A neural network that consumes a raw 3D point cloud directly, respecting the fact that the points are a set — reorder the file and the answer must not change.
Intuition
A lidar scan or a depth camera gives you a list of 3D coordinates. That list is not an image and not a sequence — it is a set. Save the same object twice and the points may come out in a different order, and nothing about the chair has changed.
Convolutional networks cannot consume that. They need a grid, so the standard move before 2017 was to build one: voxelize the cloud, or render it to images from several viewpoints. Both work, and both are wasteful in a specific way — the paper's own framing is that transforming point clouds to "regular 3D voxel grids or collections of images" "renders data unnecessarily voluminous and causes issues" [1, abstract]. Most of a voxel grid is empty, and you pay for all of it.
PointNet's contribution is to stop converting. It "directly consumes point clouds, which well respects the permutation invariance of points in the input" [1, abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2017)paperhttps://ar5iv.labs.arxiv.org/html/1612.00593
- Deep Sets (2017)paperhttps://arxiv.org/abs/1703.06114
- PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017)paperhttps://arxiv.org/abs/1706.02413