difficulty 3/5
Random forest
Bagged decision trees with a second source of randomness — each split may only consider a random subset of the features, which decorrelates the trees and lets averaging do far more work.
Intuition
Bagging averages trees grown on bootstrap resamples, and it hits a wall: the resamples overlap, so the trees are similar, and averaging similar things barely helps. If one feature is strongly predictive, every tree splits on it first, and forty trees that all begin the same way are close to one tree repeated forty times.
A random forest breaks that agreement on purpose. At each split, the tree is only allowed to look at a random handful of the features — so the dominant feature is simply unavailable in many nodes, and the tree is forced to find whatever else is there. Individually the trees get worse. Together they get much better, because they now make different mistakes [1, §1.11.2.1].
That is the counterintuitive part worth holding onto: the algorithm deliberately weakens each member in order to strengthen the ensemble.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Tree ensembles →
Sources
- scikit-learn User Guide — Ensembles: Gradient boosting, random forests, bagging, voting, stackingdocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/ensemble.html
- A Random Forest Guided Tour (2015)paperhttps://ar5iv.labs.arxiv.org/html/1511.05741
- Understanding Random Forests: From Theory to Practice (2014)paperhttps://arxiv.org/abs/1407.7502