difficulty 3/5
Bagging
Fitting the same unstable model to many bootstrap resamples of one dataset and averaging the results — variance falls, bias does not, and nothing about the model itself has to change.
Intuition
A decision tree grown to depth is a nervous model. Move one point, and a split near the root can flip to a different feature entirely, and every split beneath it changes. Two trees grown on two samples from the same population can disagree everywhere while both being reasonable.
That instability is not a defect to be fixed — it is a resource. A model that varies a lot around the right answer is exactly the kind whose average is better than any of its members. Fit it many times on slightly different data, average the predictions, and the idiosyncrasies cancel while the signal, which is the same in every fit, does not.
The trick is where the "slightly different data" comes from, because you only have one dataset. Bootstrap resampling manufactures it: draw n rows with replacement, fit, repeat [2, §7.11].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Tree ensembles →
Sources
- scikit-learn User Guide — Ensembles: Gradient boosting, random forests, bagging, voting, stackingdocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/ensemble.html
- Deep Learning (2016)book · retrieved 2026-10-07https://www.deeplearningbook.org/contents/regularization.html