difficulty 4/5
Boosting
Fitting weak learners in sequence, each one on a version of the data reweighted to emphasise what the previous ones got wrong, and combining them into a weighted vote.
Intuition
Bagging runs its models in parallel and treats them as equals. Boosting runs them in single file and treats them as specialists: each new model is asked to concentrate on precisely the examples the committee so far keeps getting wrong.
The mechanism is a change of emphasis rather than a change of data. After each round, the examples that were misclassified become heavier, so the next learner — fitting the same way it always does — finds it worth splitting somewhere that helps them. The easy examples fade into the background because they are already handled [1, §1.11.7].
This is why boosting works with models that are barely worth anything alone. A single split on a diagonal boundary is a poor classifier. A hundred of them, each aimed at a different residual pocket of error, is a very good one.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Tree ensembles →
Sources
- scikit-learn User Guide — Ensembles: Gradient boosting, random forests, bagging, voting, stackingdocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/ensemble.html
- XGBoost: A Scalable Tree Boosting System (2016)paperhttps://arxiv.org/abs/1603.02754