difficulty 4/5
Gradient boosting
Boosting generalised to any differentiable loss — each new tree is fitted to the negative gradient of the loss at the current predictions, and added with a shrinkage factor.
Intuition
AdaBoost reweights examples. Gradient boosting does something that turns out to be more general and, once seen, simpler: it fits each new model to the mistakes themselves.
For squared error that is literally the residual — what is left over after the current ensemble has had its say. Fit a small tree to those leftovers, add it on, recompute the leftovers, repeat. Every round the target changes, and the target is always "the part we have not explained yet".
The generalisation is where the name comes from. Residuals are the negative gradient of squared loss with respect to the prediction. Swap in a different loss and the negative gradient is something else — but the procedure is unchanged. That single substitution turns one algorithm into a framework for any differentiable objective [1, §1.11.1.2.3].
It is gradient descent, performed in the space of functions rather than the space of parameters, one tree per step.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Tree ensembles →
Sources
- scikit-learn User Guide — Ensembles: Gradient boosting, random forests, bagging, voting, stackingdocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/ensemble.html
- XGBoost: A Scalable Tree Boosting System (2016)paperhttps://ar5iv.labs.arxiv.org/html/1603.02754
- LightGBM: A Highly Efficient Gradient Boosting Decision Tree (2017)paperhttps://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html