Machine learning · 8 compared
Tree ensembles
Every method here is many decision trees combined into one prediction. They split into two families that want opposite things from their members, and that split explains most of the differences.
Comparison
| Name What it adds | Wants members that | Built | More members | Tuning burden |
|---|---|---|---|---|
| XGBoost 2016 A systems answer to the same algorithm — sparsity-aware splitting, a weighted quantile sketch, and cache-aware layout, scaling past billions of rows. | underfit individually | in sequence | can overfit | high |
| LightGBM 2017 Drops low-gradient instances and bundles mutually exclusive features, speeding up training by up to over 20× at almost the same accuracy. | underfit individually | in sequence | can overfit | high |
| Bagging Fit the same learner to bootstrap resamples and average. Attacks variance only, and needs a high-variance member to have anything to attack. Full page → | overfit individually | in parallel | never hurts | almost none |
| Random forest Bagging plus a random feature subset at every split. Decorrelating the trees lowers the floor that averaging cannot get past. Full page → | overfit individually | in parallel | never hurts | low |
| Extra-Trees Draws split thresholds at random instead of searching for the best one. More randomness, more individual bias, often faster to fit. | overfit individually | in parallel | never hurts | low |
| AdaBoost Reweights the training examples after each round so the next learner concentrates on what the committee keeps getting wrong. Full page → | underfit individually | in sequence | can overfit | moderate |
| Gradient boosting Fits each new tree to the negative gradient of the loss. The same procedure for any differentiable objective, not just classification error. Full page → | underfit individually | in sequence | can overfit | high |
| HistGradientBoosting Bins features into histograms before searching for splits. Orders of magnitude faster than plain gradient boosting once there are tens of thousands of rows. | underfit individually | in sequence | can overfit | high |
On the cheat sheet: Tree ensembles →
Lineage
- 2016XGBoost A systems answer to the same algorithm — sparsity-aware splitting, a weighted quantile sketch, and cache-aware layout, scaling past billions of rows. XGBoost: A Scalable Tree Boosting System the line ends here
- 2017LightGBM Drops low-gradient instances and bundles mutually exclusive features, speeding up training by up to over 20× at almost the same accuracy. LightGBM: A Highly Efficient Gradient Boosting Decision Tree the line ends here
- Bagging Full page → Fit the same learner to bootstrap resamples and average. Attacks variance only, and needs a high-variance member to have anything to attack. led to Random forest
- Random forest Full page → Bagging plus a random feature subset at every split. Decorrelating the trees lowers the floor that averaging cannot get past. the line ends here
- Extra-Trees Draws split thresholds at random instead of searching for the best one. More randomness, more individual bias, often faster to fit. the line ends here
- AdaBoost Full page → Reweights the training examples after each round so the next learner concentrates on what the committee keeps getting wrong. led to Gradient boosting
- Gradient boosting Full page → Fits each new tree to the negative gradient of the loss. The same procedure for any differentiable objective, not just classification error. the line ends here
- HistGradientBoosting Bins features into histograms before searching for splits. Orders of magnitude faster than plain gradient boosting once there are tens of thousands of rows. the line ends here
Read the Wants members that column first. It divides this table in two, and almost everything else follows from it.
Averaging methods want members that overfit. Bagging, random forests and Extra-Trees all take a low-bias, high-variance learner and cancel the variance by averaging. The documentation is direct about the requirement: these methods "work best with strong and complex models (e.g., fully developed decision trees)" [1, §1.11.3]. Adding members can only help, they train in parallel, and there is barely anything to tune.
Boosting methods want members that underfit. Each round fits what the previous rounds got wrong, so a member that leaves no error behind ends the process immediately. These "usually work best with weak models (e.g., shallow decision trees)" [1, §1.11.3]. In tree terms: "Random forests typically rely on deep trees (that overfit individually) … Boosting models build shallow trees (that underfit individually) which are faster to fit and predict" [1, §1.11.2].
That difference propagates all the way to the operational columns. Averaging is embarrassingly parallel and more members are free; boosting is inherently sequential and the number of rounds is a regularisation parameter you have to choose on held-out data.
The bottom half of the table is one algorithm
Gradient boosting, HistGradientBoosting, XGBoost and LightGBM are not four different models. They are gradient boosting with the split search made fast, and the reason they exist is stated plainly by the LightGBM authors: existing implementations "need to scan all the data instances to estimate the information gain of all possible split points, which is very time consuming" [3, abstract].
Three answers to the same bottleneck. Bin the features into histograms. Approximate the split search with a weighted quantile sketch and lay the data out for the cache [2, abstract]. Or reduce what must be scanned at all — LightGBM's Gradient-based One-Side Sampling drops instances with small gradients and Exclusive Feature Bundling merges features that are rarely nonzero together, reaching a speed-up "by up to over 20 times while achieving almost the same accuracy" [3, abstract].
Choosing between them is an engineering decision about your data's size and shape. Choosing between the two families is the modelling decision, and it is the one this table exists to make legible.
Which to reach for
Start with a random forest. It is hard to misuse, needs no scaling, and tells you within minutes whether the problem has signal in it.
Move to gradient boosting when accuracy is the objective and you have the budget to tune three interacting parameters. It usually wins on tabular data, and it will overfit silently if you let it.
Reach for Extra-Trees when fitting time matters and the data is wide — drawing thresholds at random removes the search that dominates the cost [1, §1.11.2].
One caution applies to every row: the feature importances these models expose by default are impurity-based, and impurity-based importance is biased toward high-cardinality features. See feature importance for what that costs and what to use instead.
Sources
- scikit-learn User Guide — Ensembles: Gradient boosting, random forests, bagging, voting, stackingdocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/ensemble.html
- XGBoost: A Scalable Tree Boosting System (2016)paperhttps://arxiv.org/abs/1603.02754
- LightGBM: A Highly Efficient Gradient Boosting Decision Tree (2017)paperhttps://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html