Bag or boost?
Five tree ensembles side by side, the pairs people confuse, and one run in which bagging cures variance and boosting cures bias.
prints on 2 A4 pagesfrom 5 pages and 1 cataloguefree to read · PDF with a free accountUpdated 23 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Tree ensembles catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Bagging averages members that overfit, all at once; boosting adds members that underfit, one after another.
Bag a model whose problem is variance; boost one whose problem is bias.
A forest is bagged trees that may each split on only a random subset of the features, so they disagree more.
The forest is the usual default; plain bagging can do better when few features carry the signal.
Random forest vs Gradient boosting
A forest is hard to get badly wrong and needs little tuning; boosting is usually more accurate on tables.
Start with a forest to see whether there is signal; tune boosting when the last points matter.
Learning rate vs Rounds
Each round adds a step of the learning rate's size, so a smaller rate needs more rounds for the same fit.
Fix a small rate and let early stopping choose the number of rounds.
One real run
Six models, each fitted on 100 samples of the same problem, so that their error splits into bias² and variance.
| Model | Bias² | Variance | Test MSE |
|---|---|---|---|
| stump | 16.67 | 3.01 | 20.68 |
| deep-tree | 3.65 | 9.14 | 13.79 |
| bagged-trees | 4.31 | 1.38 | 6.69 |
| random-forest | 5.81 | 0.86 | 7.67 |
| boosted-stumps | 2.23 | 1.01 | 4.24 |
| boosted-trees | 1.72 | 1.52 | 4.25 |
“Bagging cuts variance and leaves bias alone.”
“Boosting cuts bias.”
“Decorrelating the trees costs bias here.”
The run and these lines are the worked example on Boosting, as it printed: 100 training sets of 200 rows each, noise variance 1.00
Which one?
Where is the error?
- Variance, the model overfits Bagging · Random forest
- Bias, the model underfits Boosting · Gradient boosting
Do you need a baseline quickly?
- Yes, with little tuning Random forest
Does the model have to be read?
- Yes, every split followed Decision tree
Each answer is from its page’s “Why & when”.
All 24 traps on these pages
Each is the opening line of a pitfall; its page says why.
Decision tree
- An unconstrained tree will overfit, reliably.
- Small data changes produce different trees.
- Impurity-based feature importances are biased.
- Splits are axis-aligned.
Bagging
- Bagging a stable model and expecting a gain.
- Treating out-of-bag error as free cross-validation in every case.
- Reading a bagged model's feature importances as if they were a tree's.
- Assuming more members can hurt.
- Bootstrapping dependent rows.
Random forest
- Leaving max_features at "all".
- Trusting the built-in feature importances.
- Tuning the number of trees as if it were a regularisation parameter.
- Expecting calibrated probabilities.
- Using it on data with strong extrapolation demands.
Boosting
- Boosting deep trees.
- Assuming more rounds is safe.
- Outliers and label noise.
- Comparing AdaBoost's α to a probability.
- Treating boosting and bagging as interchangeable "ensembles".
Gradient boosting
- Setting the number of rounds without early stopping.
- Tuning η and M independently.
- Deep trees.
- Assuming the fast implementations are drop-in equivalent.
- Reading feature importances off a boosted model as truth.