difficulty 3/5
Feature importance
Ranking inputs by how much a fitted model relies on them — where the default measure for tree ensembles is biased toward high-cardinality features and the honest one is measured on held-out data.
Intuition
A random forest is hundreds of trees deep and nobody can read it. So every implementation ships a consolation prize: a bar chart of which features mattered. It is the most-screenshotted output in applied machine learning, and the default version of it is measurably wrong in a specific, predictable direction.
The problem is what "mattered" is measured against. The built-in measure counts how much each feature reduced impurity across all the splits that used it. A feature that takes many distinct values gets many chances to produce a split that looks good on the training data — including when it is pure noise. It accumulates credit for splits that will not replicate [1, §5.2.2].
The alternative asks a question that cannot be gamed this way: shuffle this column and see how much worse the model gets on data it has never seen. If the answer is "not at all", the model was not relying on it, whatever the impurity accounting says.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- scikit-learn User Guide — Permutation feature importancedocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/permutation_importance.html
- A Random Forest Guided Tour (2015)paperhttps://ar5iv.labs.arxiv.org/html/1511.05741
- Understanding Random Forests: From Theory to Practice (2014)paperhttps://arxiv.org/abs/1407.7502