Last updated: 2026-10-07
Decision Trees and Ensembles
Every model so far in this section fits one global equation, or one distance measure, or one grid — some single structure applied the same way to every input. A decision tree does something categorically different: it asks a sequence of yes/no questions about the input's features, each one splitting the training examples into purer and purer subsets, until a final answer can be read off directly.
Choosing a Split by Impurity FoundationalKnowledge that endures for decades — core principles
At each node, every candidate feature and threshold is tested, and whichever split reduces impurity — how mixed the classes are within a node — the most gets chosen. Run against this section's shared dataset: splitting on "feature 1 ≥ 3" alone already separates every "round" point (all below 3) from every "square" point (all at or above 4) perfectly, in one step, because the two clusters don't overlap on that axis at all. Real data is rarely this clean, which is why a tree normally needs several splits, each handling whatever impurity the previous ones left behind, rather than stopping after one.
Why a Lone Tree Overfits FoundationalKnowledge that endures for decades — core principles
Grown without any limit, a tree keeps splitting until every leaf holds examples of only one class — which, on real data, usually means it has carved out an ever-smaller, ever-more-specific region of feature space to isolate individual noisy or mislabelled points, rather than finding the general rule those points were supposed to illustrate. An unconstrained single tree is one of the easiest models in this entire section to overfit, and it is correspondingly rare to deploy one alone.
Ensembles: Combining Many Weak Trees FoundationalKnowledge that endures for decades — core principles
An ensemble trains many individually imperfect trees and combines their predictions, and the two standard ways of doing this differ in how the trees relate to each other. Bagging trains many trees independently, in parallel, each on a different random resample of the training data, and averages their votes — a random forest adds a further twist, letting each split consider only a random subset of the available features, which decorrelates the trees from each other and makes the averaging actually effective rather than averaging over near-identical trees. Boosting instead trains trees sequentially, each one specifically targeted at correcting whatever the ensemble so far gets wrong — upweighting misclassified examples (AdaBoost) or fitting directly to the residual error (gradient boosting).
| Bagging | Boosting | |
|---|---|---|
| Trees trained | Independently, in parallel | Sequentially, each depending on the last |
| Primarily reduces | Variance (overfitting to noise) | Bias (underfitting) |
| Risk | Diminishing returns past a certain forest size | Can overfit if run for too many rounds |
Related Topics
- Supervised Learning: Regression, Trees, and Ensembles — the full Gini/entropy impurity formulas, worked code, and precision/recall evaluation vocabulary, covered in depth.
- What Is a Learning Model? — this page's "a growing set of branch tests" row, contrasted against every other model family's very different stored state.
- Support Vector Machines — another classifier whose boundary is a handful of discrete decisions rather than a single global equation, built on a completely different principle (margin, not impurity).