Weak Learners And Decision Stumps

Why Use a Weak Learner?

A single decision tree can represent complicated patterns, but a deep tree may be unstable and difficult to interpret. Boosting takes a different approach: it builds a model in stages by combining learners that are deliberately simple. A learner that is only modestly informative on its own can still be useful if it contributes something not already captured by the current ensemble.

“Weak” describes a learner’s predictive ability relative to a relevant benchmark, not merely its size. For binary classification, a common benchmark is chance-level classification under a balanced, unweighted setting; a learner is informative if its error is below that benchmark. For regression, usefulness is assessed relative to a baseline prediction and a specified loss. The precise benchmark therefore depends on the response, loss, and data distribution. A shallow model is not automatically weak, and a weak learner need not be a stump.

Definition: A weak learner is a restricted or simple prediction rule that, under the learning setup being used, can provide predictive information beyond a relevant baseline. In boosting, weak learners are fitted sequentially and combined into an additive model. A decision stump is a decision tree with just one split: it divides the predictor space into two terminal regions and assigns a prediction to each region.

A stump has limited structure. For a numerical predictor, it selects a threshold and sends observations below or at that threshold to one leaf and observations above it to the other. For a categorical predictor, it can separate a selected category or group of categories from the rest. A stump that splits on one predictor cannot express a complicated interaction by itself. Across multiple stages, however, different stumps can make different partitions and contribute different adjustments.

Regression Stumps and Split Evaluation

For a regression stump using a numerical predictor \(x\), a candidate threshold \(c\) creates two regions, \(R_L=\{x:x\le c\}\) and \(R_R=\{x:x>c\}\). The stump predicts a constant within each region. If the criterion is squared error, the best constant in a region is the mean response of the observations assigned to that region. The candidate split is evaluated by adding the squared deviations from the two leaf means.

\operatorname{SSE}(c) = \sum_{i:x_i\le c}(y_i-\bar y_L)^2 + \sum_{i:x_i>c}(y_i-\bar y_R)^2

Here, \(\bar y_L\) and \(\bar y_R\) are the response means in the left and right leaves. A candidate that leaves an empty region is not a valid split. If minimum leaf-size requirements are imposed, a candidate must also satisfy those requirements. When the eligible split points are between ordered observed predictor values, only those points need to be checked: moving a threshold within a gap does not change which observations fall into either leaf.

Why the leaf prediction is the mean: For responses \(y_1,\ldots,y_m\) in one leaf, the squared-error sum for a proposed constant \(a\) is \(Q(a)=\sum_{j=1}^m(y_j-a)^2\). Differentiating gives \(Q'(a)=-2\sum_{j=1}^m(y_j-a)\), so \(Q'(a)=0\) when \(a=\bar y\). Since \(Q''(a)=2m>0\), this is the unique minimum. Thus, for each candidate partition, using each leaf’s sample mean gives the smallest possible within-leaf squared error.

Worked Example: Finding the Best Regression-Stump Split

A hypothetical portfolio analyst uses a single numerical rating \(x\) to predict an outcome \(y\). The four observations, ordered by \(x\), are \((1,1),(2,2),(3,6),(4,7)\). Consider splits after \(x=1\), \(x=2\), and \(x=3\). Find the mean prediction in each leaf, calculate the total SSE for each split, and select the best split. Use squared error and no additional leaf-size constraint.

Method. For each candidate, calculate the mean response on each side, then sum the squared deviations from those means. The responses total \(1+2+6+7=16\), so the overall mean is \(16/4=4\), which is also the prediction from a stump with no split.

Split after \(x=1\). The left leaf contains only \(y=1\), so its mean is \(1\), and its SSE is \((1-1)^2=0\). The right leaf contains \(2,6,7\), with mean \((2+6+7)/3=15/3=5\). Its SSE is \((2-5)^2+(6-5)^2+(7-5)^2=9+1+4=14\). The total is \(0+14=14\).

Split after \(x=2\). The left leaf contains \(1,2\), with mean \((1+2)/2=1.5\). Its SSE is \((1-1.5)^2+(2-1.5)^2=0.25+0.25=0.5\). The right leaf contains \(6,7\), with mean \((6+7)/2=6.5\). Its SSE is \((6-6.5)^2+(7-6.5)^2=0.25+0.25=0.5\). The total is \(0.5+0.5=1\).

Split after \(x=3\). The left leaf contains \(1,2,6\), with mean \((1+2+6)/3=9/3=3\). Its SSE is \((1-3)^2+(2-3)^2+(6-3)^2=4+1+9=14\). The right leaf contains only \(7\), with mean \(7\) and SSE \(0\). The total is \(14+0=14\).

The three total SSEs are \(14,1,14\), respectively. The split after \(x=2\) is best among the candidates because it has the smallest SSE, \(1\). Its fitted predictions are \(1.5\) for \(x=1,2\) and \(6.5\) for \(x=3,4\). Common mistake avoided: the first and third totals are \(14\), not \(13\); each squared deviation in the three-observation leaf must be included.

Classification Stumps

For classification, a stump still forms two regions, but its leaf output is a class label or a set of class probabilities rather than a regression mean. One possible split criterion is misclassification error: assign each leaf its majority class and count the observations not in that class. Another is Gini impurity. For a leaf with class proportions \(p_1,\ldots,p_K\), its Gini impurity is \(1-\sum_{k=1}^K p_k^2\); the split score is the sum of leaf impurities weighted by their leaf sizes relative to the full sample. The chosen criterion affects which split is preferred. The leaf prediction and split criterion should not be confused: one describes what the stump predicts, and the other describes how candidate partitions are compared.

Worked Example: Comparing Classification-Stump Splits

A hypothetical claims team labels eight cases as either “review” (R) or “routine” (T). In increasing order of a numerical flag score \(x\), the labels are \(R,T,R,T,T,T,R,R\). Compare a split after the second observation with a split after the fourth observation using the weighted Gini criterion. Predict the majority class in each leaf. Assume ties, if any, are resolved in favor of R.

Method. For each split, find the class proportions in both leaves, compute \(1-\sum_k p_k^2\) in each, and weight each impurity by the leaf’s share of the eight observations.

Split after observation 2. The left leaf has \(R=1,T=1\), so its proportions are \(1/2,1/2\), its Gini impurity is \(1-(1/2)^2-(1/2)^2=1/2\), and its predicted class is R under the stated tie rule. The right leaf has \(R=3,T=3\), so its impurity is also \(1/2\), and it predicts R. The weighted score is \((2/8)(1/2)+(6/8)(1/2)=1/8+3/8=1/2\).

Split after observation 4. The left leaf has \(R=2,T=2\), so its impurity is \(1-(2/4)^2-(2/4)^2=1/2\), and it predicts R by the tie rule. The right leaf has \(R=2,T=2\), so its impurity is \(1/2\) and it predicts R. The weighted score is \((4/8)(1/2)+(4/8)(1/2)=1/2\).

The two candidate splits tie at weighted Gini score \(0.5\); the criterion alone does not choose between them. Both stumps predict R in both leaves under the specified tie rule, so neither split changes the hard class prediction on these observations. Common mistake avoided: weighting is essential when leaf sizes differ; simply adding the two leaf impurities would not give the stated split score.

This example also illustrates a practical point: a split can divide predictor values without improving hard classifications. A split may still be useful under a different criterion or when leaves output class probabilities, but its value must be assessed with the objective actually being optimized.

Why Simple Learners Can Work in Sequence

A stump alone has only two regions, so its predictions are coarse. A sequence of stumps can use different predictors or different thresholds at successive stages. In an additive model, the ensemble prediction is formed from contributions by multiple learners, with coefficients or step sizes determined by the fitting procedure. The resulting model can vary across several regions even though each individual learner is simple.

The key is complementarity. If a new learner captures a pattern left unexplained by the current model, adding it may reduce the chosen loss. Repeating this process can build a useful fit from modest components. This does not guarantee improvement on new data: too many stages can fit noise, and the learner’s usefulness depends on the loss, fitting procedure, and data. Complexity control and validation remain important.

Worked Example: Combining Two Regression Stumps

Suppose a sequential additive model for a hypothetical severity measure has the current prediction \(F_0(x)=10\) for every case. A first stump contributes \(h_1(x)=2\) when \(x\le 5\) and \(h_1(x)=-1\) when \(x>5\). A later stump contributes \(h_2(x)=-1\) when \(z\le 3\) and \(h_2(x)=2\) when \(z>3\). For this illustration, both contribution coefficients are 1, so \(F(x,z)=F_0(x)+h_1(x)+h_2(x)\). Calculate the final predictions for cases A: \((x,z)=(4,2)\), B: \((6,2)\), and C: \((4,5)\). Explain what each stump adds.

Method. Determine the region for each stump separately, read off each contribution, and add the baseline and both contributions.

For A, \(x=4\le5\), so \(h_1=2\). Also \(z=2\le3\), so \(h_2=-1\). Thus \(F=10+2-1=11\).

For B, \(x=6>5\), so \(h_1=-1\), while \(z=2\le3\), so \(h_2=-1\). Thus \(F=10-1-1=8\).

For C, \(x=4\le5\), so \(h_1=2\), and \(z=5>3\), so \(h_2=2\). Thus \(F=10+2+2=14\).

The first stump distinguishes cases using \(x\); the second contributes a separate adjustment based on \(z\). Together, their predictions depend on the combination of regions defined by both splits. Common mistake avoided: the second stump is an additive contribution, not a replacement for the first stump or a fresh baseline prediction.

Interpretation and Exam Traps

A stump is useful for understanding how a weak learner creates a simple partition, but several conclusions do not follow automatically from the word “weak.” A weak learner is not necessarily random, and it is not necessarily a one-split tree. Nor does a good training split prove that an ensemble will generalize. The distinction between a learner’s complexity and its predictive strength is important: a simple rule may perform well on a particular problem, while a more flexible rule may still perform poorly.

Do not assume every stump is better than chance. In classification, the relevant baseline depends on class frequencies, the loss, and any observation weights. A raw error of less than one-half is not a universal criterion.
Do not compare unweighted leaf impurity sums when leaf sizes differ. Use the criterion’s prescribed weighting.
Do not omit a leaf’s squared deviations. For regression, compute the mean and every squared residual within each region before totaling SSE.
Do not mistake a split for an interaction model. One stump uses one partition; combining multiple stumps can create richer regional behavior, but the form of that behavior depends on which predictors and splits are used.
Do not infer out-of-sample performance from training fit alone. Sequential additions can improve training loss while eventually fitting noise; validation is needed to assess generalization.

Key takeaway: A decision stump is a one-split learner that assigns a simple prediction to each of two regions. Its simplicity can be an advantage in sequential additive learning: successive stumps can contribute distinct adjustments, while the performance of the resulting ensemble still depends on the loss, fitting procedure, and control of overfitting.

Check Your Understanding

Work each problem by identifying the partition, applying the stated criterion, and checking that all observations are included.

A regression stump splits six ordered observations into the responses \(2,4\) on the left and \(7,9,10,10\) on the right. Find each leaf mean and the total SSE. (Answer: left mean \(3\), right mean \(9\), total SSE \(2+6=8\).)
For ordered predictor values \(1,2,3,4\), the responses are \(3,3,8,10\). Calculate the SSE for splits after \(x=1\), \(x=2\), and \(x=3\), then identify the best split. (Answer: \(26\), \(2\), and \(14\), respectively; split after \(x=2\).)
A classification leaf contains 5 cases of class A and 3 of class B. Calculate its Gini impurity and state its majority-class prediction. (Answer: \(1-(5/8)^2-(3/8)^2=30/64=0.46875\); predict A.)
A candidate classification split has a left leaf of 3 cases with labels A, A, B and a right leaf of 5 cases with labels A, B, B, B, B. Calculate the weighted Gini score. (Answer: left impurity \(4/9\), right impurity \(8/25\), weighted score \((3/8)(4/9)+(5/8)(8/25)=1/6+1/5=11/30\approx0.3667\).)
An additive model has baseline \(F_0=6\). One stump adds \(+2\) if \(u\le4\), otherwise \(-1\). A second adds \(-2\) if \(v\le0\), otherwise \(+1\). With both coefficients equal to 1, find the prediction at \((u,v)=(3,2)\) and explain which regions were used. (Answer: \(6+2+1=9\); the case is in the \(u\le4\) region of the first stump and the \(v>0\) region of the second.)

What You'll Learn