Initializing A Squared-Error Boosting Model

Why a Boosting Model Needs a Starting Value

A boosting model is built in stages, so it needs a prediction before it has added any trees. For squared-error boosting, the first prediction is a single constant applied to every training observation. Choosing that constant well matters: it determines the baseline from which later stages will work.

The initialization is an optimization problem, not a guess based on a typical observation. With an ordinary squared-error objective and equally weighted training cases, the optimal constant is the arithmetic mean of the responses. The reason follows directly from minimizing the sum of squared differences.

Definition: Suppose the training data consist of $n$ response values $y_1,\ldots,y_n$, with $n\geq 1$. A constant initialization predicts the same value $c$ for every observation, regardless of its predictors. Under equal-weight squared error, its training objective is $L(c)=\sum_{i=1}^{n}(y_i-c)^2$. The initial prediction is the value of $c$ that minimizes this objective.

At this stage, the predictors do not distinguish observations: the model has not added any trees. Their values therefore do not change the optimization over $c$. The responses and the specified loss determine the starting value.

Deriving the Squared-Error Initialization

Assume the response values are finite, all observations have equal weight, and $c$ can be any real number. Different software or presentations may use the sum of squared errors, the mean squared error, or a fixed positive multiple of either. These choices have the same minimizer because multiplying an objective by a positive constant does not change which $c$ makes it smallest.

Write the objective.
For a proposed constant $c$, the error on observation $i$ is $y_i-c$. The total squared-error objective is $L(c)=\sum_{i=1}^{n}(y_i-c)^2$.

Differentiate with respect to the constant.
Each term has derivative $2(c-y_i)$, so $L'(c)=2\sum_{i=1}^{n}(c-y_i)=2\left(nc-\sum_{i=1}^{n}y_i\right)$.

Set the derivative equal to zero.
The stationary point satisfies $nc=\sum_{i=1}^{n}y_i$, and therefore $c=\frac{1}{n}\sum_{i=1}^{n}y_i=\bar{y}$.

Check that this point is a minimum.
The second derivative is $L''(c)=2n$, which is positive because $n\geq 1$. Thus the stationary point is the unique global minimum.

c_0=\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i

This calculation explains why the mean appears: moving $c$ changes every squared error, and the optimum balances their signed differences so that $\sum_{i=1}^{n}(y_i-c_0)=0$. Equivalently, the sum of the initial residuals is zero. This is a property of the minimizing constant, not a claim that each individual error is zero.

If the objective is written as $\frac{1}{n}\sum_{i=1}^{n}(y_i-c)^2$, its derivative is the derivative above divided by $n$, so the minimizing value remains $\bar{y}$. The same is true for $\frac{1}{2n}\sum_{i=1}^{n}(y_i-c)^2$. The numerical value of the reported loss changes with the convention; the starting prediction does not.

Worked Example: Finding a Baseline for Hypothetical Claim Amounts

A hypothetical model is to predict claim amounts, in thousands of dollars, from training data with responses $4, 7, 9,$ and $10$. Before adding any trees, find the equal-weight squared-error initialization and its training sum of squared errors. Use the unscaled sum $\sum_i(y_i-c)^2$.

Method and assumptions. Each of the four observations has equal weight, the objective is squared error, and the constant is unrestricted. No probability distribution, interest rate, or mortality assumption is needed: this is a calculation on the stated training responses.

Step 1: Calculate the response mean. The response sum is $4+7+9+10=30$, and the sample size is $4$. Therefore,

c_0=\bar{y}=\frac{30}{4}=7.5

Step 2: Calculate each squared error. At $c_0=7.5$, the four errors are $4-7.5=-3.5$, $7-7.5=-0.5$, $9-7.5=1.5$, and $10-7.5=2.5$. Their squares are $12.25$, $0.25$, $2.25$, and $6.25$.

Step 3: Add and check the objective. The sum is $12.25+0.25+2.25+6.25=21$. The errors sum to $-3.5-0.5+1.5+2.5=0$, as expected at the minimizing constant. The mean squared error, if that convention is used, is $21/4=5.25$. The initialization is $7.5$ thousand dollars, or $\$7{,}500$; the objective check confirms that its initial residuals are centered.

Common mistake avoided. Do not choose one observed response, such as $9$, simply because it appears representative. The objective is minimized by the mean, and the zero-sum error check supports the calculation.

Weighted Squared Error

Some training objectives assign different nonnegative weights to observations. If each weight $w_i$ is positive and the objective is $\sum_{i=1}^{n}w_i(y_i-c)^2$, differentiation gives $2\sum_i w_i(c-y_i)=0$. The minimizing constant is the weighted mean. If all weights are equal, their common value cancels and this reduces to the ordinary mean.

c_0=\frac{\sum_{i=1}^{n}w_i y_i}{\sum_{i=1}^{n}w_i}, \qquad \text{when } w_i>0 \text{ for every } i

Weights in this formula describe the objective’s relative contribution from each observation. They are not response values and should not be applied after calculating an unweighted mean. If some weights are zero, those observations have no effect; the same formula applies provided the total weight is positive. If the total weight is zero, the objective does not identify a unique starting constant.

Worked Example: Initializing with Unequal Observation Weights

A hypothetical severity model uses three training responses, in thousands of dollars: $2, 5,$ and $8$. Their respective objective weights are $1, 2,$ and $3$. Find the starting constant and the weighted sum of squared errors. The model uses $\sum_i w_i(y_i-c)^2$, not the unweighted objective.

Method and assumptions. Use the weighted mean because the stated objective weights the squared error for each observation. The weights are positive, their sum is nonzero, and the constant is unrestricted. No distributional assumption is being made.

Step 1: Multiply each response by its weight. The weighted response contributions are $1(2)=2$, $2(5)=10$, and $3(8)=24$. Their sum is $2+10+24=36$. The total weight is $1+2+3=6$.

Step 2: Find the weighted mean.

c_0=\frac{36}{6}=6

Step 3: Evaluate the stated weighted objective. The response errors at $6$ are $2-6=-4$, $5-6=-1$, and $8-6=2$; the squared errors are $16,1,$ and $4$. Applying the weights gives $1(16)=16$, $2(1)=2$, and $3(4)=12$. Thus the weighted sum is $16+2+12=30$. If the weighted mean squared error divides by total weight, it is $30/6=5$.

Check and common mistake avoided. The weighted errors sum to $1(-4)+2(-1)+3(2)=-4-2+6=0$. The unweighted mean would be $(2+5+8)/3=5$, but it would not minimize the weighted objective. Using $5$ would give weighted squared error $1(9)+2(0)+3(9)=36$, which is greater than $30$.

Why the Median Is Not the Squared-Error Starting Value

The choice of constant depends on the loss. Squared error penalizes a difference $e$ by $e^2$, while absolute error penalizes it by $|e|$. The mean minimizes the equal-weight sum of squared errors; a median minimizes the equal-weight sum of absolute errors. Using a median for a squared-error initialization silently changes the optimization problem.

Worked Example: Comparing the Mean and Median Under Squared Error

A hypothetical portfolio segment has five training response values, in thousands of dollars: $1, 2, 2, 5,$ and $10$. Find the squared-error starting constant, calculate its sum of squared errors, and compare it with the objective at the sample median.

Method and assumptions. There are five equally weighted responses and the objective is the unrestricted sum of squared errors. The median is calculated only as a comparison; it is not assumed to be the appropriate initializer for this loss.

Step 1: Calculate the mean. The sum is $1+2+2+5+10=20$, so $c_0=20/5=4$.

Step 2: Calculate the objective at the mean. The errors are $1-4=-3$, $2-4=-2$, $2-4=-2$, $5-4=1$, and $10-4=6$. The squared errors are $9,4,4,1,$ and $36$. Their sum is $9+4+4+1+36=54$.

Step 3: Compare the median. The ordered responses are already $1,2,2,5,10$, so the median is $2$. At $c=2$, the squared errors are $1,0,0,9,$ and $64$, with sum $74$. Since $54<74$, the mean gives the smaller squared-error objective. The mean squared error at the initializer is $54/5=10.8$.

Check and common mistake avoided. At $c=4$, the errors sum to $-3-2-2+1+6=0$, confirming the derivative condition. The large response $10$ pulls the mean upward, as squared error gives substantial influence to a large deviation. Choosing the median because it seems more typical would use the wrong loss criterion.

Conditions, Constraints, and Practical Checks

The mean result requires the stated objective and an unrestricted real-valued constant. If a model requires $c$ to lie in a specified interval, the unconstrained mean may not be allowed. For squared error over an interval $[a,b]$, the constrained minimizer is the mean if it lies in the interval; otherwise it is the nearest endpoint. This follows because the squared-error objective decreases up to its unique unconstrained minimum and increases after it.

Keep the following distinctions clear when interpreting an initialization:

Training responses determine the baseline. For an equal-weight constant model, calculate the mean of the training responses, not the mean of a separate test set.
The loss determines the appropriate summary. The mean is the answer for squared error; a different loss can imply a different optimal constant.
The objective convention determines the reported loss value. A sum, an average, or a fixed positive multiple has a different numerical loss, but the same minimizer.
Weights change the calculation. When observations have unequal positive weights in the objective, use the weighted mean rather than the ordinary mean.

A useful numerical check is to verify that the initial residuals sum to zero in the equal-weight case, or that their weighted sum is zero in the weighted case. This check can reveal an arithmetic error, but it does not replace checking that the objective and weights were read correctly.

Key takeaway: Before any trees are added, an unrestricted constant that minimizes equal-weight squared training error is the response mean. With positive observation weights, it is the weighted response mean. The result comes from minimizing the stated objective, so confirm the loss, weights, and allowed values of the constant before using it.

Check Your Understanding

For each problem, identify the objective carefully before calculating the starting constant.

A constant is fitted under equal-weight squared error to responses $6, 8,$ and $11$. What is the initialization? (Answer: $25/3$, or approximately $8.3333$.)
Four equal-weight responses are $2, 4, 4,$ and $10$. Find the squared-error initialization and its mean squared error. (Answer: Initialization $5$; mean squared error $9$.)
Responses $4, 7,$ and $10$ have respective weights $2, 1,$ and $3$. Find the minimizing constant under weighted squared error. (Answer: $7.5$.)
An objective is $\frac{1}{2n}\sum_i(y_i-c)^2$ for equal-weight responses $0, 3,$ and $9$. Find the initialization and the objective value at that constant. (Answer: Initialization $4$; objective value $7$.)
A constant must satisfy $c\geq 5$, and the equal-weight responses are $1, 4,$ and $6$. Find the constrained squared-error initialization. (Answer: $5$.)

What You'll Learn