Squared Loss and the Mean Prediction

Why the Mean Is the Natural Prediction Under Squared Loss

A constant prediction ignores differences between individuals and assigns the same value to every case. It can still be a useful baseline: a more complicated prediction procedure can be judged against a simple rule that always predicts one number. The choice of that number depends on how prediction errors are scored. Under squared loss, the arithmetic mean is the best constant prediction for a fixed set of outcomes.

Here, “best” means that the prediction gives the smallest total squared error on the specified outcomes. It does not mean that the mean is best for every error measure, every prediction task, or every future data set. The result follows from the mathematical form of squared loss.

Definition: For numerical outcomes \(y_1,\ldots,y_n\) and a constant prediction \(c\), the error for outcome \(i\) is \(y_i-c\), and its squared loss is \((y_i-c)^2\). The total squared error is \(\sum_{i=1}^{n}(y_i-c)^2\), and the root mean squared error (RMSE) is \(\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-c)^2}\). The arithmetic mean of the outcomes is \(\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i\).

This result concerns a fixed, finite set of \(n\) numerical outcomes, with every outcome contributing equally. The prediction \(c\) is restricted to be the same for every outcome, and all outcomes must be measured on a scale where subtraction and squaring are meaningful. No probability model is needed to establish which \(c\) minimizes the observed total squared error.

The Mean Minimizes Total Squared Error

The key is to compare any candidate constant \(c\) with the mean \(\bar{y}\). Write each deviation from \(c\) as the sum of a deviation from the mean and the difference between the mean and \(c\):

y_i-c=(y_i-\bar{y})+(\bar{y}-c).

Square this expression and sum over all observations. The cross-product disappears because deviations from the mean sum to zero.

\begin{aligned} \sum_{i=1}^{n}(y_i-c)^2 &=\sum_{i=1}^{n}\left[(y_i-\bar{y})+(\bar{y}-c)\right]^2\\ &=\sum_{i=1}^{n}(y_i-\bar{y})^2 +2(\bar{y}-c)\sum_{i=1}^{n}(y_i-\bar{y}) +n(\bar{y}-c)^2\\ &=\sum_{i=1}^{n}(y_i-\bar{y})^2+n(\bar{y}-c)^2. \end{aligned}

To justify the cancellation, use the definition of the mean:

\sum_{i=1}^{n}(y_i-\bar{y}) =\sum_{i=1}^{n}y_i-n\bar{y} =\sum_{i=1}^{n}y_i-n\left(\frac{1}{n}\sum_{i=1}^{n}y_i\right) =0.

Theorem: For any finite set of numerical outcomes \(y_1,\ldots,y_n\) and any constant prediction \(c\), \[ \sum_{i=1}^{n}(y_i-c)^2 =\sum_{i=1}^{n}(y_i-\bar{y})^2+n(c-\bar{y})^2. \] Since \(n(c-\bar{y})^2\) is nonnegative and equals zero exactly when \(c=\bar{y}\), the total squared error is minimized at \(c=\bar{y}\). If RMSE is used instead, it has the same minimizing value of \(c\), because dividing by the fixed positive number \(n\) and taking a square root do not change which total is smallest.

The identity gives more than the location of the minimum. It says exactly how much extra total squared error results from using \(c\) instead of the mean: the increase is \(n(c-\bar{y})^2\). A prediction farther from the mean incurs a larger penalty, and that penalty grows with the number of outcomes. The minimum total squared error itself is the sum of squared deviations from the mean.

Worked Example: Comparing Constant Predictions

Suppose a hypothetical set of three daily outcomes, measured in units, is \(2, 4,\) and \(9\). Find the constant prediction that minimizes total squared error, then compare it with predictions of 4 and 6.

The mean is \[ \bar{y}=\frac{2+4+9}{3}=\frac{15}{3}=5. \] The total squared error at \(c=5\) is \[ (2-5)^2+(4-5)^2+(9-5)^2=9+1+16=26. \] Therefore the RMSE is \[ \sqrt{\frac{26}{3}}=\sqrt{8.667}\approx 2.944\text{ units}. \]

At \(c=4\), the total squared error is \[ (2-4)^2+(4-4)^2+(9-4)^2=4+0+25=29, \] so the RMSE is \(\sqrt{29/3}\approx 3.109\) units. At \(c=6\), the total squared error is \[ (2-6)^2+(4-6)^2+(9-6)^2=16+4+9=29, \] giving the same RMSE, about \(3.109\) units.

The decomposition predicts this difference. Either alternative is one unit from the mean, so its total squared error exceeds the minimum by \(n(c-\bar{y})^2=3(1)^2=3\); indeed, \(29-26=3\). Of these three constants, predicting 5 has the smallest total squared error and RMSE.

What the Result Means for RMSE

RMSE first averages the squared errors and then takes a square root. For a fixed set of \(n\) outcomes, both operations preserve the ordering of candidate total squared errors. If one constant has a smaller total squared error than another, it also has a smaller mean squared error and a smaller RMSE. Thus, when the choices are all constant predictions evaluated on the same outcomes, the RMSE-minimizing choice is their arithmetic mean.

The units help distinguish the two summaries. A squared error is measured in squared outcome units, and total squared error is also in squared outcome units. RMSE is measured in the original outcome units, which makes its size easier to compare with typical prediction errors. The change in units does not change the minimizing prediction.

Worked Example: Finding the Minimum RMSE for Outcomes in Hours

A hypothetical set of four task durations is \(10, 12, 12,\) and \(18\) hours. Compare the mean prediction with the constant prediction of 12 hours.

The mean duration is \[ \bar{y}=\frac{10+12+12+18}{4}=\frac{52}{4}=13\text{ hours}. \] Using \(c=13\), the squared errors are \(9, 1, 1,\) and \(25\) hours squared. The total squared error is \(9+1+1+25=36\) hours squared, so \[ \mathrm{RMSE}(13)=\sqrt{\frac{36}{4}}=\sqrt{9}=3.000\text{ hours}. \]

Using \(c=12\), the squared errors are \(4, 0, 0,\) and \(36\) hours squared. Thus, \[ \mathrm{RMSE}(12)=\sqrt{\frac{4+0+0+36}{4}} =\sqrt{10}\approx 3.162\text{ hours}. \]

The mean prediction has the smaller RMSE: 3.000 hours rather than about 3.162 hours. Its total squared error is also smaller, 36 rather than 40 hours squared. The identity accounts for the increase: \(n(c-\bar{y})^2=4(12-13)^2=4\), and \(40-36=4\).

Training Outcomes and Evaluation Outcomes

The minimizing result applies to the particular outcomes being considered. This matters when a constant prediction is used as a baseline in prediction assessment. If the baseline is built from training data, calculate the training mean and use it to make predictions for new cases. Then calculate RMSE using those cases’ outcomes only to assess the predictions.

A mean calculated from the evaluation outcomes is the constant that would minimize RMSE on those outcomes, but it is not a fair prediction for those same cases if their outcomes were not available when the predictions were made. It uses the answers to construct the predictions being evaluated. The mathematical minimum is valid; the mistake would be treating that hindsight calculation as an honest out-of-sample prediction.

Worked Example: A Training Mean as a Fixed Baseline

Suppose a hypothetical training set has outcomes \(5, 7, 8,\) and \(12\) units. A separate evaluation set has outcomes \(6, 9,\) and \(15\) units. Use the training mean as a constant prediction for the evaluation cases, and compare it with a constant of 9 units.

The training mean is \[ \bar{y}_{\text{train}}=\frac{5+7+8+12}{4}=\frac{32}{4}=8\text{ units}. \] The resulting evaluation errors are \(6-8=-2\), \(9-8=1\), and \(15-8=7\) units. Therefore, \[ \mathrm{RMSE}_{\text{evaluation}}(8) =\sqrt{\frac{(-2)^2+1^2+7^2}{3}} =\sqrt{\frac{54}{3}} =\sqrt{18}\approx 4.243\text{ units}. \]

For \(c=9\), the evaluation errors are \(-3, 0,\) and \(6\) units, so \[ \mathrm{RMSE}_{\text{evaluation}}(9) =\sqrt{\frac{(-3)^2+0^2+6^2}{3}} =\sqrt{\frac{45}{3}} =\sqrt{15}\approx 3.873\text{ units}. \] On this evaluation set, 9 has a lower RMSE than the training-mean baseline of 8. That comparison does not change how the baseline was formed: 8 was calculated from the training outcomes, before using the evaluation outcomes to assess its performance.

For comparison, the evaluation mean is \((6+9+15)/3=10\) units. It would give an evaluation RMSE of \[ \sqrt{\frac{(6-10)^2+(9-10)^2+(15-10)^2}{3}} =\sqrt{\frac{42}{3}}=\sqrt{14}\approx 3.742\text{ units}. \] This is the minimum possible RMSE among constants for these three evaluation outcomes. But if the outcomes \(6,9,15\) were held back for evaluation, using their mean of 10 as the predictions would use held-back information. It is a mathematical benchmark for this set, not a prediction that could have been produced in advance without that information.

Common Mistakes and Why the Choice Matters

A common mistake is to say, “The mean minimizes prediction error,” without naming the error measure or the restriction on predictions. The result is specifically about minimizing the sum of squared errors, or equivalently RMSE, when the prediction is constrained to be one constant and the same equally weighted outcomes are used. With a different loss function, the minimizing constant can differ; the earlier material on absolute loss and the median treats that other criterion.

Another mistake is to choose a prediction because it is closest to most outcomes, or because positive and negative errors appear to balance. Squared loss depends on the magnitude of each error after squaring. A large miss can contribute much more than several small misses, so intuition based only on the number of outcomes on either side of a prediction can be misleading.

The mean is useful as a baseline because its optimality is precise and easy to verify. A more complex model may improve RMSE, but a comparison is meaningful only when the predictions are assessed on the same cases and outcomes, and when the predictions were formed without using those evaluation outcomes. The mean-minimization result tells you which constant is optimal for a chosen set; it does not by itself guarantee good performance on new observations.

Do not confuse the prediction restriction. The theorem concerns one constant prediction for every case, not a separate freely chosen prediction for each observed outcome.
Do not confuse the metric. The mean minimizes squared loss and RMSE for a fixed set; that does not establish that it minimizes every possible error measure.
Do not confuse fitting with evaluation. The mean of evaluation outcomes describes the best constant after those outcomes are known, but it cannot serve as an honest prediction made before they were observed.

Key takeaway: For numerical outcomes with equal contributions, the arithmetic mean uniquely minimizes total squared error among constant predictions, unless all candidate constants are restricted to the same value. Since RMSE is a monotone transformation of total squared error on a fixed set, it is minimized by that same mean.

Check Your Understanding

Use the definition of squared loss and the mean-minimization identity to answer the following.

For outcomes \(3, 6,\) and \(9\), calculate the mean and the total squared error when predicting that mean.
Why does taking the square root of mean squared error not change which constant minimizes the criterion?
For outcomes \(4, 4,\) and \(10\), compare the total squared error at the mean with the total squared error at \(c=5\).
A training set has a mean outcome of 20 units. Explain how that value can be used as a constant baseline for a separate evaluation set.
Why is the mean of the evaluation outcomes not an honest prediction for those same cases when their outcomes were held back?