Explained Variance and Error for Quantitative Traits

Why Quantitative-Trait Prediction Needs More Than One Metric

A prediction score can rank people well while giving trait values that are systematically too high, too low, or too spread out. Conversely, predictions can be close to the overall average yet fail to distinguish individuals. For a quantitative trait, evaluation should therefore describe both the magnitude of prediction errors and how predictions relate to observed values.

This tutorial considers a held-out validation set of $n$ individuals. For individual $i$, let $y_i$ be the observed trait value and let $\hat{y}_i$ be the prediction produced without using that individual’s validation outcome to fit or tune the prediction procedure. The values must be on the same trait scale and in the same units. A prediction might, for example, be a predicted height in centimetres or a predicted concentration in milligrams per litre.

The validation set should represent the population to which performance is intended to apply. Its individuals should be independent for the simple summaries developed here, or the sampling design and dependence must otherwise be accounted for. Predictions and any comparator predictions should be fixed without using validation outcomes. These conditions make the computed metrics evaluations of predictions on new data, rather than summaries of fit to the data used to construct them.

Definition: The mean squared error (MSE) of predictions on a validation set is the average squared difference between observed and predicted trait values: $$ \mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2. $$ MSE is nonnegative and is expressed in squared trait units. A value of zero means every prediction exactly matches its observed value. Smaller MSE indicates smaller average squared errors on the evaluated set.

Squaring makes larger errors contribute more heavily than smaller errors. MSE also combines several features of prediction performance, including systematic offsets and variation in individual errors; it does not by itself reveal which feature is responsible. Its units are squared, so an MSE of 9 centimetres squared is not an average error of 9 centimetres. The square root of MSE returns to the trait’s original units, but the metric considered here is MSE itself.

Out-of-Sample R-Squared Requires a Baseline

The phrase “out-of-sample R-squared” is used for a comparison between prediction error and a reference prediction. The reference must be specified to interpret the result. One common choice is the validation-set mean, $\bar{y}$, used as the same prediction for every validation individual. This choice describes how much smaller the model’s squared error is than the error from predicting the validation mean.

Definition: When the validation outcomes are not all identical, the R-squared relative to the validation-set mean is $$ R^2_{\mathrm{val\ mean}} =1-\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} {\sum_{i=1}^{n}(y_i-\bar{y})^2}, \qquad \bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i. $$ The denominator is the squared error of the constant prediction $\bar{y}$ on these validation outcomes.

This definition has a direct baseline interpretation. A value of 1 means zero prediction error; 0 means the model’s squared error equals that of predicting the validation mean; a negative value means the model has greater squared error than that mean predictor. A negative value is possible because an out-of-sample prediction procedure is not required to perform better than a mean calculated from the validation outcomes.

The validation mean is a useful reference for describing variation in this particular set, but it is not generally available when predicting a new person: it uses the outcomes being evaluated. A deployable comparator could instead be a constant estimated from training data or predictions from a simpler model. If the comparator’s prediction for individual $i$ is $b_i$, define its squared error as $\sum_i(y_i-b_i)^2$. The relative skill score is then

R^2_{\mathrm{baseline}} =1-\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} {\sum_{i=1}^{n}(y_i-b_i)^2}.

This score compares the model directly with the stated baseline. It equals the validation-mean version only when the baseline predictions are the validation mean. A score can therefore change when the reference changes, even though the model predictions and validation outcomes stay the same. If the baseline has zero error, this ratio is undefined; if all validation outcomes are identical, the validation-mean version is undefined because its denominator is zero.

Worked Example: Calculating Error and R-Squared

Suppose a hypothetical validation set has observed trait values of 2, 4, 6, and 8 units, with predictions of 3, 4, 5, and 8 units. Calculate MSE and R-squared relative to the validation mean.

The prediction errors are $-1, 0, 1,$ and $0$ units. Thus the sum of squared errors is $$ (-1)^2+0^2+1^2+0^2=2\ \text{units}^2, $$ and $$ \mathrm{MSE}=\frac{2}{4}=0.5\ \text{units}^2. $$

The observed mean is $\bar{y}=(2+4+6+8)/4=5$ units. The squared error from predicting 5 for everyone is $$ (2-5)^2+(4-5)^2+(6-5)^2+(8-5)^2=9+1+1+9=20\ \text{units}^2. $$ Therefore, $$ R^2_{\mathrm{val\ mean}}=1-\frac{2}{20}=0.90. $$ On these four validation observations, the predictions have 90% less squared error than the constant prediction equal to their observed mean. This is a comparison with that baseline, not a claim that 90% of an individual’s trait is explained.

Correlation Measures Association, Not Error Against a Baseline

Pearson correlation describes the strength and direction of a linear association between observed and predicted values. It is unchanged by adding a constant to all predictions or multiplying all predictions by a positive constant. This invariance is useful for describing ranking or linear association, but it also means correlation does not penalize an incorrect offset or scale in the same way as squared error.

Definition: Provided both observed and predicted values vary, the sample Pearson correlation between the validation outcomes and predictions is $$ r_{y,\hat{y}}= \frac{\sum_{i=1}^{n}(y_i-\bar{y})(\hat{y}_i-\bar{\hat{y}})} {\sqrt{\left[\sum_{i=1}^{n}(y_i-\bar{y})^2\right] \left[\sum_{i=1}^{n}(\hat{y}_i-\bar{\hat{y}})^2\right]}}, \qquad \bar{\hat{y}}=\frac{1}{n}\sum_{i=1}^{n}\hat{y}_i. $$ The squared correlation $r_{y,\hat{y}}^2$ measures squared linear association. It is not, in general, the out-of-sample R-squared defined by comparison with a baseline.

A short derivation identifies a setting in which the two quantities do coincide. Regress the observed validation outcomes on the predictions using an ordinary least squares line with an intercept, and evaluate the fitted values on those same observations. Let $S_{yy}=\sum_i(y_i-\bar{y})^2$, $S_{\hat{y}\hat{y}}=\sum_i(\hat{y}_i-\bar{\hat{y}})^2$, and $S_{y\hat{y}}=\sum_i(y_i-\bar{y})(\hat{y}_i-\bar{\hat{y}})$. The fitted line’s slope is $S_{y\hat{y}}/S_{\hat{y}\hat{y}}$. Its regression sum of squares is therefore

\mathrm{SSR} =\frac{S_{y\hat{y}}^2}{S_{\hat{y}\hat{y}}}.

With an intercept, the fitted residuals sum to zero and are orthogonal to the fitted values’ deviations from their mean. Thus the total sum of squares decomposes as $S_{yy}=\mathrm{SSR}+\mathrm{SSE}$, so the ordinary in-sample regression R-squared is

\frac{\mathrm{SSR}}{S_{yy}} =\frac{S_{y\hat{y}}^2}{S_{yy}S_{\hat{y}\hat{y}}} =r_{y,\hat{y}}^2.

That equality concerns the fitted values from a regression recalibrated on the same outcomes. It does not establish that the original, fixed predictions have that R-squared against the mean baseline. In particular, recalibrating on validation outcomes uses those outcomes to change the predictions, so its apparent fit is not an evaluation of the unchanged prediction procedure on new data.

Worked Example: Perfect Correlation but Poor Prediction Error

For a hypothetical validation set, the observed values are 1, 2, 3, and 4 units, while the predictions are 12, 14, 16, and 18 units. Calculate correlation and R-squared relative to the validation mean.

Every prediction is $10+2y_i$. The predictions are an exact positive linear transformation of the observed values, so their Pearson correlation is $r_{y,\hat{y}}=1$, and $r_{y,\hat{y}}^2=1$.

The prediction errors are $-11,-12,-13,$ and $-14$ units. The squared errors sum to $$ 121+144+169+196=630\ \text{units}^2, $$ so $\mathrm{MSE}=630/4=157.5\ \text{units}^2$. The observed mean is 2.5 units, and the validation-mean baseline sum of squares is $$ (1-2.5)^2+(2-2.5)^2+(3-2.5)^2+(4-2.5)^2=5\ \text{units}^2. $$ Consequently, $$ R^2_{\mathrm{val\ mean}}=1-\frac{630}{5}=-125. $$ The predictions have perfect positive linear association but much greater squared error than predicting the validation mean. Their scale and offset are badly mismatched to the observed trait scale. The correlation does not reveal this problem; the error metric and baseline-relative R-squared do.

The Baseline Changes the Interpretation

When validation outcomes differ in average level from the training data, a training-set mean can be a more operational comparator than the validation mean. It is available before validation outcomes are observed. The two baselines answer different questions: one compares with the validation mean for that set, while the other compares with a fixed prediction that could have been made from training data.

Worked Example: Comparing Two Mean Baselines

Suppose a hypothetical training set has mean trait value 4 units. In a separate validation set, the observed values are 6, 8, 10, and 12 units, and a fitted prediction method gives predictions of 5, 7, 9, and 11 units. Compare the method with the training-mean baseline and with the validation-mean baseline.

Each model prediction is 1 unit below its observation, so the model sum of squared errors is $4(1^2)=4\ \text{units}^2$, and its MSE is $4/4=1\ \text{unit}^2$.

The training-mean baseline predicts 4 units for every person. Its squared error is $$ (6-4)^2+(8-4)^2+(10-4)^2+(12-4)^2 =4+16+36+64=120\ \text{units}^2. $$ The model’s skill relative to this fixed baseline is $$ R^2_{\mathrm{baseline}}=1-\frac{4}{120}=0.9667. $$

The validation mean is 9 units, giving a baseline sum of squares of $$ (6-9)^2+(8-9)^2+(10-9)^2+(12-9)^2 =9+1+1+9=20\ \text{units}^2. $$ Thus $R^2_{\mathrm{val\ mean}}=1-4/20=0.80$. Both scores use the same model errors, but one compares them with a training-available baseline and the other with the validation mean. Reporting the baseline makes the values interpretable and comparable.

Choosing and Reporting Metrics Carefully

No single metric fully describes quantitative prediction. MSE reports average squared error in squared trait units; a baseline-relative R-squared reports improvement or deterioration against an explicit reference; and correlation reports linear association. Read together, they distinguish prediction accuracy from association. If predictions are systematically shifted or rescaled, MSE and R-squared will reflect the resulting errors, while correlation can remain high.

For a clear report, state the validation population and trait units, the way predictions were generated, the MSE, and the exact baseline used for any R-squared. Report correlation separately if it addresses a relevant question, and do not label squared correlation as variance explained without stating the specific setting that makes that interpretation valid. These metrics describe performance on the evaluated population and scale; they do not by themselves establish why predictions succeed or fail.

Common mistake: “The squared correlation is 0.8, so the model explains 80% of the trait variance out of sample.” This reasoning treats association as error reduction against a baseline. The correction is to calculate the prediction errors, state the comparator, and compute $1-\mathrm{SSE}_{\mathrm{model}}/\mathrm{SSE}_{\mathrm{baseline}}$ for the chosen baseline. Squared correlation and baseline-relative R-squared answer different questions in general.

Key takeaway: For quantitative-trait prediction, always pair the metric with its definition and reference. MSE measures squared prediction error; out-of-sample R-squared compares that error with a stated baseline; correlation measures linear association and is not generally a substitute for either.

Check Your Understanding

Use the definitions and examples above to answer these questions.

A model has a negative R-squared relative to the validation mean. What does that say about its squared error compared with the mean predictor?
Why can adding a constant to every prediction leave correlation unchanged while changing MSE?
What information must be stated to interpret a baseline-relative R-squared?
When does the in-sample R-squared from regressing outcomes on predictions with an intercept equal squared correlation, and why does this not generally validate the original predictions?
Why are the validation mean and the training mean distinct possible baselines for an out-of-sample comparison?