Exam SRM · Bias, Variance, and Overfitting · Tutorial 759 of 926

Variance as Training-Sample Sensitivity

Learn to quantify how much a fitted prediction changes when the training sample changes, while holding the input and fitting procedure fixed.

Intermediate 10 min read

What You'll Learn

Define prediction variance at a fixed input across repeated training samples
Distinguish training-sample sensitivity from response noise and prediction error
Calculate the variance of predictions from repeated fitted models
Use the sample variance to estimate repeated-sample prediction variance
Compare prediction stability at the same input and identify common calculation traps

Introduction

A fitted model can give different predictions when it is trained on different samples from the same population. The size of this variation matters: a prediction that changes substantially with the training data is less stable than one that remains similar. To assess that stability, hold the input fixed and examine the predictions produced by repeated training samples.

This tutorial focuses on prediction variance as a measure of training-sample sensitivity. It is not the variability of outcomes around their conditional mean, and it is not the prediction error of one fitted model. The key comparison is between predictions at the same input from models fitted to different training samples.

Definition: Let $\mathcal D$ denote a training sample drawn from a specified population, and let $\hat f_{\mathcal D}(x)$ be the prediction at a fixed input $x$ from a specified fitting procedure. The prediction variance at $x$ is $$ \operatorname{Var}_{\mathcal D}\!\left(\hat f_{\mathcal D}(x)\right) = E_{\mathcal D}\!\left[ \left\{ \hat f_{\mathcal D}(x) - E_{\mathcal D}\!\left[\hat f_{\mathcal D}(x)\right] \right\}^2 \right]. $$ The expectation and variance are over repeated training samples generated under the same population mechanism, using the same sample size and fitting procedure. The input $x$ is held fixed.

The repeated-sample mean prediction, $E_{\mathcal D}[\hat f_{\mathcal D}(x)]$, is the center of this distribution of fitted predictions. Prediction variance describes their spread around that center. A variance of zero means the procedure always gives the same prediction at $x$ across the specified training samples. A larger variance means greater sensitivity at that input.

Calculating Prediction Variance from Replicate Fits

In practice, the full distribution over all possible training samples is usually unknown. Instead, we can generate $B$ independent training samples under the same setup, fit the procedure to each, and record the $B$ predictions at the same input. Their average is $$ \overline{\hat f}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat f_b(x), $$ and the usual estimate of repeated-sample prediction variance is the sample variance $$ s^2(x)=\frac{1}{B-1}\sum_{b=1}^{B}\left\{\hat f_b(x)-\overline{\hat f}(x)\right\}^2. $$

Here, $\hat f_b(x)$ is the prediction from the $b$th fitted model. The denominator $B-1$ makes $s^2(x)$ an unbiased estimate of the variance of the replicate predictions when those predictions are independent and identically distributed with finite variance. The exact population variance uses an expectation, not a denominator of $B-1$; the sample variance is an estimate based on finitely many replicate fits.

Worked Example: Estimating Prediction Variance at a Fixed Risk Profile

A hypothetical insurer studies a continuous outcome, measured in thousands of dollars, for a fixed risk profile $x$. The analyst independently draws $B=5$ training samples of the same size from the same population and applies the same deterministic fitting procedure to each. The resulting predictions at $x$ are $7, 9, 10, 12,$ and $12$ thousand dollars. Calculate the mean prediction and the sample estimate of prediction variance at $x$.

Method. Compute the mean across the five fitted predictions, then sum their squared deviations from that mean and divide by $B-1$. No parametric distribution for the replicate predictions is assumed.

Mean prediction. $$ \overline{\hat f}(x)=\frac{7+9+10+12+12}{5}=\frac{50}{5}=10. $$

Squared deviations. The deviations from 10 are $-3,-1,0,2,2$, so their squares are $9,1,0,4,4$. Their sum is $$ 9+1+0+4+4=18. $$

Sample variance. Since $B-1=4$, $$ s^2(x)=\frac{18}{4}=4.5 $$ in squared thousands of dollars. The estimated standard deviation of the predictions is $\sqrt{4.5}\approx 2.121$ thousand dollars. The calculation is checked by noting that the squared deviations sum to 18 and that $18/4=4.5$.

Common mistake avoided. Dividing by 5 gives $18/5=3.6$, which is the variance of these five values when treated as the entire distribution of interest. Here they are a sample of replicate fits used to estimate repeated-sample variance, so the sample-variance denominator is 4.

Why the Pairwise-Spread Formula Works

Prediction variance can also be interpreted through differences between independent replicate predictions. This gives a useful justification for why larger changes across training samples indicate greater variance.

Result: If $\hat f_1(x)$ and $\hat f_2(x)$ are independent predictions from two training samples generated under the same procedure, then $$ E\!\left[\{\hat f_1(x)-\hat f_2(x)\}^2\right] = 2\operatorname{Var}_{\mathcal D}\!\left(\hat f_{\mathcal D}(x)\right). $$

Write the variance of a difference.
For independent random variables, the variance of their difference is the sum of their variances: $$ \operatorname{Var}\!\left(\hat f_1(x)-\hat f_2(x)\right) = \operatorname{Var}(\hat f_1(x))+\operatorname{Var}(\hat f_2(x)). $$ The covariance term is zero by independence.

Use the common replicate distribution.
Both predictions have the same mean $\mu_x=E_{\mathcal D}[\hat f_{\mathcal D}(x)]$ and the same variance $V_x=\operatorname{Var}_{\mathcal D}(\hat f_{\mathcal D}(x))$. Therefore, the variance of their difference is $V_x+V_x=2V_x$.

Relate variance to the squared difference.
The difference has mean $\mu_x-\mu_x=0$. For any random variable $Z$ with mean zero, $E[Z^2]=\operatorname{Var}(Z)$. Thus $E[\{\hat f_1(x)-\hat f_2(x)\}^2]=2V_x$, as claimed.

The result requires independent replicate predictions with the same distribution. If two training samples overlap or are otherwise dependent, the covariance need not be zero, and the stated equality need not hold.

Worked Example: Comparing Two Procedures by Their Prediction Spread

An analyst compares two deterministic procedures for predicting a hypothetical claim-related outcome at one fixed profile. For each procedure, four independent training samples of the same size are drawn from the same population. Procedure A gives predictions $49, 50, 50,$ and $51$; Procedure B gives $44, 48, 52,$ and $56$. All predictions are in thousands of dollars. Calculate the sample variance for each procedure and identify which is more sensitive at this input.

Procedure A. Its mean prediction is $$ \overline{\hat f}_A(x)=\frac{49+50+50+51}{4}=\frac{200}{4}=50. $$ The deviations are $-1,0,0,1$, whose squares sum to $1+0+0+1=2$. With $B-1=3$, $$ s_A^2(x)=\frac{2}{3}\approx 0.667. $$

Procedure B. Its mean prediction is $$ \overline{\hat f}_B(x)=\frac{44+48+52+56}{4}=\frac{200}{4}=50. $$ The deviations are $-6,-2,2,6$, whose squares sum to $36+4+4+36=80$. Therefore, $$ s_B^2(x)=\frac{80}{3}\approx 26.667. $$

Both procedures have the same mean prediction, but Procedure B has a much larger estimated prediction variance at this input. Its predictions are therefore more sensitive to changes in the training sample in these replicates. The checks are that each set sums to 200 and that the squared-deviation totals, 2 and 80, match the stated variances after division by 3.

Common mistake avoided. Comparing only the means would miss the difference in stability. A common center does not imply a common spread.

Prediction Variance Is Not Prediction Error

The earlier material on the bias-variance-noise decomposition distinguishes variation due to the fitted prediction from the variability of outcomes around their conditional mean. That distinction is essential here. Prediction variance concerns how $\hat f_{\mathcal D}(x)$ changes as $\mathcal D$ changes. Response noise concerns how $Y$ varies at a fixed $x$, even if the conditional mean $E[Y\mid x]$ is known exactly.

For example, suppose the true conditional mean at a profile is $100$, while outcomes vary around that mean. A procedure might always predict $100$, giving prediction variance zero at that profile, even though individual outcomes are not all $100$. Conversely, a procedure might produce changing predictions across samples even when the outcome noise is small. The two sources of variation answer different questions.

Worked Example: Separating Prediction Spread from Response Noise

For a hypothetical outcome, suppose the true conditional mean at a fixed input is $100$ units and the outcome satisfies $Y\mid x \sim N(100,16)$, where 16 is the variance. A fitting procedure is applied to four independent training samples and gives predictions $98, 100, 100,$ and $102$. Calculate the sample estimate of prediction variance and state the outcome variance. Explain why the two values are not interchangeable.

Prediction mean. $$ \overline{\hat f}(x)=\frac{98+100+100+102}{4}=\frac{400}{4}=100. $$ The deviations are $-2,0,0,2$, so the squared deviations sum to $4+0+0+4=8$. The sample estimate of prediction variance is $$ s^2(x)=\frac{8}{4-1}=\frac{8}{3}\approx 2.667 $$ square units.

Outcome variance. From the stated conditional distribution, $\operatorname{Var}(Y\mid x)=16$ square units. This describes outcome variability around the true conditional mean, whereas $8/3$ estimates the variation among fitted predictions across training samples. They measure different random quantities. The check is that the four predictions average to 100 and have squared deviations totaling 8.

Common mistake avoided. Do not label the outcome variance 16 as the variance of the fitted predictions, or treat the fitted-prediction variance $8/3$ as the full uncertainty in an individual outcome.

Conditions for a Fair Sensitivity Comparison

A comparison of prediction variances is informative only when the repeated fits are defined consistently. To isolate the effect of changing training samples, keep the following fixed:

The input. Compare predictions at the same $x$, not at different profiles whose predictions may naturally differ.
The population mechanism and sample size. Training samples should be generated under the same specified setup and have the same size. Changing sample size changes the experiment being compared.
The fitting procedure. Use the same preprocessing, estimation, and tuning rules for every replicate. If tuning is part of the procedure, repeat that tuning in each replicate when measuring the sensitivity of the full procedure.
The role of algorithmic randomness. For a deterministic fitting procedure, there is no extra algorithmic randomness. For a stochastic procedure, either hold its random seed or other randomization fixed to isolate training-sample sensitivity, or deliberately include algorithmic randomness in the repeated procedure. State which quantity is being measured.

Prediction variance can depend on $x$. A procedure may give stable predictions in one region of the input space and unstable predictions in another. A single variance value therefore summarizes sensitivity only at the specified input and under the specified repeated-sample setup. It does not establish that the procedure is stable everywhere.

Common Exam Traps

Varying the input instead of the training sample. Differences between predictions at different inputs do not estimate training-sample prediction variance.
Using the wrong denominator. For $B$ observed replicate predictions used to estimate variance, use $B-1$. Dividing by $B$ instead calculates the spread of those values when treated as the full distribution.
Confusing spread with accuracy. Low prediction variance means predictions are stable across the specified samples; it does not by itself mean they are close to the true conditional mean.
Confusing prediction variance with outcome noise. The former varies across training samples; the latter varies across outcomes at fixed $x$.
Changing several parts of the experiment at once. If sample size, input, or fitting rules differ across replicates, the resulting spread cannot be attributed solely to the intended change in training data.

Key takeaway: To assess training-sample sensitivity, fix an input and a fitting procedure, vary the training sample under a specified common setup, and measure the spread of the resulting predictions. Prediction variance is a property of that repeated-sample experiment, not a universal constant of a model.

Check Your Understanding

For each problem, distinguish the repeated-sample prediction spread from any other quantity mentioned.

A deterministic procedure is fitted to three independent training samples of equal size. At the same input, the predictions are $5, 7,$ and $9$. Find the mean prediction and sample variance. (Answer: mean $7$; sample variance $4$.)
At a fixed profile, four replicate predictions are $20, 20, 22,$ and $22$. Calculate the sample variance, using the replicate predictions as a sample rather than as the entire distribution. (Answer: $4/3$.)
Two independent fits from the same procedure have mean prediction $30$ at a fixed input, and the prediction variance is $5$. Find the expected squared difference between their predictions. (Answer: $10$.)
At fixed $x$, outcomes satisfy $Y\mid x\sim N(40,25)$, and three replicate fitted predictions are $38,40,$ and $42$. Find the sample estimate of prediction variance and the conditional outcome variance. (Answer: prediction variance estimate $4$; conditional outcome variance $25$.)
A procedure gives predictions $10,12,14$ at input $x_1$, and $20,22,24$ at a different input $x_2$, using the same three training samples. Why is the difference between the average predictions at $x_1$ and $x_2$ not a measure of training-sample sensitivity? (Answer: it compares different inputs; sensitivity must be assessed from replicate predictions at the same input.)

Back to Tutorials