How Sample Size Changes the Bias-Variance Balance

Why More Training Observations Can Help

A fitted model depends on the training observations used to estimate it. If a different representative sample would produce a noticeably different fitted model, the model has substantial estimation variance. With an appropriate sampling design and a fixed estimation procedure, a larger sample often makes those fitted results more stable.

That improvement has a limit. More observations can estimate the best fit within a chosen model class more precisely, but they do not necessarily make that model class capable of representing the underlying relationship. If the model is systematically too simple or otherwise misspecified, its approximation bias can remain even as estimation variance falls.

Definition: At a fixed predictor value \(x\), let \(f(x)=E[Y\mid X=x]\) be the true conditional mean response, and let \(\hat f_D(x)\) be the fitted prediction from training data \(D\). The estimation variance at \(x\) is \(\operatorname{Var}_D(\hat f_D(x))\), the variation in fitted predictions over possible training samples. The bias is \(E_D[\hat f_D(x)]-f(x)\). For a fixed model class, approximation bias describes the systematic difference between the best fit that class can achieve and the true relationship.

Here, “more representative” means that the additional observations come from the population and sampling process relevant to the prediction task, rather than systematically overrepresenting some cases or omitting others. Sample size alone does not guarantee representativeness.

The Sample-Size Effect on Estimation Variance

A basic illustration is the sample mean. Suppose \(X_1,\ldots,X_n\) are independent observations from the same distribution, each with mean \(\mu\) and finite variance \(\sigma^2\). Their sample mean estimates \(\mu\).

\bar X=\frac{1}{n}\sum_{i=1}^{n}X_i, \qquad E[\bar X]=\mu, \qquad \operatorname{Var}(\bar X)=\frac{\sigma^2}{n}.

The expected estimate is centered at \(\mu\), so the estimator is unbiased under these assumptions. Its variance decreases in proportion to \(1/n\), while its standard error decreases in proportion to \(1/\sqrt n\). Thus, multiplying the sample size by four divides the variance by four and the standard error by two; it does not divide the standard error by four.

Why the variance formula holds: Independence makes the covariance between distinct observations zero. Therefore,

\operatorname{Var}(\bar X) =\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^{n}X_i\right) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(X_i) =\frac{1}{n^2}(n\sigma^2) =\frac{\sigma^2}{n}.

If the observations are dependent, or their distributions differ in ways the calculation does not account for, the zero-covariance and common-variance steps may fail. The formula is not a general rule for every dataset.

Worked Example: Precision of a Normal-Sample Mean

A hypothetical pricing analyst uses \(n\) independent, representative observations of a quantity that follows \(X_i\sim N(\mu,100)\), where the second parameter is the variance. Compare the estimator’s variance, standard error, and probability of estimating \(\mu\) within 5 units when \(n=4\) and \(n=16\).

Method. Since the observations are independent and normally distributed, \(\bar X\sim N(\mu,100/n)\). The estimator is unbiased, and its standard error is \(\sqrt{100/n}\).

For \(n=4\), the variance is \(100/4=25\), and the standard error is \(\sqrt{25}=5\). The standardized distance for an error of 5 is \(5/5=1\). Hence \(P(|\bar X-\mu|\leq 5)=P(|Z|\leq 1)=0.6827\), where \(Z\sim N(0,1)\).

For \(n=16\), the variance is \(100/16=6.25\), and the standard error is \(\sqrt{6.25}=2.5\). Now the standardized distance is \(5/2.5=2\), so \(P(|\bar X-\mu|\leq 5)=P(|Z|\leq 2)=0.9545\).

Check. The sample size has increased by a factor of 4, the variance has fallen by a factor of 4, and the standard error has halved. The probability of an error within 5 units has increased, as expected. This calculation avoids the common mistake of treating standard error as if it decreased in direct proportion to sample size.

Why Approximation Bias Can Remain

A useful way to separate sources of prediction error is to consider a fixed predictor value \(x\). Suppose a new response is \(Y=f(x)+\varepsilon\), where the new response noise has mean zero and variance \(\sigma^2\), and is independent of the training data. Then

E_{D,\varepsilon}\left[(Y-\hat f_D(x))^2\right] = \sigma^2 + \operatorname{Var}_D(\hat f_D(x)) + \left(E_D[\hat f_D(x)]-f(x)\right)^2.

The first term is response noise, the second is estimation variance, and the third is squared bias. Increasing the sample size may reduce the second term. It does not, by itself, remove the first term or guarantee that the third term shrinks. In particular, when the model class is fixed and cannot express the true relationship, its best attainable fit can remain systematically different from that relationship.

Worked Example: A Fixed Model Misses a Predictor Effect

In a hypothetical study, the true conditional mean response is \(f(x)=8+4x\), for \(x\in\{0,1\}\). Training responses follow \(Y_i=8+4x_i+\varepsilon_i\), with independent errors \(\varepsilon_i\sim N(0,4)\). The training design contains exactly half \(x_i=0\) and half \(x_i=1\). The analyst fits an intercept-only model, whose prediction is the sample mean. Compare its expected squared prediction error at \(x=0\) for \(n=25\) and \(n=100\). Assume a new response at \(x=0\) has independent error with the same variance 4.

Method. An intercept-only model cannot represent the difference between the two predictor groups. With a balanced design, the mean of the two true group means is \((8+12)/2=10\). Thus the expected fitted intercept is 10, whereas the true conditional mean at \(x=0\) is 8.

The bias at \(x=0\) is \(10-8=2\), and the squared bias is \(2^2=4\). Since the response errors have variance 4, the sample mean has estimation variance \(4/n\).

For \(n=25\), the estimation variance is \(4/25=0.16\). The expected squared prediction error for a new response is \(4+0.16+4=8.16\): response noise \(4\), estimation variance \(0.16\), and squared bias \(4\).

For \(n=100\), the estimation variance is \(4/100=0.04\), so the expected squared prediction error is \(4+0.04+4=8.04\).

Check. The approximation bias remains 2 in both cases because the fitted model still has no predictor effect. The total prediction error falls by only \(8.16-8.04=0.12\); it does not fall to zero. This avoids the mistake of assuming that more data can repair a limitation of the chosen model class.

When the Nominal Sample Size Overstates the Information

The \(1/n\) variance result depends on independent observations. In many settings, observations arrive in groups and are more alike within a group than across groups. Counting every record as independent then overstates the information in the sample.

Worked Example: Repeated Records Within Clusters

A hypothetical insurer studies a response \(Z_{ij}\) for \(K=10\) independent groups, with \(m=5\) records in each group. Assume \(Z_{ij}=\mu+U_j+\varepsilon_{ij}\), where the group effects \(U_j\sim N(0,4.5)\) and the record errors \(\varepsilon_{ij}\sim N(0,4.5)\) are mutually independent. The total of 50 records is balanced across groups. Compare the variance of the overall sample mean with the variance that would apply if all 50 records were independent with the same marginal variance.

Method. Each record has marginal variance \(4.5+4.5=9\). Two records in the same group share \(U_j\), so their covariance is \(4.5\). Their correlation is \(4.5/9=0.5\).

The overall mean can be written as the average of the \(K\) group means. A group mean has variance \(4.5+4.5/5=4.5+0.9=5.4\): the group effect contributes 4.5, and averaging five independent record errors contributes \(4.5/5=0.9\). Since the 10 groups are independent, the overall mean has variance \(5.4/10=0.54\).

If all 50 records were independent, the variance would instead be \(9/50=0.18\). That calculation is too small for this clustered design. An equivalent effective sample size, defined here as marginal variance divided by the actual variance of the mean, is \(9/0.54=16.6667\), approximately 16.67 records.

Check. The total count is 50, but the shared group effects make its information closer to that of 16.67 independent records under this definition. Adding more independent groups would generally help more than adding the same number of highly similar records within existing groups. This avoids applying the independent-observation formula without checking the sampling structure.

More Data Cannot Correct an Unrepresentative Sample

A large sample can be precisely unrepresentative. For example, if an estimator is based only on one population segment while the target includes several segments, increasing the number of observations from the first segment can reduce variation around the wrong target. The estimate may become more stable without becoming more accurate for the intended population.

Worked Example: A Larger Sample From the Wrong Segment

A hypothetical target population is 70% urban and 30% rural. The response has mean 100 in the urban segment and mean 140 in the rural segment. An analyst samples only urban cases and uses their sample mean to estimate the target-population mean. Assume urban observations are independent and follow \(N(100,400)\). Compare the estimator’s mean squared error for \(n=25\) and \(n=100\).

Method. The target-population mean is \(0.70(100)+0.30(140)=70+42=112\). Because the sample contains only urban observations, the expected sample mean is 100. Its bias relative to the target is \(100-112=-12\), and its squared bias is 144.

For \(n=25\), the sample-mean variance is \(400/25=16\). Its mean squared error for estimating 112 is \(16+144=160\). For \(n=100\), the variance is \(400/100=4\), and the mean squared error is \(4+144=148\).

Check. The variance falls by 12, but the bias remains \(-12\) because the rural segment is still absent. The larger sample improves precision around 100, not representativeness of the target population. This avoids confusing a large sample with a well-designed sample.

Conditions and Common Traps

The usual sample-size argument is strongest when the model and estimation procedure are held fixed, the added observations are relevant and representative, and the observations are independent or their dependence is correctly accounted for. Under those conditions, additional data often reduce estimation uncertainty. The result is not a guarantee that every fitted model’s prediction error will fall for every finite increase in sample size.

Do not confuse variance with bias. Variance describes how estimates change across training samples; bias describes systematic displacement from the target.
Do not confuse standard error with variance. If variance scales as \(1/n\), standard error scales as \(1/\sqrt n\).
Do not count correlated records as independent. Shared subjects, locations, time periods, or groups can make the effective information much smaller than the record count suggests.
Do not assume additional data are representative. A larger sample drawn from the wrong population can become more precise about the wrong quantity.
Do not assume all error disappears. Even with very large samples, approximation bias from a fixed inadequate model and irreducible response noise can remain.

Key takeaway: More representative training observations often reduce estimation variance, but they do not automatically remove approximation bias or response noise. Check the model class, sampling frame, and dependence structure before interpreting a larger sample as a substantial improvement in prediction.

Check Your Understanding

For each problem, distinguish the changing variance from any bias or response-noise contribution.

Independent observations follow \(N(\mu,64)\). Find the variance and standard error of the sample mean for \(n=25\) and \(n=100\). State the factor by which each changes. (Answer: Variances \(2.56\) and \(0.64\); standard errors \(1.6\) and \(0.8\); variance is divided by 4 and standard error by 2.)
At a fixed predictor value, the true conditional mean is 30. A fixed model has expected fitted prediction 26, prediction-estimator variance 2.25, and new-response noise variance 9. Find squared bias and expected squared prediction error. (Answer: Bias \(-4\), squared bias \(16\), and expected squared prediction error \(9+2.25+16=27.25\).)
A population is 60% segment A with mean 20 and 40% segment B with mean 35. A sample contains only segment A observations, and its sample mean estimates the population mean. Find the population mean and the estimator’s bias, using estimate minus target as the bias convention. (Answer: Population mean \(0.60(20)+0.40(35)=26\); bias \(20-26=-6\), regardless of the number of A observations.)
There are \(K=8\) independent groups with \(m=4\) observations per group. Each observation has marginal variance 16, and within-group correlation is 0.25. Using the equal-group design effect \(1+(m-1)\rho\), find the variance of the overall mean and its effective sample size. (Answer: Design effect \(1+3(0.25)=1.75\); variance \((16/32)(1.75)=0.875\); effective sample size \(16/0.875\approx18.2857\).)
A fixed prediction procedure has squared bias 9, response-noise variance 25, and estimation variance \(25/n\). Find expected squared prediction error at \(n=100\) and \(n=400\), and explain which component changed. (Answer: At \(n=100\), \(25+9+0.25=34.25\); at \(n=400\), \(25+9+0.0625=34.0625\). Only estimation variance changed.)

What You'll Learn