Normal Errors and Small-Sample Inference

Why Normal Errors Matter for Small Samples

Ordinary least squares (OLS) produces a slope and intercept by minimizing the sum of squared residuals. That calculation does not require the errors to be normally distributed. Normality matters for a different reason: together with the regression model's other assumptions, it gives exact finite-sample distributions for coefficient estimates and makes small-sample tests and confidence intervals possible.

The word exact is important. Under the normal-error model, the relevant sampling distributions are known for a finite sample, rather than justified only as approximations that become more accurate as the sample grows. This tutorial develops that result for simple linear regression. The results are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Model and assumptions: For observations \(i=1,\ldots,n\), write \(Y_i=\beta_0+\beta_1x_i+\epsilon_i\), where the \(x_i\) values are fixed or the model is understood conditional on their observed values. Assume the errors are independent and identically distributed as \(N(0,\sigma^2)\), with \(\sigma^2>0\). Also require \(n>2\) and \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2>0\). The errors have conditional mean zero and constant conditional variance; the additional normality assumption specifies their conditional distribution.

Here \(\beta_0\) and \(\beta_1\) are the true intercept and slope, while \(\sigma^2\) is the common error variance. The conditions \(n>2\) and \(S_{xx}>0\) ensure that there are residual degrees of freedom and that the predictor values are not all identical. Without \(S_{xx}>0\), a slope cannot be estimated from the data.

The assumption concerns the unobserved errors, not the observed residuals. Residuals are calculated after fitting the line and are not independent draws from the error distribution. A residual plot can help assess whether the normal-error model is plausible, but residuals do not directly reveal the errors.

What Conditional Normality Gives

The OLS slope and intercept are linear combinations of the observed responses. Under the model above, each response is a fixed mean plus a normal error. Therefore, any linear combination of the responses is normally distributed. In particular, the coefficient estimates are normally distributed around their true values.

\hat{\beta}_1 \sim N\left(\beta_1,\frac{\sigma^2}{S_{xx}}\right), \qquad \hat{\beta}_0 \sim N\left(\beta_0,\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}\right)\right).

These variances describe repeated samples of errors at the same predictor values. They show, for example, that the slope estimate is more precise when \(S_{xx}\) is larger, all else equal. The distribution of the slope estimate is normal even when \(\sigma^2\) is unknown; what changes in practice is that an estimated standard error must be used.

Worked Example: The Exact Distribution of a Slope Estimate

Suppose a hypothetical model for monthly claim-processing time has \(n=6\) observations, \(S_{xx}=20\), and independent errors distributed as \(N(0,9)\), in squared time units. The true slope is \(\beta_1=1.2\) time units per predictor unit. Find the sampling distribution of \(\hat{\beta}_1\), its standard deviation, and the probability that it is within 1.5 time units per predictor unit of the true slope.

Method. Apply the normal sampling distribution of the OLS slope, which is valid here because the errors are independent, normal, and have variance \(\sigma^2=9\).

First, the mean of \(\hat{\beta}_1\) is the true slope, \(1.2\). Its variance is \(9/20=0.45\), so its standard deviation is \(\sqrt{0.45}=0.670820\ldots\). Thus \(\hat{\beta}_1\sim N(1.2,0.45)\), where the second parameter is the variance.

Standardizing the requested event gives \[ P(|\hat{\beta}_1-1.2|<1.5) =P\left(\left|Z\right|<\frac{1.5}{\sqrt{0.45}}\right), \] where \(Z\sim N(0,1)\). The standardized bound is \(1.5/0.670820\ldots=2.236068\ldots\), so the probability is approximately \(P(|Z|<2.2361)=0.9747\). The method uses the known \(\sigma^2\); it does not require estimating it from the six observations.

Check. The interval around the mean is roughly 2.24 standard deviations in each direction, so a probability near 0.975 is consistent with the standard normal distribution. A common mistake is to use \(S_{xx}\), rather than \(\sigma^2/S_{xx}\), as the variance. The correct variance is \(9/20\), not 20.

From a Normal Estimate to a t Statistic

In most applications, \(\sigma^2\) is unknown. Estimate it using the residual sum of squares. Let \(\hat{Y}_i=\hat{\beta}_0+\hat{\beta}_1x_i\), \(e_i=Y_i-\hat{Y}_i\), and \(\mathrm{SSE}=\sum_{i=1}^n e_i^2\). The residual variance estimate is \(s^2=\mathrm{SSE}/(n-2)\); its square root \(s\) is the estimated error standard deviation. The estimated standard error of the slope is \(s/\sqrt{S_{xx}}\).

Theorem: Under the model and assumptions stated above, \(\mathrm{SSE}/\sigma^2\) has a chi-square distribution with \(n-2\) degrees of freedom and is independent of \(\hat{\beta}_1\). Consequently, for any specified slope value \(\beta_{1,0}\), \(t=(\hat{\beta}_1-\beta_{1,0})/(s/\sqrt{S_{xx}})\) has a \(t\) distribution with \(n-2\) degrees of freedom when \(\beta_1=\beta_{1,0}\).

The same result applies to the intercept with its own estimated standard error. The degrees of freedom are \(n-2\) because two parameters, the intercept and slope, are estimated. A \(t\) distribution has heavier tails than the standard normal distribution, reflecting the extra uncertainty from estimating \(\sigma\). As the degrees of freedom increase, the \(t\) distribution approaches the standard normal distribution.

Why the t Result Holds

The justification combines two facts about normal errors: linear combinations of independent normal variables are normal, and orthogonal components of a multivariate normal vector are independent. The slope estimate uses one component of the response variation; the residuals occupy the remaining part after fitting an intercept and slope.

Write the slope estimate as a linear combination.
Let \(c_i=x_i-\bar{x}\). The OLS slope is \(\hat{\beta}_1=\sum_i c_iY_i/S_{xx}\). Since \(\sum_i c_i=0\) and \(\sum_i c_ix_i=S_{xx}\), substituting the regression model gives \(\hat{\beta}_1=\beta_1+\sum_i c_i\epsilon_i/S_{xx}\).

Obtain its normal distribution.
The error combination is normal, has mean zero, and has variance \(\sigma^2\sum_i c_i^2/S_{xx}^2=\sigma^2/S_{xx}\), since \(\sum_i c_i^2=S_{xx}\). This establishes \(\hat{\beta}_1\sim N(\beta_1,\sigma^2/S_{xx})\).

Describe the residual variation.
Fitting an intercept and slope removes two dimensions of the \(n\)-dimensional response vector. The remaining residual vector has \(n-2\) orthogonal normal components, so its squared length divided by \(\sigma^2\), which is \(\mathrm{SSE}/\sigma^2\), has a chi-square distribution with \(n-2\) degrees of freedom.

Form the t ratio.
The residual components are orthogonal to the component determining the slope estimate. For a multivariate normal vector, those orthogonal components are independent. Thus the standardized normal numerator \((\hat{\beta}_1-\beta_1)/(\sigma/\sqrt{S_{xx}})\) is independent of the chi-square estimate of \(\sigma^2\). Dividing that numerator by the square root of the independent chi-square variable divided by its degrees of freedom gives a \(t_{n-2}\) variable.

This argument explains why normality is essential to the exact finite-sample t result: it supplies normal linear combinations, a chi-square residual sum of squares, and the required independence. The result is not a claim that every collection of errors with mean zero and constant variance has an exact t statistic.

Confidence Intervals and a Small-Sample Calculation

For a two-sided confidence interval with confidence level \(1-\alpha\), use the \(1-\alpha/2\) quantile of the \(t\) distribution with \(n-2\) degrees of freedom. The interval is centered at the estimated slope and its margin of error is that quantile times the estimated standard error. This interval has exact coverage under the stated assumptions.

\hat{\beta}_1 \mathbin{\pm} t_{1-\alpha/2,\,n-2}\frac{s}{\sqrt{S_{xx}}}

Worked Example: A 95% Slope Interval with Six Residual Degrees of Freedom

A hypothetical study of repair duration and equipment age reports \(n=8\), \(\hat{\beta}_1=1.5\), \(S_{xx}=12\), and \(\mathrm{SSE}=24\). Assume the simple-regression model has independent \(N(0,\sigma^2)\) errors with a common variance. Compute a 95% confidence interval for the slope.

Method. Estimate the error variance, calculate the slope standard error, and use the \(t\) critical value with \(8-2=6\) degrees of freedom.

The residual degrees of freedom are \(n-2=6\). Thus \(s^2=24/6=4\) and \(s=2\). The estimated standard error of the slope is \(s/\sqrt{S_{xx}}=2/\sqrt{12}=\sqrt{4/12}=0.577350\ldots\). For 95% confidence, \(t_{0.975,6}\approx2.447\). The margin of error is \(2.447\sqrt{4/12}=1.412776\ldots\).

The endpoints are \(1.5-1.412776\ldots=0.087224\ldots\) and \(1.5+1.412776\ldots=2.912776\ldots\). Rounded to three decimal places, the interval is \((0.087,2.913)\).

Check. The interval is symmetric around 1.5, and its width is twice the margin, about 2.826. A common exam trap is to use the standard normal critical value 1.96, which would give a margin of about 1.132 and an unjustifiably narrow interval here. Because \(\sigma^2\) is estimated and there are only six residual degrees of freedom, the appropriate critical value is from \(t_6\), not the standard normal distribution.

OLS Does Not Require Normal Errors

The OLS estimates are obtained from the observed \(x_i\) and \(Y_i\) values by minimizing \(\sum_i(Y_i-b_0-b_1x_i)^2\). Provided \(S_{xx}>0\), that calculation is defined whether the errors are normal, skewed, or have another distribution. Normality is not a prerequisite for drawing the least-squares line.

This distinction does not mean that all statistical conclusions remain unchanged when normality is removed. Under other suitable assumptions, coefficient estimates can have useful properties, and large-sample approximations may be available. But the exact small-sample normal and t distributions derived here are not guaranteed just because the OLS coefficients can be calculated. Nor does assuming normality excuse a failure of the model's independence, linear conditional mean, or constant-variance assumptions.

Worked Example: Calculating OLS Without Claiming Exact t Inference

A hypothetical analyst records predictor \(x\) and response \(Y\) for four service cases: \((0,2),(1,3),(2,5),(3,6)\). Calculate the OLS line and residual variance estimate. Then state what can and cannot be concluded if normal errors have not been assumed.

Method. Calculate the centered sums and use the least-squares formulas. This arithmetic requires \(S_{xx}>0\), but does not require a normal-error assumption.

The means are \(\bar{x}=(0+1+2+3)/4=1.5\) and \(\bar{Y}=(2+3+5+6)/4=4\). The centered predictor values are \(-1.5,-0.5,0.5,1.5\), so \(S_{xx}=2.25+0.25+0.25+2.25=5\). The centered response values are \(-2,-1,1,2\); hence \(S_{xY}=(-1.5)(-2)+(-0.5)(-1)+(0.5)(1)+(1.5)(2)=3+0.5+0.5+3=7\).

Therefore \(\hat{\beta}_1=7/5=1.4\) and \(\hat{\beta}_0=\bar{Y}-\hat{\beta}_1\bar{x}=4-(1.4)(1.5)=1.9\). The fitted values are \(1.9,3.3,4.7,6.1\), giving residuals \(0.1,-0.3,0.3,-0.1\). Thus \(\mathrm{SSE}=0.01+0.09+0.09+0.01=0.20\), and \(s^2=0.20/(4-2)=0.10\).

The OLS line is \(\hat{Y}=1.9+1.4x\), and the residual variance calculation is available. But without the normal-error assumption, these data alone do not justify treating the slope t ratio as exactly \(t_2\)-distributed. A common mistake is to infer that an exact t test is valid simply because the OLS calculation and \(s^2\) are available. Those are separate claims: computing the quantities does not establish the distribution needed for exact inference.

Common Mistakes and Practical Interpretation

Confusing errors with residuals. The model assumes a distribution for unobserved errors conditional on predictor values. Residuals are fitted quantities and need not be independent normal observations.
Using a normal critical value in a small-sample t interval. When the error variance is estimated, the exact reference distribution under the model is \(t_{n-2}\).
Assuming normality makes OLS possible. The least-squares formulas are algebraic. Normality is used to obtain the exact sampling distributions, not to define the fitted line.
Treating normality as the only assumption. The exact results also rely on the linear model, independent errors, constant variance, and a nonconstant predictor.
Forgetting the residual degrees of freedom. In simple regression, two coefficients are estimated, so the error degrees of freedom are \(n-2\), not \(n-1\).

Key takeaway: With independent errors conditionally distributed as \(N(0,\sigma^2)\), the OLS slope is normal with variance \(\sigma^2/S_{xx}\), and replacing \(\sigma\) with the residual estimate yields an exact \(t_{n-2}\) statistic. OLS itself can still be computed without normality; exact small-sample inference is the part that depends on it.

Check Your Understanding

For each problem, identify which result follows from the stated assumptions and keep known error variance distinct from estimated error variance.

In a simple regression with independent \(N(0,9)\) errors and \(S_{xx}=25\), what are the variance and standard deviation of \(\hat{\beta}_1\)? (Answer: variance \(=0.36\); standard deviation \(=0.6000\).)
A regression has \(n=9\), independent normal errors with common variance, and a nonconstant predictor. How many residual degrees of freedom are used for the exact slope t statistic? (Answer: \(7\).)
A regression has \(n=8\), \(\hat{\beta}_1=2.4\), \(s^2=9\), and \(S_{xx}=36\). For testing \(H_0:\beta_1=2\), what is the t statistic and its degrees of freedom, assuming the model's normal-error conditions hold? (Answer: \(t=0.800\) with \(6\) degrees of freedom.)
With known error variance \(\sigma^2=4\) and \(S_{xx}=10\), what is the sampling distribution of the slope estimate? (Answer: \(\hat{\beta}_1\sim N(\beta_1,0.4)\), with standard deviation \(\sqrt{0.4}\approx0.6325\).)
A regression has \(n=7\), \(\hat{\beta}_1=1.2\), \(s^2=2.25\), and \(S_{xx}=9\). Assuming independent normal errors with constant variance, give a 95% slope interval using \(t_{0.975,5}=2.571\), rounded to three decimals. (Answer: standard error \(=0.500\); interval \((-0.086,2.486)\).)
For a data set with \(S_{xx}>0\), can the OLS slope be calculated if normality of the errors is not assumed, and does the exact small-sample t result automatically follow? (Answer: The OLS slope can be calculated; the exact t result does not automatically follow.)

What You'll Learn