Why the Error Distribution Matters for Regression Inference
Regression estimates can be calculated whether or not the errors are normally distributed. The distinction matters when we use those estimates to quantify uncertainty. With normal errors and the other classical assumptions, regression t and F procedures have exact reference distributions in a finite sample. Without normality, those same reference distributions are generally not exact; large-sample approximations may instead be justified under additional conditions.
This is a statement about the distribution of a procedure over repeated samples, conditional on the predictors when they are treated as fixed. It does not say that the observed residuals must look normal, nor does it make every inference valid merely because a sample is large. Independence, the variance assumptions, the design, and the standard-error calculation all matter.
Under this model, the least-squares estimate is \(\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\), and the residual sum of squares is \(SSE=(Y-X\hat\beta)^\mathsf{T}(Y-X\hat\beta)\). The residual variance estimate is \(s^2=SSE/(n-p)\). In simple linear regression with an intercept, \(p=2\), so the residual degrees of freedom are \(n-2\). These definitions do not themselves require normally distributed errors; normality is what supplies exact finite-sample distribution results for inference.
Why the Coefficient t Statistic Is Exact
The key ingredients are the normal distribution of the coefficient estimate, a chi-square distribution for the scaled residual sum of squares, and independence between those two quantities. Their combination produces a Student t distribution rather than merely an approximation to one.
Since \(Y=X\beta+\varepsilon\), least squares gives \(\hat\beta=\beta+(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon\). Under the normal-error model, this is multivariate normal with mean \(\beta\) and covariance matrix \(\sigma^2(X^\mathsf{T}X)^{-1}\).
The residual vector is \(Y-X\hat\beta=(I-H)\varepsilon\), where \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) is the projection matrix onto the columns of \(X\). The residual projection has rank \(n-p\), so \(SSE/\sigma^2\sim\chi^2_{n-p}\).
The fitted and residual projections are orthogonal. For a multivariate normal error vector, these orthogonal projections are independent. Thus the coefficient estimate is independent of \(SSE\).
For any coefficient contrast \(c^\mathsf{T}\beta\), the estimated contrast has variance \(\sigma^2 c^\mathsf{T}(X^\mathsf{T}X)^{-1}c\). When testing \(H_0:c^\mathsf{T}\beta=c_0\), replacing \(\sigma\) by \(s\) gives the statistic below. Its numerator, when divided by the true standard deviation, is standard normal under the null. Its variance estimate contributes an independent chi-square quantity divided by its degrees of freedom. That ratio is exactly a Student t random variable.
The degrees of freedom are \(n-p\), not \(n\), because \(p\) fitted coefficients use up \(p\) dimensions in estimating the residual variation. If the errors are not normal, the coefficient estimate need not be normal and the residual sum of squares need not have the stated chi-square distribution. Consequently, the finite-sample t reference distribution is generally no longer exact.
Worked Example: Exact Slope Inference in a Small Normal-Error Sample
A hypothetical analyst models annual claim cost \(Y\), in thousands of dollars, against a fixed risk score \(x\). The four observed pairs are \((0,3),(1,4),(2,7),(3,12)\). Assume independent errors distributed as \(N(0,\sigma^2)\) with a common unknown variance, and include an intercept. Test \(H_0:\beta_1=0\) against a two-sided alternative at the 5% level. The \(t_{2}\) critical value is 4.303.
Method. Fit the least-squares line, calculate its residual variance and slope standard error, then use the exact t reference distribution justified by the stated normal-error model.
The predictor mean is \(\bar x=(0+1+2+3)/4=1.5\); the response mean is \(\bar y=(3+4+7+12)/4=6.5\). Therefore, \(S_{xx}=\sum(x_i-\bar x)^2=2.25+0.25+0.25+2.25=5\). Also, \(S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)=5.25+1.25+0.25+8.25=15\). Thus \(\hat\beta_1=S_{xy}/S_{xx}=15/5=3\), and \(\hat\beta_0=\bar y-\hat\beta_1\bar x=6.5-3(1.5)=2\). The fitted values are \(2,5,8,11\), giving residuals \(1,-1,-1,1\) and \(SSE=1+1+1+1=4\).
There are \(n-p=4-2=2\) residual degrees of freedom, so \(s^2=4/2=2\). The slope standard error is \(\sqrt{s^2/S_{xx}}=\sqrt{2/5}=\sqrt{0.4}\approx0.63246\). The test statistic is \(t=3/0.63246\approx4.7434\). Since \(|4.7434|>4.303\), reject the null at the 5% level under the assumed model.
Check. In this simple regression, the equivalent one-degree-of-freedom statistic is \(F=t^2\approx4.7434^2=22.50\), with degrees of freedom \((1,2)\). The conclusion agrees with the two-sided t test. A common mistake is to use a normal critical value because the coefficient estimate is being tested; here the exact finite-sample reference is \(t_2\), not the standard normal distribution.
Why the Overall and Joint F Procedures Are Exact
For a nested comparison, the unrestricted model estimates all \(p\) coefficients, while the restricted model imposes \(q\) independent linear restrictions. Let \(RSS_R\) and \(RSS_U\) denote their residual sums of squares. Under the null restrictions, the increase \(RSS_R-RSS_U\) measures the variation attributable to the restrictions. Under the normal-error model, this increase divided by \(\sigma^2\) has a chi-square distribution with \(q\) degrees of freedom. It is independent of \(RSS_U/\sigma^2\), which has a chi-square distribution with \(n-p\) degrees of freedom. Dividing each by its degrees of freedom and taking their ratio gives the exact F statistic:
The nested-model F result and the t result are related, but the number of restrictions determines which procedure is convenient. A single restriction can be tested with an F statistic equal to the square of the corresponding t statistic. For multiple restrictions, the F statistic tests them jointly; separate coefficient t tests do not generally answer the same question.
Worked Example: Exact Joint Test of Two Restrictions
A hypothetical insurer models annual administrative cost using an intercept and three fixed portfolio predictors. There are \(n=15\) observations. The unrestricted regression has \(p=4\) coefficients, and its residual sum of squares is \(RSS_U=22\). A restricted model setting two specified predictor coefficients to zero has \(RSS_R=34\). Under independent normal errors with common variance, test the two restrictions at 5%; use the supplied critical value \(F_{2,11}=3.98\).
Method. Use the exact nested-model F test. There are \(q=2\) restrictions and \(n-p=15-4=11\) unrestricted residual degrees of freedom.
The increase in residual sum of squares is \(34-22=12\). Its mean square is \(12/2=6\). The unrestricted residual mean square is \(22/11=2\). Thus \(F=6/2=3.00\), with degrees of freedom \((2,11)\).
Since \(3.00<3.98\), do not reject the two restrictions jointly at the 5% level. This conclusion is conditional on the stated model assumptions. A common mistake is to divide by the restricted model's residual degrees of freedom; the denominator uses the unrestricted model's \(n-p=11\) degrees of freedom.
Large-Sample Inference Without Exact Normality
When the errors are not normal, exact finite-sample t and F claims usually do not follow. For sufficiently large samples, however, a central limit effect can make suitably standardized coefficient estimates approximately normal. This requires regularity conditions, such as independent observations or an appropriate weak-dependence structure, finite variances, a correctly specified conditional mean, and a design in which no small number of observations dominates the estimate. Large \(n\) alone is not a guarantee.
If errors are independent, have conditional mean zero, and have a common variance, the usual least-squares standard errors can often support asymptotic normal inference even when the error distribution is nonnormal. If the error variance changes with the predictors, the usual homoskedastic standard-error formula may be wrong. A heteroskedasticity-robust covariance estimate is then used, subject to the same large-sample qualifications. The resulting tests and intervals are approximations, not exact finite-sample t or F procedures.
Worked Example: Robust Large-Sample Slope Inference
In a hypothetical study of 500 accounts, a regression estimates the change in annual cost, measured in thousands of dollars, for each one-point increase in a fixed risk score. The estimated slope is \(0.84\). The error distribution is not assumed normal, and the variance appears to change across risk scores. A heteroskedasticity-robust standard error of \(0.30\) is supplied. Use a large-sample normal approximation to test \(H_0:\beta_1=0\) against a two-sided alternative and form a 95% interval. Use \(z_{0.975}=1.96\) and \(\Phi(2.80)\approx0.99744\).
Method. Because unequal variances are allowed, use the supplied robust standard error rather than the classical homoskedastic one. Treat the resulting Wald statistic as approximately standard normal, not as an exact finite-sample t statistic.
The standardized statistic is \(z=0.84/0.30=2.80\). The two-sided approximate p-value is \(2[1-\Phi(2.80)]\approx2(1-0.99744)=2(0.00256)=0.00512\), or 0.0051 to four decimal places. Since this is below 0.05, reject the null using the stated approximation.
The 95% interval is \(0.84\pm1.96(0.30)=0.84\pm0.588\), giving \((0.252,1.428)\) thousand dollars per score point. The interval excludes zero, consistent with the test. A common mistake is to use a normal approximation with the ordinary homoskedastic standard error despite the stated unequal variances. Large sample size does not repair a standard error that estimates the wrong uncertainty.
Common Traps and Practical Meaning
Exactness is a property of the model and the statistic together. It is not established by a large sample, a visually plausible histogram of residuals, or the fact that regression software prints a t or F value. Conversely, nonnormal errors do not prevent calculation of least-squares coefficients; they change what can be claimed about finite-sample inference.
- Do not confuse exact with approximate. Under the classical normal-error assumptions, the stated t and F distributions hold at each permitted sample size. Under asymptotic conditions without normality, they are approximations that improve as the sample grows.
- Do not treat normality as a substitute for equal variance or independence. The exact classical results require all of the stated error assumptions. Normal but correlated or heteroskedastic errors do not satisfy \(\varepsilon\sim N(0,\sigma^2I_n)\).
- Do not assume robust standard errors create exact tests. Their role is to estimate uncertainty appropriately under variance differences; the resulting large-sample reference distribution remains approximate.
- Do not infer normal errors solely from a small residual sample. Residuals are fitted-model quantities, not the unobserved errors themselves, and a small sample gives limited information about the error distribution.
For actuarial work, this distinction affects how much confidence to place in reported significance when datasets are limited, errors are skewed, or variability changes across risks. State which assumptions justify the inference, identify whether the reference distribution is exact or asymptotic, and use a standard error appropriate to the variance structure.
Check Your Understanding
For each problem, identify the applicable inference and carry out the requested calculation.
- A simple regression with an intercept has \(n=9\), \(S_{xx}=49\), residual mean square \(s^2=4\), and estimated slope \(1.2\). Assuming independent normal errors with common variance, calculate the slope standard error and t statistic for testing a zero slope, and state the degrees of freedom. (Answer: standard error \(=\sqrt{4/49}=0.2857\); \(t=4.20\) with 7 degrees of freedom.)
- A nested regression has \(n=20\), \(p=3\), \(q=2\), \(RSS_R=50\), and \(RSS_U=38\). Under the classical normal-error model, calculate the joint-test F statistic and its degrees of freedom. (Answer: \(F=(12/2)/(38/17)=2.6842\), with degrees of freedom \((2,17)\).)
- A large-sample regression estimates a coefficient as \(0.52\), with a supplied heteroskedasticity-robust standard error of \(0.20\). Using a normal approximation, calculate the two-sided statistic and p-value; use \(\Phi(2.60)=0.99534\). (Answer: \(z=2.60\); approximate p-value \(=2(1-0.99534)=0.0093\).)
- Using the estimate and robust standard error in Question 3, form a 95% large-sample interval with critical value 1.96. (Answer: \(0.52\pm1.96(0.20)=0.52\pm0.392\), so the interval is \((0.128,0.912)\).)
- A regression has a very large sample, but its errors are heteroskedastic and nonnormal. Explain whether the ordinary finite-sample t distribution is exact, and identify an appropriate approach to a one-coefficient test. (Answer: the ordinary finite-sample t distribution is not justified as exact; use a heteroskedasticity-robust standard error with a large-sample normal/Wald approximation, subject to appropriate regularity conditions.)