Integrated Inference and ANOVA Problems

Putting Regression Inference Together

Regression questions often combine several calculations that use the same fitted model but answer different questions. A coefficient test concerns one model parameter; a linear contrast concerns a specified combination of parameters; an ANOVA test compares explained variation with residual variation; and an interval at a predictor setting may describe either a mean response or a new observation. The calculations are connected, but their targets and standard errors are not interchangeable.

The methods below assume the classical multiple linear regression model \(Y=X\beta+\varepsilon\), where \(X\) is a fixed design matrix with full column rank, \(E[\varepsilon]=0\), \(\operatorname{Var}(\varepsilon)=\sigma^2 I\), and the errors are normally distributed. Thus, conditional on \(X\), the errors are independent \(N(0,\sigma^2)\). These assumptions support the exact \(t\)- and \(F\)-inference used here. If they do not hold, the usual exact results need not apply. Let \(n\) be the number of observations and \(p\) the number of estimated regression coefficients, including the intercept. The residual degrees of freedom are \(n-p\).

Definition: A coefficient test evaluates a hypothesis about one component of \(\beta\). A linear contrast is a specified linear combination \(c^\mathsf{T}\beta\). An overall regression \(F\) test evaluates whether all slope coefficients are zero, while a partial \(F\) test evaluates whether a specified group of coefficients can be set to zero when comparing nested models. A mean-response interval concerns the conditional mean at a predictor setting; a prediction interval concerns one new response at that setting.

Coefficient Inference and Linear Contrasts

Under the stated model assumptions, the least-squares estimator is \(\hat\beta\), and the error variance is estimated by \(\operatorname{MSE}=\operatorname{SSE}/(n-p)\). For a coefficient estimate \(\hat\beta_j\) with standard error \(SE(\hat\beta_j)\), a two-sided test of \(H_0:\beta_j=0\) uses \(t=\hat\beta_j/SE(\hat\beta_j)\), with \(n-p\) degrees of freedom. A confidence interval uses the same standard error and degrees of freedom.

\hat\beta_j \pm t_{1-\alpha/2,n-p}SE(\hat\beta_j)

For a contrast \(c^\mathsf{T}\beta\), the estimate is \(c^\mathsf{T}\hat\beta\). Its estimated variance must account for the covariances among coefficient estimates:

\widehat{\operatorname{Var}}(c^\mathsf{T}\hat\beta) =c^\mathsf{T}\widehat{\operatorname{Cov}}(\hat\beta)c

The corresponding test divides \(c^\mathsf{T}\hat\beta\) minus its null value by the square root of this variance estimate. The confidence interval is the estimate plus or minus the \(t\) critical value times that standard error.

Worked Example: A Coefficient Test and a Contrast

A hypothetical severity model has three coefficients, including the intercept, and \(n=23\) observations. Its coefficient estimates for two predictors are \(\hat\beta_1=2.4\) and \(\hat\beta_2=1.1\). Their standard errors are 0.8 and 0.5, and their estimated covariance is 0.10. Use \(t_{0.975,20}=2.086\). Find 95% intervals for each coefficient and for \(\beta_1-2\beta_2\), and test \(H_0:\beta_1=0\) at the 5% level.

Method and degrees of freedom. There are \(23-3=20\) residual degrees of freedom. The supplied critical value therefore applies to each two-sided 95% interval and the 5% coefficient test.

Coefficient intervals. For \(\beta_1\), the margin is \(2.086(0.8)=1.6688\), giving \(2.4\pm1.6688=(0.7312,4.0688)\). For \(\beta_2\), the margin is \(2.086(0.5)=1.043\), giving \(1.1\pm1.043=(0.057,2.143)\).

Test of \(\beta_1=0\). The test statistic is \(2.4/0.8=3.00\). Since \(|3.00|>2.086\), reject the null hypothesis at the 5% level. This agrees with the 95% interval for \(\beta_1\), which excludes zero.

Contrast interval. The contrast estimate is \(2.4-2(1.1)=0.2\). Its estimated variance is \(0.8^2+(-2)^2(0.5^2)+2(1)(-2)(0.10)=0.64+1.00-0.40=1.24\). Therefore its standard error is \(\sqrt{1.24}=1.1136\) to four decimals. The margin is \(2.086\sqrt{1.24}=2.3229\) to four decimals, so the 95% interval is \((-2.1229,2.5229)\), rounded to four decimals.

Check and trap avoided. The covariance contribution is negative because the contrast multiplies the second coefficient by \(-2\). Omitting that contribution would give variance \(1.64\), not \(1.24\), and an incorrect interval. The contrast interval includes zero, which is consistent with the contrast test not rejecting a zero value.

ANOVA and Nested-Model Comparisons

For a regression with an intercept, the total sum of squares is split into regression and error sums of squares: \(\operatorname{SST}=\operatorname{SSR}+\operatorname{SSE}\). With \(k\) slope predictors, the regression degrees of freedom are \(k\), the error degrees of freedom are \(n-k-1\), and the total degrees of freedom are \(n-1\). Mean squares are sums of squares divided by their corresponding degrees of freedom. The overall \(F\) statistic is \(\operatorname{MSR}/\operatorname{MSE}\), testing that all \(k\) slopes are zero.

For nested models, the reduced model is contained in the full model: every fit available to the reduced model is also available to the full model. If the full model adds \(q\) coefficients, the partial \(F\) statistic compares the reduction in SSE with the full-model estimate of error variance:

F=\frac{(\operatorname{SSE}_{R}-\operatorname{SSE}_{F})/q} {\operatorname{SSE}_{F}/(n-p_F)}

Here \(p_F\) is the number of coefficients in the full model, including the intercept. Under the null hypothesis that all \(q\) added coefficients are zero, the statistic has an \(F\) distribution with \(q\) and \(n-p_F\) degrees of freedom, under the classical regression assumptions. A large value is evidence against that joint null hypothesis.

Worked Example: Overall ANOVA and a Nested-Model Test

A hypothetical portfolio study uses \(n=20\) observations. A full model has three slope predictors and an intercept, with \(\operatorname{SST}=300\) and \(\operatorname{SSE}_F=120\). A reduced model retaining one of the three predictors has \(\operatorname{SSE}_R=180\). At the 5% level, use the supplied critical values \(F_{0.95;3,16}=3.24\) and \(F_{0.95;2,16}=3.63\). Construct the full-model ANOVA calculations, test overall regression significance, and compare the models.

Full-model ANOVA. The regression sum of squares is \(300-120=180\). The total degrees of freedom are \(20-1=19\); regression degrees of freedom are 3; and residual degrees of freedom are \(20-4=16\). Thus \(\operatorname{MSR}=180/3=60\) and \(\operatorname{MSE}=120/16=7.5\). The overall statistic is \(60/7.5=8.00\). Since \(8.00>3.24\), reject the null that all three slopes are zero.

Nested comparison. The full model adds two slopes, so \(q=2\). The reduction in SSE is \(180-120=60\), and the partial statistic is \((60/2)/7.5=30/7.5=4.00\). Since \(4.00>3.63\), reject the joint null that both added slopes are zero, conditional on the retained predictor.

Check and trap avoided. The full model has four coefficients including its intercept, so its residual degrees of freedom are 16. The nested comparison uses the same full-model MSE and uses \(q=2\) in the numerator degrees of freedom. Using the reduced model's residual degrees of freedom or dividing the SSE reduction by three would answer a different, incorrect calculation.

Why the Interval Formulas Differ

At a predictor setting \(x_0\), let \(Y_0\) denote a new response and let \(\hat{Y}_0\) be the fitted value. The fitted conditional mean is \(E[Y_0\mid x_0]\). Its estimation uncertainty comes from estimating the regression coefficients. A new observation has that same coefficient-estimation uncertainty plus its own random error with variance \(\sigma^2\).

This gives a direct justification for the distinction. Write the prediction error as \(Y_0-\hat{Y}_0=(E[Y_0\mid x_0]-\hat{Y}_0)+\varepsilon_0\). Under the model, the new error \(\varepsilon_0\) has mean zero, variance \(\sigma^2\), and is independent of the coefficient estimates based on the existing observations. Therefore, the two variance contributions add. For simple linear regression, with \(S_{xx}=\sum_i(x_i-\bar{x})^2\), the estimated standard errors are:

SE_{\text{mean}}=\sqrt{\operatorname{MSE}\left(\frac{1}{n}+ \frac{(x_0-\bar{x})^2}{S_{xx}}\right)}

SE_{\text{prediction}}=\sqrt{\operatorname{MSE}\left(1+\frac{1}{n}+ \frac{(x_0-\bar{x})^2}{S_{xx}}\right)}

Both intervals use the residual degrees of freedom and a \(t\) critical value. The extra 1 inside the prediction standard error represents the variance of the new response's error. It is not included for a conditional mean.

Worked Example: Mean-Response and Prediction Intervals at the Center

A hypothetical simple regression uses \(n=16\) observations, \(\bar{x}=8\), and \(\operatorname{MSE}=9\). At \(x_0=8\), the fitted mean is 24. Since \(x_0=\bar{x}\), the leverage contribution \((x_0-\bar{x})^2/S_{xx}\) is zero. Use \(t_{0.975,14}=2.145\). Calculate 95% intervals for the mean response and a single new response.

Mean response. Its standard error is \(\sqrt{9(1/16)}=\sqrt{9/16}=0.75\). The margin is \(2.145(0.75)=1.60875\). Thus the interval is \(24\pm1.60875=(22.39125,25.60875)\), or \((22.3913,25.6088)\) rounded to four decimals.

New response. Its standard error is \(\sqrt{9(1+1/16)}=\sqrt{9(17/16)}=\sqrt{153/16}=\sqrt{153}/4=3.092329\ldots\). The margin is \(2.145(3.092329\ldots)=6.633046\ldots\). The prediction interval is \((24-6.633046\ldots,24+6.633046\ldots)\), or \((17.367,30.633)\) rounded to three decimals.

Check and trap avoided. The prediction interval is wider because it includes new-response variability. The exact prediction standard error is \(\sqrt{153}/4\), not an approximation obtained by rounding the expression inside the square root prematurely. Using the mean-response standard error for a new observation would substantially understate the interval's width.

A Reliable Order for Integrated Problems

When a problem mixes inference tasks, identify each target before calculating. A coefficient, a contrast, an overall set of slopes, a group of added slopes, a conditional mean, and a new response are not the same target. The following order helps keep the calculations separate and makes a final consistency check possible.

Record the model size and assumptions.
Count all fitted coefficients, including the intercept, to obtain residual degrees of freedom. Check that the stated regression assumptions support the requested exact inference.

Name the hypothesis or quantity.
For a coefficient or contrast, identify its null value. For a nested comparison, identify the added coefficients. For an interval, decide whether its target is a mean or an individual response.

Use the matching uncertainty calculation.
Include covariance terms for a contrast, use full-model MSE for a partial \(F\) test, and include the extra unit variance term for a prediction interval.

Check interpretation and arithmetic.
Compare a two-sided coefficient test with its matching confidence interval. Confirm that a prediction interval is wider than the corresponding mean interval and that the nested-model numerator degrees of freedom equal the number of added coefficients.

Common Exam Traps

Mixing up the target. A confidence interval for a mean response is not a prediction interval for a new observation; only the latter includes the new observation's error variance.
Dropping covariance in a contrast. The variance of a weighted sum includes cross-products. A negative contrast weight can make the covariance contribution negative.
Using the wrong degrees of freedom. Coefficient tests and full-model MSE use \(n-p\), while a partial \(F\) numerator uses the number of added coefficients.
Using the reduced-model MSE in the partial test. The standard partial \(F\) calculation uses the full model's residual mean square in its denominator.
Confusing overall and partial tests. The overall test asks whether all slopes are zero; a partial test asks whether the specified added slopes are jointly zero given the retained predictors.

These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The central discipline is to match each formula to its target and assumptions, then check whether the result tells a coherent story: a test and its corresponding interval should agree, a new-response interval should be wider than a mean interval, and a nested test should compare only the coefficients actually added.

Check Your Understanding

For each problem, identify the inference target before calculating. Values are hypothetical, and any critical values needed are supplied.

A regression has \(n=22\), three fitted coefficients including the intercept, and a slope estimate of 1.8 with standard error 0.6. Use \(t_{0.975,19}=2.093\). Find its 95% confidence interval and test a zero slope at the 5% level. (Answer: interval \((0.5442,3.0558)\); \(t=3.00\), so reject a zero slope.)
Two coefficient estimates are \(\hat\beta_1=0.7\) and \(\hat\beta_2=1.2\), with variances 0.09 and 0.04 and covariance \(-0.018\). For \(3\beta_1+\beta_2\), calculate the estimate and standard error. Use \(t_{0.975,25}=2.060\) for a 95% interval. (Answer: estimate \(3.3\), variance \(0.742\), standard error \(\sqrt{0.742}=0.8614\), interval approximately \((1.5255,5.0745)\).)
A full regression with \(n=24\) has four slopes plus an intercept and \(\operatorname{SSE}_F=180\). A nested reduced model has two slopes plus an intercept and \(\operatorname{SSE}_R=240\). Test the two added slopes using \(F_{0.95;2,19}=3.52\). (Answer: \(F=(60/2)/(180/19)=3.1667\); do not reject at 5%.)
In a simple regression, \(n=10\), \(\bar{x}=5\), \(S_{xx}=20\), \(\operatorname{MSE}=4\), and the fitted value at \(x_0=7\) is 12. Use \(t_{0.975,8}=2.306\). Find the 95% mean-response and prediction intervals, rounding endpoints to three decimals. (Answer: standard errors \(\sqrt{0.8}=0.8944\) and \(\sqrt{5.2}=2.2804\); intervals \((9.937,14.063)\) and \((6.742,17.258)\), respectively.)
A regression with \(n=15\) has two slopes, \(\operatorname{SST}=500\), and \(\operatorname{SSR}=300\). Find \(\operatorname{SSE}\), the overall \(F\) statistic, and \(R^2\). Use \(F_{0.95;2,12}=3.89\) to make the 5% decision. (Answer: \(\operatorname{SSE}=200\), \(\operatorname{MSE}=200/12\), \(\operatorname{MSR}=150\), \(F=9.00\), \(R^2=0.60\); reject the overall null.)

What You'll Learn