Regression Inference Requires the Right Reference Distribution
A fitted regression line describes an estimated relationship. Inference asks how much uncertainty remains about the population coefficients and whether the data are compatible with particular coefficient values. The calculations are compact, but their interpretation depends on using the right standard error, degrees of freedom, and alternative hypothesis.
For a simple linear regression with a fixed predictor, write the model as \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), for \(i=1,\ldots,n\). For the exact small-sample t and F procedures in this tutorial, assume the errors are independent with \(\varepsilon_i\sim N(0,\sigma^2)\), with a common variance, and that the predictor values are fixed. The fitted coefficients are \(b_0\) and \(b_1\); \(s^2=\mathrm{SSE}/(n-2)\) estimates \(\sigma^2\), where SSE is the residual sum of squares. The residual degrees of freedom are \(n-2\), because two coefficients have been estimated.
For multiple regression with \(k\) predictors and an intercept, the same normal, independent, equal-variance error assumptions give residual degrees of freedom \(n-k-1\). A coefficient t test uses that residual degrees of freedom. The overall F test instead compares the model with all \(k\) slopes set to zero against the model with those slopes unrestricted.
Why the t Statistic Has Its Reference Distribution
The t procedure is not a rule to apply to any estimate divided by any standard error. Under the simple-regression model, normal errors imply that the slope estimate is normally distributed around its true value, with variance \(\sigma^2/S_{xx}\). The residual variance supplies an independent estimate of \(\sigma^2\). These two facts produce the t distribution after standardization.
Under the model, \(Z=(b_1-\beta_1)/(\sigma/\sqrt{S_{xx}})\) has a standard normal distribution.
The residual sum of squares satisfies \(\mathrm{SSE}/\sigma^2\sim\chi^2_{n-2}\), and it is independent of \(b_1\). Thus \(s^2/\sigma^2=(\mathrm{SSE}/\sigma^2)/(n-2)\).
A standard normal variable divided by the square root of an independent chi-square variable divided by its degrees of freedom has a t distribution. Therefore, under a hypothesized value \(\beta_{1,0}\), \((b_1-\beta_{1,0})/(s/\sqrt{S_{xx}})\sim t_{n-2}\).
Inverting this two-sided test gives the \(100(1-\alpha)\%\) confidence interval \(b_1\pm t_{1-\alpha/2,n-2}s/\sqrt{S_{xx}}\). A one-sided test uses a one-sided critical value instead; it does not use the two-sided critical value with the same confidence level.
Worked Example: Slope and Intercept Inference in a Simple Regression
A hypothetical insurer studies the relationship between a centered risk score \(x\), measured in score points, and annual claim cost \(Y\), measured in thousands of dollars. For \(n=10\) independent observations, assume the fixed-predictor normal regression model described above. The fitted output is \(b_0=5\), \(b_1=2.4\), \(s=2\), and \(S_{xx}=6.25\). Construct 95% confidence intervals for both coefficients and test \(H_0:\beta_1=0\) against a two-sided alternative. Use \(t_{0.975,8}=2.306\).
Method and degrees of freedom. Two coefficients are estimated, so the residual degrees of freedom are \(10-2=8\). The slope standard error is \(s/\sqrt{S_{xx}}=2/\sqrt{6.25}=2/2.5=0.8\). Since the predictor is centered, \(\bar{x}=0\), and the intercept standard error is \(s\sqrt{1/n}=2\sqrt{0.1}=0.6324555\ldots\).
Slope interval. The margin of error is \(2.306(0.8)=1.8448\). The interval is \(2.4\pm1.8448\), or \([0.5552,4.2448]\) thousand dollars per score point. The data are compatible, at the 95% confidence level, with a positive slope as well as other values in this interval.
Intercept interval. Carry the unrounded standard error through the calculation: \(2.306(2\sqrt{0.1})=1.458442\ldots\). Thus \(5\pm1.458442\ldots=[3.5416,6.4584]\) to four decimal places. The intercept is the mean response at \(x=0\), because \(x\) was centered.
Slope test. Under \(H_0:\beta_1=0\), the statistic is \(t=(2.4-0)/0.8=3.000\), with 8 degrees of freedom. Since \(|3.000|>2.306\), reject the null at the 5% two-sided level. The two-sided p-value is approximately 0.0171. This is evidence against a zero population slope under the stated model; it is not the probability that the null hypothesis is true.
Check and common mistake avoided. The slope interval excludes zero, agreeing with the two-sided test at the 5% level. Using \(n-1=9\) degrees of freedom would be wrong: the residual variance is estimated after fitting both intercept and slope, leaving \(n-2=8\).
The Overall F Test in Multiple Regression
For a regression with \(k\) predictors and an intercept, the overall F test has null hypothesis \(H_0:\beta_1=\cdots=\beta_k=0\). Its alternative is that at least one slope is nonzero. Let SSR be the regression sum of squares and SSE the residual sum of squares. The statistic is \(F=(\mathrm{SSR}/k)/(\mathrm{SSE}/(n-k-1))\), with numerator degrees of freedom \(k\) and denominator degrees of freedom \(n-k-1\). Under the null and the model assumptions, it has an \(F_{k,n-k-1}\) distribution.
The numerator measures explained variation per predictor degree of freedom; the denominator measures residual variation per residual degree of freedom. A large F statistic is evidence against the joint null. Rejecting it does not establish that every predictor is useful, nor does it by itself identify which slope differs from zero.
Worked Example: Calculating the Overall F Test
A hypothetical health study fits an intercept and \(k=2\) predictors to \(n=15\) observations. Assume independent normal errors with constant variance and a correctly specified linear mean. The fitted model has \(\mathrm{SSR}=80\) and \(\mathrm{SSE}=52\). Test whether both population slopes are zero at the 5% level. Use the critical value \(F_{0.95;2,12}=3.885\).
Set the degrees of freedom. The numerator degrees of freedom are \(k=2\). The denominator degrees of freedom are \(n-k-1=15-2-1=12\).
Calculate the mean squares. The regression mean square is \(\mathrm{MSR}=80/2=40\). The residual mean square is \(\mathrm{MSE}=52/12=13/3=4.3333\ldots\).
Calculate and compare. The observed statistic is \(F=40/(13/3)=120/13=9.2308\). Since \(9.2308>3.885\), reject the joint null at the 5% level. The data provide evidence that at least one of the two population slopes is nonzero.
Check and common mistake avoided. The denominator uses 12 residual degrees of freedom, not \(n-1=14\) and not \(n-k=13\). The latter omits the fitted intercept. The test is joint: it does not say that both slopes are individually significant.
Why the Simple-Regression F Test Equals the Slope t Test
The overall F test has a useful special case. In simple regression, there is just one slope, so the overall null is \(H_0:\beta_1=0\). The regression sum of squares is \(b_1^2S_{xx}\), while \(\mathrm{MSE}=s^2\). Consequently,
Thus the simple-regression overall F test and the two-sided slope t test produce the same p-value and decision. This identity applies to testing a zero slope in simple regression with an intercept; it does not make a multiple-regression overall F test equivalent to one individual coefficient t test.
Worked Example: Checking the Simple-Regression F Result
Use the slope test from the first example, where \(t=3.000\) on 8 residual degrees of freedom. Verify the corresponding overall F statistic and decision. The model has one predictor and satisfies the stated regression assumptions.
Square the t statistic. \(F=t^2=3.000^2=9.000\), with numerator degrees of freedom 1 and denominator degrees of freedom 8.
Compare the tests. The 5% critical value is \(F_{0.95;1,8}=5.318\). Because \(9.000>5.318\), the overall F test rejects the zero-slope null, just as the two-sided t test did. The p-value is approximately 0.0171 for both tests.
Check and common mistake avoided. Squaring the t statistic corresponds to a two-sided slope test. A one-sided slope test does not have the same p-value as this overall F test.
Inference for a Population Correlation
For independent observations from a bivariate normal population, a test of \(H_0:\rho=0\) can use the sample correlation \(r\). The test statistic is \(t=r\sqrt{(n-2)/(1-r^2)}\), with \(n-2\) degrees of freedom. This is the same test as the simple-regression slope test when the data are viewed as a response and predictor. The exact t result requires the stated bivariate-normal model; it should not be treated as exact for arbitrary dependent or strongly nonnormal paired data.
For inference about a nonzero correlation, Fisher’s transformation is often used. Set \(z_r=\tfrac12\ln((1+r)/(1-r))\). For a bivariate normal population, \(z_r\) is approximately normal with standard error \(1/\sqrt{n-3}\), particularly as the sample size increases. A confidence interval on the transformed scale is \(z_r\pm z_{1-\alpha/2}/\sqrt{n-3}\); apply the inverse transformation \(\tanh\) to both endpoints to obtain an approximate interval for \(\rho\). This interval is approximate, not an exact small-sample t interval.
Worked Example: Testing and Estimating a Population Correlation
In a hypothetical study, \(n=25\) independent pairs of measurements are sampled from a bivariate normal population, and the sample correlation is \(r=0.60\). Test \(H_0:\rho=0\) against a two-sided alternative at the 5% level, then construct an approximate 95% Fisher-transformation interval. Use \(t_{0.975,23}=2.069\) and \(z_{0.975}=1.96\).
Test the zero-correlation null. Here \(r^2=0.36\), \(1-r^2=0.64\), and \(n-2=23\). Therefore \(t=0.60\sqrt{23/0.64}=0.60\sqrt{35.9375}=3.5969\) to four decimal places, with 23 degrees of freedom. Since \(3.5969>2.069\), reject \(H_0\) at the 5% level. The data support a nonzero population correlation under the model.
Construct the Fisher interval. The transformed estimate is \(z_r=\tfrac12\ln(1.60/0.40)=\tfrac12\ln(4)=0.693147\ldots\). The standard error is \(1/\sqrt{25-3}=1/\sqrt{22}\). The margin is \(1.96/\sqrt{22}=0.417874\ldots\). Carrying the unrounded values gives transformed endpoints approximately \(0.275274\) and \(1.111021\).
Transform back. Applying \(\tanh\) to the two endpoints gives approximately \(0.2685\) and \(0.8044\). The approximate 95% interval for \(\rho\) is therefore \([0.2685,0.8044]\), rounded to four decimal places. It excludes zero, consistent with the two-sided test.
Check and common mistake avoided. The transformation produces an interval on the \(z\) scale first; its endpoints are not themselves correlation limits. Transforming both endpoints back is essential. The test uses \(n-2=23\) degrees of freedom, while the approximate Fisher standard error uses \(n-3=22\).
Common Interpretation Errors
A p-value is calculated under the null hypothesis: it is the probability, assuming that null and the model, of observing a test statistic at least as extreme as the one obtained. For a two-sided t test, extremeness includes both tails, so the p-value is \(2P(T_\nu\geq |t_{\mathrm{obs}}|)\). The p-value is not the probability that the null is true, and it is not the probability that the observed result occurred “by chance” without specifying a model and null.
For example, if \(t_{\mathrm{obs}}=6\) with 8 degrees of freedom, the two-sided p-value is \(2P(T_8\geq6)\approx0.0003234\), not 0.001. To three decimal places this is 0.000; to four decimal places it is 0.0003. A reported p-value is not the effect size, and statistical significance does not establish practical importance or causation.
- Wrong degrees of freedom: simple-regression coefficient tests use \(n-2\); multiple-regression coefficient tests use \(n-k-1\). The overall F test has \(k\) numerator and \(n-k-1\) denominator degrees of freedom.
- Wrong tail: a two-sided alternative requires probability in both tails. A directional alternative must be specified before examining the result.
- Wrong units: a slope has response units per predictor unit. An intercept has response units and refers to the mean response at predictor value zero. Rescaling a predictor changes the numerical slope and its units.
- Overstating an F test: rejection of the overall null says at least one slope is nonzero, not that every slope is nonzero or that the model is causally correct.
- Overinterpreting an interval: a confidence interval describes the procedure’s long-run coverage under its assumptions; it is not a probability statement about a fixed coefficient after the data are observed.
Check Your Understanding
For each problem, identify the relevant reference distribution and show enough calculation to justify the conclusion.
- A simple regression has \(n=14\), estimated slope \(b_1=1.2\), and slope standard error \(0.4\). Construct a 95% interval using \(t_{0.975,12}=2.179\). (Answer: \(1.2\pm2.179(0.4)=[0.3284,2.0716]\) to four decimal places.)
- A simple regression based on \(n=11\) observations has \(b_1=-0.9\) and standard error \(0.3\). Test \(H_0:\beta_1=0\) against a two-sided alternative at 5%, using \(t_{0.975,9}=2.262\). State the degrees of freedom and decision. (Answer: \(t=-3.000\), 9 degrees of freedom; reject because \(3.000>2.262\).)
- A multiple regression with an intercept has \(n=18\), \(k=3\), \(\mathrm{SSR}=90\), and \(\mathrm{SSE}=60\). Calculate the overall F statistic and test at 5%, using \(F_{0.95;3,14}=3.344\). (Answer: \(F=(90/3)/(60/14)=7.000\), with degrees of freedom \((3,14)\); reject the joint null.)
- For \(n=27\) independent bivariate-normal pairs, the sample correlation is \(r=-0.40\). Test \(H_0:\rho=0\) against a two-sided alternative at 5%, using \(t_{0.975,25}=2.060\). (Answer: \(t=-0.40\sqrt{25/0.84}\approx-2.1822\), with 25 degrees of freedom; reject because \(|t|>2.060\).)
- A two-sided coefficient test reports \(p=0.03\). Explain what this says about the null and whether it means there is a 3% probability that the null is true. (Answer: Under the null and model, results at least as extreme have probability 0.03; it is not a 3% probability that the null is true.)