Exact Profile Intervals for Gaussian Regression Slopes

Why Gaussian Regression Is a Special Case

For many models, a profile likelihood interval has no simple closed form and its endpoints are found by tracing a curve. Gaussian linear regression is unusually tractable: the profile likelihood for a slope can be written directly in terms of the usual regression \(t\) statistic. With the appropriate finite-sample calibration, the profile interval is exactly the familiar \(t\)-based interval—not merely a large-sample approximation.

The calibration matters. The profile likelihood-ratio statistic is a nonlinear function of the squared \(t\) statistic. Using a chi-square cutoff for that likelihood ratio is generally only an asymptotic procedure; it does not, in finite samples, give the usual exact \(t\) interval. The distinction follows from the model and can be derived explicitly.

Model and Profile Likelihood

The likelihood, as a function of \(\beta\) and \(\sigma^2\), is proportional to \((\sigma^2)^{-n/2}\exp\{-\|Y-X\beta\|^2/(2\sigma^2)\}\). For any fixed coefficient constraint \(c^\mathsf{T}\beta=b\), let \(\operatorname{RSS}(b)\) be the smallest residual sum of squares among coefficient vectors satisfying that constraint. Maximizing the likelihood over the remaining coefficients and over \(\sigma^2\) defines the profile likelihood for \(b\).

For a positive residual sum of squares \(R\), the likelihood at that residual sum of squares is maximized over \(\sigma^2\) at \(\hat\sigma^2=R/n\). Substitution shows that the constrained and unrestricted profile likelihoods have ratio \[ \frac{L_{\mathrm{prof}}(b)}{L_{\mathrm{prof}}(\hat b)} =\left(\frac{\operatorname{RSS}}{\operatorname{RSS}(b)}\right)^{n/2}, \qquad \hat b=c^\mathsf{T}\hat\beta. \] Thus the profile likelihood-ratio statistic is \(2\{\ell_{\mathrm{prof}}(\hat b)-\ell_{\mathrm{prof}}(b)\}=n\log\{\operatorname{RSS}(b)/\operatorname{RSS}\}\).

Constrained Residual Sum of Squares

To verify the result, write \(\delta=b-c^\mathsf{T}\hat\beta\). Among coefficient changes \(u=\beta-\hat\beta\) satisfying \(c^\mathsf{T}u=\delta\), the extra squared residual sum is \(u^\mathsf{T}X^\mathsf{T}Xu\). The minimizing change is \[ u_*=\frac{\delta}{v}(X^\mathsf{T}X)^{-1}c, \] which satisfies \(c^\mathsf{T}u_*=\delta\). For any other feasible \(u=u_*+w\), \(c^\mathsf{T}w=0\), and \[ u^\mathsf{T}X^\mathsf{T}Xu =u_*^\mathsf{T}X^\mathsf{T}Xu_*+2w^\mathsf{T}X^\mathsf{T}Xu_*+w^\mathsf{T}X^\mathsf{T}Xw. \] The middle term is zero because \(X^\mathsf{T}Xu_*=(\delta/v)c\) and \(c^\mathsf{T}w=0\). The final term is nonnegative, so \(u_*\) minimizes the extra sum of squares. Finally, \(u_*^\mathsf{T}X^\mathsf{T}Xu_*=\delta^2/v\). The ordinary least-squares residual is orthogonal to every column of \(X\), so the squared residual at \(\hat\beta+u_*\) is \(\operatorname{RSS}+u_*^\mathsf{T}X^\mathsf{T}Xu_*\), proving the result. \(\square\)

The variance of the estimated contrast, apart from the unknown factor \(\sigma^2\), is \(v\). The usual estimated standard error is therefore \(s\sqrt v\), where \(s^2=\operatorname{RSS}/\nu\). Substituting the constrained-sum-of-squares identity into the profile likelihood ratio gives a direct link to the regression statistic.

Exact Calibration Gives the Usual \(t\) Interval

For the statistic identity, the residual-sum-of-squares formula gives \[ \frac{\operatorname{RSS}(b)}{\operatorname{RSS}} =1+\frac{(b-c^\mathsf{T}\hat\beta)^2}{v\,\operatorname{RSS}} =1+\frac{T(b)^2}{\nu}, \] because \(s^2=\operatorname{RSS}/\nu\). Taking \(n\) times the logarithm proves the stated expression for \(W(b)\).

For the exact distribution, under the null, \(c^\mathsf{T}\hat\beta-b\) is normal with mean zero and variance \(\sigma^2v\), since \(\hat\beta\) is a linear function of the Gaussian response. Also, \(\operatorname{RSS}/\sigma^2\) has a chi-square distribution with \(\nu\) degrees of freedom: it is the squared length of the response projected onto the residual space, whose dimension is \(n-p\). The contrast \(c^\mathsf{T}\hat\beta-b\) depends on the projection onto the column space of \(X\), whereas the residual sum of squares depends on the orthogonal residual projection. These projections are independent for a multivariate normal vector. Thus the standardized numerator is an independent standard normal variable, and the denominator \(s/\sigma=\sqrt{(\operatorname{RSS}/\sigma^2)/\nu}\) is the square root of an independent chi-square variable divided by its degrees of freedom. Their ratio has a \(t_\nu\) distribution.

It follows that the exact two-sided acceptance condition is \(|T(b)|\leq t_{1-\alpha/2,\nu}\). Since \(W(b)\) is an increasing function of \(T(b)^2\), this condition is equivalent to \[ W(b)\leq n\log\!\left(1+\frac{t_{1-\alpha/2,\nu}^2}{\nu}\right). \] Solving the condition on \(|T(b)|\) for \(b\) yields the displayed interval. Its coverage is exactly \(1-\alpha\) under the model, because the \(t\) pivot has that exact distribution. \(\square\)

Why a Chi-Square Cutoff Is Different

A common profile-likelihood convention uses the asymptotic \(\chi^2_1\) cutoff \(3.841\) for a 95% interval. In this regression model, that cutoff is not the exact finite-sample calibration above. For the six-observation example, setting \(W(b)=3.841\) gives \[ \frac{\operatorname{RSS}(b)}{\operatorname{RSS}} =\exp(3.841/6)\approx1.897, \qquad T(b)^2=4(1.897-1)\approx3.588. \] Thus \(|T(b)|\approx1.894\), producing the interval \(3\pm1.894(0.4140)\), or approximately \((2.216,3.784)\). This is much narrower than the exact \(t\) interval because \(1.894\) is less than the exact \(t_4\) critical value \(2.776\).

The distinction is not a contradiction. The chi-square rule is an asymptotic likelihood-ratio calibration; the \(t\) rule uses the exact finite-sample distribution specific to Gaussian regression with unknown variance. The two cutoffs approach one another in large samples, but they need not agree when the residual degrees of freedom are small.

Interpretation and a Common Pitfall

In this setting, profiling the variance and nuisance regression coefficients does not require a numerical search for the interval endpoints. The constrained residual sum of squares is quadratic in the slope displacement, and the exact profile likelihood-ratio statistic is a monotone transformation of the squared \(t\) statistic. That structure makes the exact interval available in closed form.

• Wrong reasoning: “A profile likelihood interval uses a likelihood-ratio statistic, so the exact cutoff must be the 95th percentile of \(\chi^2_1\).”
• Correction: A chi-square cutoff is an asymptotic calibration. Under Gaussian regression with unknown variance, the finite-sample statistic is \(n\log(1+T^2/\nu)\), where \(T\sim t_\nu\) under the null. Transform the exact \(t_\nu\) cutoff through this expression to obtain the exact profile cutoff.

This result is useful as a check on profile-likelihood calculations and as a reminder that the likelihood curve alone does not determine a confidence interval: the calibration rule matters. In Gaussian regression, exact \(t\) calibration recovers the conventional interval while retaining its profile-likelihood interpretation.