Profile Intervals for Functions of Model Parameters

Why Profile a Function of Parameters?

A scientific question often concerns a combination of model parameters rather than any one parameter on its own. In a regression model, for example, the combined effect of two predictors may be represented by the sum of their coefficients. An interval for that sum must account for how the likelihood behaves while all parameter vectors consistent with the proposed sum are considered.

This is the purpose of a profile likelihood. Instead of holding nuisance parameters at their estimates or combining separate intervals, we fix the scalar quantity of interest and maximize the likelihood over every compatible parameter vector. The resulting profile can be used to construct a likelihood-ratio confidence set.

Definition: Let $L(\theta)$ be a likelihood for a parameter vector $\theta\in\Theta$, and let $g:\Theta\to\mathbb{R}$ be a scalar function. For a proposed value $\psi$, the profile likelihood for $\psi=g(\theta)$ is $$ L_p(\psi)=\sup_{\theta\in\Theta:\,g(\theta)=\psi} L(\theta). $$ If the unconstrained maximum $\hat L=\sup_{\theta\in\Theta}L(\theta)$ exists, the profile likelihood-ratio statistic is $$ \Lambda(\psi)=2\{\ell(\hat\theta)-\ell_p(\psi)\}, $$ where $\ell_p(\psi)=\log L_p(\psi)$. A likelihood-ratio confidence set with cutoff $k$ is $\{\psi:\Lambda(\psi)\leq k\}$.

The constrained maximum is taken over the entire set $g(\theta)=\psi$, not just over parameter vectors obtained by separately estimating components of $\theta$. Under the usual regularity conditions for Wilks' theorem—an identifiable model, a true parameter in the interior, sufficient smoothness, and a regular scalar constraint—the likelihood-ratio statistic is asymptotically distributed as $\chi^2_1$ when the proposed $\psi$ is the true value. Thus an asymptotic $1-\alpha$ confidence set uses the cutoff $\chi^2_{1,1-\alpha}$. A finite-sample Gaussian model can also provide an exact cutoff, derived below.

Worked Example: Profiling the Sum of Two Normal Means

Let $Y_1\sim N(\theta_1,1)$ and $Y_2\sim N(\theta_2,1)$ independently, with observed values $y_1=1$ and $y_2=3$. The target is $\psi=\theta_1+\theta_2$. The unconstrained estimates are $\hat\theta_1=1$, $\hat\theta_2=3$, and $\hat\psi=4$.

For a fixed $\psi$, maximize the likelihood subject to $\theta_1+\theta_2=\psi$. This is equivalent to minimizing $(1-\theta_1)^2+(3-\theta_2)^2$ under that constraint. Write $\theta_2=\psi-\theta_1$. Differentiating the resulting quadratic with respect to $\theta_1$ gives the constrained estimates $\hat\theta_1(\psi)=(\psi-2)/2$ and $\hat\theta_2(\psi)=(\psi+2)/2$. The constrained residual sum of squares is $$ \left(1-\frac{\psi-2}{2}\right)^2+ \left(3-\frac{\psi+2}{2}\right)^2 =\frac{(4-\psi)^2}{2}. $$

Because the variance is known to be 1, the log-likelihood difference gives $\Lambda(\psi)=(\psi-4)^2/2$. The asymptotic 95% likelihood-ratio set, using $\chi^2_{1,0.95}=3.841$, satisfies $(\psi-4)^2/2\leq3.841$. Thus $|\psi-4|\leq\sqrt{7.682}\approx2.772$, giving the interval $(1.228,6.772)$. The constrained fit, rather than separate intervals for $\theta_1$ and $\theta_2$, directly answers the question about their sum.

Profiling a Coefficient Sum in Gaussian Regression

Consider the Gaussian linear model $Y\sim N(X\beta,\sigma^2 I_n)$, where $X$ is a fixed $n\times p$ design matrix of full column rank, $\beta\in\mathbb{R}^p$, and $\sigma^2>0$ is unknown. Let $\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y$, let $\mathrm{RSS}_0=\|Y-X\hat\beta\|^2$, and set $\nu=n-p$. Assume $\nu>0$ and $\mathrm{RSS}_0>0$. A coefficient sum is a linear target $q=c^\mathsf{T}\beta$, where $c$ is a specified nonzero vector; for example, $c=(0,1,1)^\mathsf{T}$ targets $\beta_1+\beta_2$ in a model with an intercept.

Theorem: For a fixed $q$, the constrained least-squares residual sum of squares is $$ \mathrm{RSS}(q)=\mathrm{RSS}_0+ \frac{(q-c^\mathsf{T}\hat\beta)^2}{v}, \qquad v=c^\mathsf{T}(X^\mathsf{T}X)^{-1}c. $$ After profiling out both $\beta$ subject to $c^\mathsf{T}\beta=q$ and the unknown dispersion $\sigma^2$, the likelihood-ratio statistic is $$ \Lambda(q)=n\log\left(\frac{\mathrm{RSS}(q)}{\mathrm{RSS}_0}\right) =n\log\left(1+\frac{(q-c^\mathsf{T}\hat\beta)^2}{v\,\mathrm{RSS}_0}\right). $$

Proof. Put $A=(X^\mathsf{T}X)^{-1}$, $\hat q=c^\mathsf{T}\hat\beta$, and $v=c^\mathsf{T}Ac>0$. The constrained least-squares coefficient vector is $$ \hat\beta(q)=\hat\beta+\frac{q-\hat q}{v}Ac. $$ It satisfies the constraint because $c^\mathsf{T}\hat\beta(q)=\hat q+(q-\hat q)c^\mathsf{T}Ac/v=q$. The unconstrained residual $Y-X\hat\beta$ is orthogonal to the columns of $X$. Therefore, for any coefficient vector $b$, $\|Y-Xb\|^2=\mathrm{RSS}_0+(b-\hat\beta)^\mathsf{T}X^\mathsf{T}X(b-\hat\beta)$. Substituting $\hat\beta(q)$ into the second term gives $(q-\hat q)^2c^\mathsf{T}Ac/v^2=(q-\hat q)^2/v$. This proves the formula for $\mathrm{RSS}(q)$, and the displayed constrained vector attains the minimum.

For any fixed coefficient vector with residual sum of squares $R>0$, the Gaussian log-likelihood is $-\frac n2\log(2\pi\sigma^2)-R/(2\sigma^2)$. Its maximum over $\sigma^2>0$ occurs at $\hat\sigma^2=R/n$; substituting gives the maximized log-likelihood $-\frac n2\{\log(2\pi)+1+\log(R/n)\}$. Subtract the constrained maximized log-likelihood, using $R=\mathrm{RSS}(q)$, from the unconstrained one, using $R=\mathrm{RSS}_0$, and multiply by 2. The result is $n\log\{\mathrm{RSS}(q)/\mathrm{RSS}_0\}$. Substitution of the residual-sum-of-squares formula proves the likelihood-ratio expression. $\square$

Worked Example: Deriving an Interval for an Orthogonal-Design Coefficient Sum

Suppose $n=8$, the model includes an intercept and two predictors $x_1,x_2$, and the columns $1,x_1,x_2$ are mutually orthogonal with squared lengths 8. Assume the fitted coefficients are $\hat\beta_0=2$, $\hat\beta_1=0.5$, and $\hat\beta_2=1$, with $\mathrm{RSS}_0=8$. These summaries are consistent, for example, with columns having entries $\pm1$ that are mutually orthogonal, and a residual column also orthogonal to all three with squared length 8. Here $p=3$, $\nu=8-3=5$, and the target $q=\beta_1+\beta_2$ has estimate $\hat q=1.5$.

Since the two predictor columns are orthogonal and each has squared length 8, their coefficient covariance factor is $v=1/8+1/8=1/4$. Thus $\mathrm{RSS}(q)=8+(q-1.5)^2/(1/4)=8+4(q-1.5)^2$, and $\Lambda(q)=8\log\{1+(q-1.5)^2/2\}$.

For the exact 95% interval, use $t_{5,0.975}\approx2.5706$, as justified by the finite-sample result below. The estimated variance of the contrast is $\mathrm{MSE}\,v=(8/5)(1/4)=0.4$, so its standard error is $\sqrt{0.4}\approx0.6325$. The margin is $2.5706(0.6325)\approx1.626$, giving the exact interval $(1.5-1.626,\;1.5+1.626)=(-0.126,\;3.126)$, to three decimal places.

Exact Finite-Sample Calibration

The likelihood-ratio expression alone does not determine its finite-sample cutoff. In the Gaussian model, the same contrast can be tested using an exact $F$ statistic, which supplies a likelihood-ratio cutoff with exact coverage.

Theorem: Under the fixed-design Gaussian model above, if the true coefficient vector satisfies $c^\mathsf{T}\beta=q$, then $$ F(q)=\frac{\{\mathrm{RSS}(q)-\mathrm{RSS}_0\}/1}{\mathrm{RSS}_0/\nu} $$ has an $F_{1,\nu}$ distribution. Consequently, an exact $1-\alpha$ confidence set for $q$ is given by $F(q)\leq F_{1,\nu;1-\alpha}$. Equivalently, it is the profile likelihood-ratio set with cutoff $n\log\{1+F_{1,\nu;1-\alpha}/\nu\}$, and its endpoints are $$ c^\mathsf{T}\hat\beta\ \pm\ t_{\nu,1-\alpha/2} \sqrt{\frac{\mathrm{RSS}_0}{\nu}\,c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}. $$

Proof. Under the null constraint $c^\mathsf{T}\beta=q$, the full-model residual sum of squares and the constrained-model residual sum of squares come from nested orthogonal projections. Their difference is the squared length of the projection onto a one-dimensional subspace. Under Gaussian errors, $\{\mathrm{RSS}(q)-\mathrm{RSS}_0\}/\sigma^2$ therefore has a $\chi^2_1$ distribution, while $\mathrm{RSS}_0/\sigma^2$ has a $\chi^2_\nu$ distribution. The two quantities are independent because the corresponding projection subspaces are orthogonal. Their ratio after division by their degrees of freedom is $F_{1,\nu}$, proving the asserted distribution.

From the definition of $F(q)$, $\mathrm{RSS}(q)/\mathrm{RSS}_0=1+F(q)/\nu$. Hence $\Lambda(q)=n\log\{1+F(q)/\nu\}$, a strictly increasing function of $F(q)$. The exact $F$ acceptance condition is therefore precisely the stated likelihood-ratio condition. Finally, $\mathrm{RSS}(q)-\mathrm{RSS}_0=(q-\hat q)^2/v$, so $F(q)\leq F_{1,\nu;1-\alpha}$ is equivalent to $(q-\hat q)^2\leq v(\mathrm{RSS}_0/\nu)F_{1,\nu;1-\alpha}$. Since $F_{1,\nu;1-\alpha}=t_{\nu,1-\alpha/2}^2$, taking square roots gives the endpoints. $\square$

Worked Example: Accounting for Correlated Predictors

In another Gaussian regression with an intercept and two predictors, suppose $n=8$, $\mathrm{RSS}_0=8$, $(\hat\beta_1,\hat\beta_2)=(0.5,1.5)$, and the predictor cross-product matrix is $$ X_c^\mathsf{T}X_c= \begin{pmatrix} 8&4\\ 4&8 \end{pmatrix}, $$ where $X_c$ contains the centered predictor columns. This matrix is positive definite because its determinant is $64-16=48>0$. The intercept is orthogonal to these columns, so the sum contrast uses the inverse of this displayed matrix.

The inverse is $\frac1{48}\begin{pmatrix}8&-4\\-4&8\end{pmatrix}$. For $c=(1,1)^\mathsf{T}$, $$ v=c^\mathsf{T}(X_c^\mathsf{T}X_c)^{-1}c =\frac{8-4-4+8}{48}=\frac16. $$ Thus $\hat q=0.5+1.5=2$, $\nu=8-3=5$, and the standard error is $\sqrt{(8/5)(1/6)}=\sqrt{4/15}\approx0.5164$. The exact 95% margin is $2.5706(0.5164)\approx1.327$, giving $(0.673,3.327)$.

The off-diagonal entries matter: omitting the covariance would give $v=1/8+1/8=1/4$, not $1/6$. The contrast variance is determined by the full inverse cross-product matrix, not by adding the separate coefficient variances alone.

Interpreting and Reporting the Interval

For a Gaussian coefficient sum, the profile likelihood curve is symmetric around $c^\mathsf{T}\hat\beta$, because the constrained residual sum of squares depends on the squared difference from that estimate. Its exact finite-sample interval agrees with the usual $t$-interval for the same linear contrast. The profile formulation remains useful: it shows exactly how the contrast changes the constrained fit and makes clear which likelihood-ratio cutoff is being used.

An asymptotic profile interval instead uses the $\chi^2_1$ cutoff in $\Lambda(q)\leq\chi^2_{1,1-\alpha}$. That is a large-sample calibration, not generally the exact finite-sample calibration in a Gaussian model with estimated dispersion. For a coefficient sum, report the target, the point estimate, the interval, and whether its cutoff is asymptotic or exact. The dispersion estimate and the covariance among coefficients affect the interval through $\mathrm{RSS}_0/\nu$ and $c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$.

Common mistake: Profiling a coefficient sum is not the same as profiling each coefficient separately and adding the resulting interval endpoints. Separate intervals do not preserve the joint likelihood constraint, and adding coefficient variances while ignoring covariance gives the wrong uncertainty. Instead, impose the single constraint $c^\mathsf{T}\beta=q$, maximize over all compatible parameters, and use the full contrast factor $c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$.

Check Your Understanding

Use the definitions and Gaussian contrast results above to answer these questions.

What parameter vectors enter the maximization defining $L_p(\psi)$ for a scalar target $g(\theta)$?
In the Gaussian linear model, what is $v$ for the target $c^\mathsf{T}\beta$, and why can it not generally be calculated by adding coefficient variances alone?
Why does maximizing the Gaussian likelihood over $\sigma^2$ produce a log likelihood involving the logarithm of the residual sum of squares?
Which distribution supplies the exact finite-sample cutoff for the Gaussian coefficient-contrast interval, and what degrees of freedom does it use?
How does the exact Gaussian likelihood-ratio cutoff differ in purpose from the asymptotic $\chi^2_1$ cutoff?

Profile Intervals for Functions of Model Parameters

What You'll Learn