How Nuisance Estimation Shapes a Profile Interval

Why the Nuisance Estimate Matters

A likelihood interval for a logistic-regression slope can change substantially depending on what is done with the intercept. If the intercept is held at its unrestricted maximum-likelihood estimate, the slope is assessed against one fixed baseline risk. If the intercept is reestimated for each candidate slope, the baseline risk can adjust to accommodate that slope. The latter construction is the profile likelihood.

This distinction is not a technical detail. Reestimating the intercept can make candidate slope values fit better, changing both the curvature of the likelihood and the likelihood-ratio interval. The interval may be wider or differently placed than one obtained by treating the estimated intercept as known.

Definition: Let $\psi$ be a scalar target parameter and $\lambda$ a nuisance-parameter vector. For log-likelihood $\ell(\psi,\lambda)$, the profile log-likelihood is $$ \ell_p(\psi)=\sup_{\lambda\in\Lambda(\psi)}\ell(\psi,\lambda), $$ where $\Lambda(\psi)$ is the allowable nuisance-parameter set at target value $\psi$. A profile likelihood-ratio confidence set retains values of $\psi$ for which $2\{\ell(\hat\psi,\hat\lambda)-\ell_p(\psi)\}$ does not exceed the chosen likelihood-ratio cutoff.

For a logistic regression with intercept $\alpha$ and slope $\beta$, the target might be $\beta$ and the nuisance parameter $\alpha$. The profile curve at a proposed $\beta$ is found by maximizing the likelihood over $\alpha$, not by automatically inserting the unrestricted estimate $\hat\alpha$.

A Grouped Logistic Model

Suppose two independent groups have binomial responses $Y_0\sim\operatorname{Bin}(n_0,p_0)$ and $Y_1\sim\operatorname{Bin}(n_1,p_1)$, with $\operatorname{logit}(p_j)=\alpha+\beta x_j$, where $x_0=0$, $x_1=1$, and $\operatorname{logit}(p)=\log\{p/(1-p)\}$. Thus $\alpha$ is the log-odds in group 0 and $\beta$ is the difference in log-odds between groups.

Ignoring binomial coefficients, which do not depend on the parameters, the log-likelihood is

\ell(\alpha,\beta) =y_0\log p_0+(n_0-y_0)\log(1-p_0) +y_1\log p_1+(n_1-y_1)\log(1-p_1).

At a fixed candidate $\beta$, the fitted probabilities must obey $\operatorname{logit}(p_1)-\operatorname{logit}(p_0)=\beta$. Reestimating the intercept chooses the pair of probabilities satisfying this constraint that maximizes the likelihood.

Worked Example: The Intercept Moves at a Fixed Slope

Take $n_0=n_1=10$, $y_0=4$, and $y_1=8$. The unrestricted estimates are $\hat p_0=4/10=0.4$ and $\hat p_1=8/10=0.8$, so

\hat\alpha=\operatorname{logit}(0.4) =\log(0.4/0.6)\approx-0.4055,\qquad \hat\beta=\operatorname{logit}(0.8)-\operatorname{logit}(0.4) \approx1.3863-(-0.4055)=1.7918.

Now constrain the slope to $\beta=0$. The fitted probabilities must then be equal. Differentiating the log-likelihood with respect to the common intercept gives the score $y_0+y_1-(n_0+n_1)p$. Setting it to zero yields $p=(4+8)/20=0.6$, and hence the constrained intercept is $\hat\alpha(0)=\operatorname{logit}(0.6)\approx0.4055$. The intercept has moved from about $-0.4055$ to $0.4055$ to accommodate the zero-slope constraint.

If instead the intercept is fixed at $\hat\alpha=-0.4055$, the fitted probability in group 0 remains $0.4$, and at $\beta=0$ the probability in group 1 is also $0.4$. This is a different fit from the profile fit, which uses probability $0.6$ in both groups.

What Profiling Guarantees

At each slope, the profiled likelihood is at least as large as the likelihood evaluated at any one fixed intercept, because the profile maximization includes that intercept among its candidates. When the fixed intercept is the unrestricted estimate, the two curves share the same maximum at $\hat\beta$. This gives an exact comparison of their likelihood-ratio statistics.

Result: Suppose $(\hat\psi,\hat\lambda)$ is an unrestricted maximum of $\ell$, and the fixed-nuisance curve uses $\lambda=\hat\lambda$. Assume also that $\hat\lambda\in\Lambda(\psi)$ for every allowable $\psi$ under comparison. Then for every such $\psi$, $$ \ell_p(\psi)\geq \ell(\psi,\hat\lambda) \quad\Longrightarrow\quad 2\{\ell(\hat\psi,\hat\lambda)-\ell_p(\psi)\} \leq 2\{\ell(\hat\psi,\hat\lambda)-\ell(\psi,\hat\lambda)\}. $$ Consequently, for the same likelihood-ratio cutoff, the profile confidence set contains the set obtained from this fixed-nuisance curve.

To prove the result, fix any allowable $\psi$. By definition, $\ell_p(\psi)$ is the supremum over allowable nuisance values. Since $\hat\lambda$ is one such value, that supremum is at least $\ell(\psi,\hat\lambda)$. Subtracting each side from the same unrestricted maximum and multiplying by two reverses the inequality, giving the displayed comparison. A value of $\psi$ whose fixed-nuisance likelihood-ratio statistic is below the cutoff must therefore also have a profile statistic below the cutoff. $\square$

The containment concerns this particular fixed-nuisance construction; it does not say every possible approximate interval must be nested inside a profile interval. Nor does the likelihood-ratio cutoff make a finite-sample interval exact in general. In regular models, a common large-sample choice for a nominal 95% interval for one scalar target is the 95th percentile of $\chi^2_1$, approximately $3.841$.

Profile Curvature and Efficient Information

The same effect appears locally near the maximum. The curvature of the profile log-likelihood is governed not by the target’s information alone, but by the information remaining after accounting for its association with nuisance parameters.

Result: Suppose $\ell(\psi,\lambda)$ is twice continuously differentiable near an interior unrestricted maximum $(\hat\psi,\hat\lambda)$. Assume the nuisance-parameter Hessian is nonsingular there and the nuisance maximizer $\hat\lambda(\psi)$ is locally differentiable. Write the observed information blocks at the maximum as $I_{\psi\psi}=-\ell_{\psi\psi}$, $I_{\psi\lambda}=-\ell_{\psi\lambda}$, and $I_{\lambda\lambda}=-\ell_{\lambda\lambda}$, with $I_{\lambda\lambda}$ positive definite. Then $$ -\ell_p''(\hat\psi) =I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ The right-hand side is the Schur complement, also called the efficient information for $\psi$ in the presence of $\lambda$.

For the derivation, the nuisance maximizer satisfies the nuisance score equation $\ell_\lambda(\psi,\hat\lambda(\psi))=0$. Differentiate this equation with respect to $\psi$. The chain rule gives $\ell_{\lambda\psi}+\ell_{\lambda\lambda}\hat\lambda'(\psi)=0$, so $\hat\lambda'(\psi)=-\ell_{\lambda\lambda}^{-1}\ell_{\lambda\psi}$.

Differentiate the profile log-likelihood $\ell_p(\psi)=\ell(\psi,\hat\lambda(\psi))$. Its first derivative is $\ell_\psi+\ell_\lambda\hat\lambda'(\psi)=\ell_\psi$, since the nuisance score is zero along the profile. Differentiating once more and substituting the expression for $\hat\lambda'(\psi)$ gives

\ell_p''(\hat\psi) =\ell_{\psi\psi} -\ell_{\psi\lambda}\ell_{\lambda\lambda}^{-1}\ell_{\lambda\psi} =-I_{\psi\psi}+I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}.

Negating this expression proves the stated curvature formula. The derivation uses the differentiability and nonsingularity assumptions above; the formula is a local result at an interior maximum. $\square$

When the full information matrix is positive definite, the subtracted term is nonnegative. Thus the profile curvature is no greater than the curvature obtained by holding the nuisance parameter fixed. Less curvature corresponds, locally, to greater uncertainty about the target when nuisance estimation is allowed for.

Worked Example: Logistic Information With an Estimated Intercept

For the two-group data above, the logistic information contribution from a group with size $n$ and fitted probability $p$ is $np(1-p)$ times the outer product of its covariate row $(1,x)$. At the unrestricted fit, group 0 contributes $10(0.4)(0.6)=2.4$ and group 1 contributes $10(0.8)(0.2)=1.6$. Adding their contributions gives

I= \begin{pmatrix} 4.0&1.6\\ 1.6&1.6 \end{pmatrix}, \qquad I_{\beta\beta\cdot\alpha} =1.6-\frac{(1.6)^2}{4.0} =0.96.

If the intercept were treated as fixed, the slope information would be $1.6$, corresponding to a local variance approximation $1/1.6=0.625$. With the intercept estimated, the efficient slope information is $0.96$, giving the larger local variance approximation $1/0.96\approx1.0417$. The ratio is $1.0417/0.625\approx1.67$. These are local quadratic approximations, not the likelihood-ratio interval endpoints; they show how allowing the intercept to adjust reduces the information available for the slope.

Worked Profile and Fixed-Intercept Intervals

Continue with $y_0=4$, $y_1=8$, and $n_0=n_1=10$. For a proposed slope, let $a$ and $b$ denote the fitted probabilities in groups 0 and 1. At the profile maximum over the intercept, the intercept score is zero: $4+8-10a-10b=0$. Therefore $a+b=1.2$, while the slope constraint requires $\beta=\operatorname{logit}(b)-\operatorname{logit}(a)$.

Twice the log-likelihood loss from the unrestricted fit, written as a function of $a$ with $b=1.2-a$, is

D(a)=2\left[ 4\log\frac{0.4}{a} +6\log\frac{0.6}{1-a} +8\log\frac{0.8}{1.2-a} +2\log\frac{0.2}{a-0.2} \right].

The profile interval endpoints solve $D(a)=3.841$, and the corresponding slopes are $\operatorname{logit}(1.2-a)-\operatorname{logit}(a)$. Numerical solution gives endpoint pairs approximately $(a,b)=(0.61135,0.58865)$ and $(0.2505,0.9495)$. Substitution into the slope constraint gives approximately $-0.0946$ and $4.030$, respectively.

For comparison, hold the intercept fixed at $\hat\alpha=-0.4055$. Group 0 then remains at its unrestricted fit, and the group 1 probability $q=\operatorname{logit}^{-1}(\hat\alpha+\beta)$ determines the likelihood loss. The fixed-intercept endpoints solve

2\left[ 8\log\frac{0.8}{q} +2\log\frac{0.2}{1-q} \right]=3.841.

The numerical solutions are approximately $q=0.5006$ and $q=0.9637$. Converting back to slope using $\beta=\operatorname{logit}(q)-\hat\alpha$ gives approximately $0.408$ and $3.684$. Thus the profile interval is about $[-0.0946,4.030]$, while the plug-in fixed-intercept interval is about $[0.408,3.684]$. The profile set is wider here and includes zero; the fixed-intercept set does not.

This contrast reflects the model fits, not a different choice of likelihood-ratio cutoff. At zero slope, for example, the profile fit moves the intercept to make both group probabilities $0.6$. The fixed-intercept fit instead leaves both probabilities at $0.4$. The former can explain the pooled response level while imposing no slope, whereas the latter cannot adjust its baseline.

Worked Example: Comparing the Curves at Zero Slope

At $\beta=0$, profiling gives $a=b=0.6$. The likelihood-ratio loss from the unrestricted fit is

D_p(0)=2\left[ 4\log\frac{0.4}{0.6}+6\log\frac{0.6}{0.4} +8\log\frac{0.8}{0.6}+2\log\frac{0.2}{0.4} \right]\approx3.45.

This is below $3.841$, so zero is retained by the profile likelihood-ratio set. With the intercept fixed, the probability in each group at zero slope is $0.4$, and group 0 contributes no loss. The fixed-intercept loss is

D_f(0)=2\left[ 8\log\frac{0.8}{0.4}+2\log\frac{0.2}{0.6} \right]\approx6.70.

This exceeds $3.841$, so zero is rejected by that plug-in construction. The likelihood comparison proves the direction of the difference: the profile loss cannot exceed the fixed-intercept loss at any slope. The numerical values show how large that difference can be in this data set.

Common Pitfall and Interpretation

Wrong: “The nuisance estimate is $\hat\alpha$, so the likelihood for every candidate slope should use $\hat\alpha$.” That treats the estimated intercept as if it were known.
Correction: For a profile likelihood, fix the candidate slope and maximize over the intercept anew. The nuisance estimate generally depends on the constrained slope.
Wrong: “The profile curve must be more sharply curved because it fits each slope as well as possible.” Maximizing the likelihood raises the curve and reduces its likelihood-ratio loss relative to fixing the nuisance value. Locally, the efficient-information formula shows the corresponding reduction in curvature.

A fixed-intercept interval can be useful as a diagnostic comparison, but it does not propagate uncertainty in the estimated intercept. In applications, the profile interval describes the slope after allowing the nuisance coefficient to adapt under each candidate slope. Its endpoints need not be symmetric around $\hat\beta$; the likelihood curve itself need not be symmetric.

Key takeaway: Profiling is a constrained reestimation operation, not a substitution of unrestricted nuisance estimates. The changing nuisance fit reshapes the target likelihood curve and can materially alter its confidence interval.

Check Your Understanding

Answer each question using the distinction between reestimating and fixing a nuisance coefficient.

In a logistic regression with an intercept and one slope, what is maximized to obtain the profile log-likelihood at a proposed slope?
For the two-group example, explain why the fitted group probabilities at slope zero are $0.6$ under profiling but $0.4$ when the intercept is fixed at its unrestricted estimate.
Why must the profile likelihood-ratio loss be no larger than the fixed-intercept loss at the same candidate slope?
What role does the Schur complement play in the local curvature of a profile log-likelihood?
In the worked interval comparison, which interval retained slope zero, and what likelihood-fit difference explains the result?