What Is Optimized in a Profile?
A profile likelihood at a specified value of a target parameter is not obtained by simply evaluating the likelihood at that value while leaving every other parameter unchanged. The other parameters are reoptimized subject to the target being fixed. This distinction is central in GLM inference: the resulting likelihood measures how well the model can fit when the target is constrained, allowing the nuisance parameters to adjust as effectively as possible.
Write the full parameter as \(\theta=(\psi,\lambda)\), where \(\psi\) is a scalar target and \(\lambda\) collects the nuisance parameters. For a fixed target value \(\psi_0\), the profile log-likelihood is the supremum of the log-likelihood over all allowed nuisance values. A maximum is attained when some nuisance value achieves that supremum; if not, the supremum still defines the profile value.
In logistic regression, suppose the target is the coefficient \(\beta_1\), with intercept \(\alpha\) and other slopes collected in \(\gamma\). At a proposed value \(b\), the required optimization is
The constraint is \(\beta_1=b\); the intercept and every other slope remain free. The same principle applies to other GLMs. For example, fixing a Poisson-regression slope still requires optimizing the intercept and any unfixed slopes.
Constrained Logistic Likelihood and Its Scores
For independent grouped binomial observations, let \(Y_i\) be the number of successes out of \(m_i\) trials, with covariates \(x_{i1}\) and \(x_{i2},\ldots,x_{ip}\). The logistic model has \(\operatorname{logit}(p_i)=\alpha+\beta_1x_{i1}+\sum_{j=2}^p\beta_jx_{ij}\). Ignoring binomial coefficients, which do not depend on the parameters, its log-likelihood is
Once \(\beta_1=b\) is fixed, the remaining linear predictor is an offset \(b x_{i1}\) plus a linear combination of the free parameters. Let \(z_i=(1,x_{i2},\ldots,x_{ip})^\mathsf{T}\) and \(\gamma=(\alpha,\beta_2,\ldots,\beta_p)^\mathsf{T}\). Then \(\eta_i=bx_{i1}+z_i^\mathsf{T}\gamma\), and the nuisance score equations are
These equations say that the fitted residuals have zero weighted sums in the directions represented by the free design columns. In particular, the intercept score requires the total observed successes to equal the total fitted successes. The score equations for other nuisance slopes require the corresponding covariate-weighted residual sums to vanish.
Proof. Differentiating the nuisance score gives the Hessian
where \(Z\) has rows \(z_i^\mathsf{T}\), and \(W\) is diagonal with entries \(m_i p_i(1-p_i)>0\) for finite predictors. For any nonzero vector \(v\),
The strict inequality follows because full column rank implies \(Zv\ne0\), and every diagonal entry of \(W\) is positive. Thus the Hessian is negative definite everywhere, so the log-likelihood is strictly concave in the nuisance parameters. A differentiable concave function with zero gradient at an interior point is globally maximized there; strict concavity makes that maximizer unique. Therefore a finite score solution is the unique constrained maximum. \(\square\)
The rank and finiteness conditions matter. If the nuisance design is rank deficient, the maximizer need not be unique. If the data are separated in a logistic model, a finite maximum may fail to exist; in that case the profile is defined by a supremum, not by claiming a finite score-zero estimate.
Worked Examples
Worked Example: Fixing a Poisson Slope
Suppose independent Poisson counts are observed at \(x=0\) and \(x=1\), with observed counts \(3\) and \(5\). Use the model \(\log(\mu_x)=\alpha+\beta x\), and find the constrained estimate of \(\alpha\) when \(\beta=\log 2\).
For a Poisson observation, the parameter-dependent log-likelihood is \(y\log(\mu)-\mu\). With the slope fixed, the two means are \(\mu_0=\exp(\alpha)\) and \(\mu_1=2\exp(\alpha)\). The constrained log-likelihood is therefore
Its intercept score is \(8-3\exp(\alpha)\). Setting it to zero gives \(\exp(\hat\alpha)=8/3\), so \(\hat\alpha=\log(8/3)\). The second derivative is \(-3\exp(\alpha)<0\), confirming this is the unique constrained maximum. The maximized parameter-dependent log-likelihood is
The slope was held at \(\log 2\), but the intercept was estimated anew under that restriction. This is the basic profile operation in a simple GLM.
Worked Example: Reoptimizing Both Logistic Nuisance Parameters
Consider four grouped-binomial observations, with \(x_1\) the target-coefficient covariate and \(x_2\) the other slope covariate. The columns give the number of trials and observed successes.
| \(x_1\) | \(x_2\) | Trials \(m_i\) | Successes \(y_i\) |
|---|---|---|---|
| 0 | 0 | 20 | 4 |
| 1 | 0 | 20 | 9 |
| 0 | 1 | 24 | 8 |
| 1 | 1 | 24 | 12 |
Fit \(\operatorname{logit}(p_i)=\alpha+\beta_1x_{i1}+\beta_2x_{i2}\), fixing \(\beta_1=\log 2\). We solve for both \(\alpha\) and \(\beta_2\); they are not specified in advance. Consider the candidate values \(\alpha=\log(1/3)\) and \(\beta_2=\log(3/2)\). The four linear predictors imply the probabilities
For example, at \((x_1,x_2)=(1,0)\), the odds are \((1/3)\times2=2/3\), giving probability \((2/3)/(1+2/3)=2/5\). At \((0,1)\), the odds are \((1/3)(3/2)=1/2\), giving probability \(1/3\).
The intercept score is observed successes minus fitted successes. The observed total is \(4+9+8+12=33\). The fitted total is \(20(1/4)+20(2/5)+24(1/3)+24(1/2)=5+8+8+12=33\), so the intercept score is zero. The \(\beta_2\) score uses only the \(x_2=1\) observations: their observed successes total \(8+12=20\), and their fitted successes total \(24(1/3)+24(1/2)=8+12=20\). Thus that score is also zero. The nuisance design columns \(1\) and \(x_2\) have full rank, so the strict-concavity result establishes that these values are the unique constrained estimates.
The parameter-dependent log-likelihood at the constrained maximum is
Each success count multiplies the log fitted probability, and each failure count multiplies the log fitted failure probability. The binomial coefficient terms are omitted because they are constant across parameter values. The fitted probability \(1/3\) in the third cell follows from its linear predictor \(\log(1/2)\); it is not \(0.4\).
Worked Example: Fixing the Logistic Slope at Zero
Use the same grouped data, but now fix \(\beta_1=0\). Since the fitted probability is then the same for the two \(x_1\) values within each \(x_2\) group, the nuisance score equations can be solved separately by group. For \(x_2=0\), there are \(13\) successes in \(40\) trials, so the fitted probability must be \(13/40\). For \(x_2=1\), there are \(20\) successes in \(48\) trials, so the fitted probability must be \(20/48=5/12\).
The corresponding logits give
Indeed, the odds for \(13/40\) are \(13/27\), and the odds for \(5/12\) are \(5/7\). Subtracting the first fitted logit from the second gives \(\hat\beta_2=\log(5/7)-\log(13/27)=\log(135/91)\). The observed and fitted success totals match within each \(x_2\) group, so both nuisance scores vanish. Full rank of the two nuisance design columns again gives the unique constrained maximum.
The constrained log-likelihood, again omitting parameter-independent binomial coefficients, is
The total differs from the value at \(\beta_1=\log 2\), as expected: each target value has its own nuisance estimates and its own maximized constrained likelihood. The calculation does not, by itself, determine an interval or a cutoff.
A Reliable Optimization Workflow
For a general GLM, the same constrained-optimization structure applies even when the nuisance score equations do not have closed-form solutions. At a fixed target value, treat its contribution to the linear predictor as fixed, construct the likelihood for the remaining parameters, and solve the nuisance optimization. A numerical optimizer may be needed, but the target must remain fixed throughout.
For a coefficient target, impose \(\beta_1=b\) and identify all remaining free parameters.
Substitute the fixed value into the GLM likelihood or linear predictor; retain every nuisance parameter as an optimization variable.
Solve their score equations when an interior solution exists, or use a suitable constrained numerical maximization.
Verify the nuisance scores or optimizer convergence, inspect rank and existence conditions, and evaluate the likelihood at the fitted nuisance values.
The restricted score equations are a useful diagnostic, not merely a computational detail. A claimed constrained maximum with a nonzero nuisance score is not the profile likelihood value. Likewise, numerical convergence alone does not establish a finite maximum if the likelihood can increase along a diverging parameter sequence.
Common Mistake: Holding Nuisance Estimates Fixed
A common error is to take the unrestricted fitted intercept and slopes, replace the target coefficient by \(b\), and call the resulting likelihood the profile likelihood. That evaluates one particular point on the constrained parameter space. It generally does not maximize over that space.
The correction is to re-estimate all nuisance parameters at each fixed target value. In the grouped logistic example, \(\beta_1=0\) led to \(\hat\alpha=\log(13/27)\) and \(\hat\beta_2=\log(135/91)\); fixing the target changed the nuisance fit so that the groupwise score equations were satisfied. The profile value is the likelihood at that constrained optimum, not at nuisance values borrowed from another fit.
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- In a logistic regression with an intercept and three slopes, if the second slope is the target and is fixed, which parameters are optimized to compute its profile likelihood?
- Why does a zero nuisance score identify the unique constrained logistic maximum under the full-rank and finite-solution conditions in the theorem?
- In the Poisson example, write the intercept score equation at fixed \(\beta=\log 2\) and verify the resulting estimate.
- For the grouped-binomial example at \(\beta_1=\log 2\), verify the fitted success total in each \(x_2\) group.
- Explain why evaluating the likelihood at nuisance estimates from an unrestricted fit does not generally give a profile likelihood value.