Reoptimizing Nuisance Parameters in a Profile

Why Reoptimization Is Part of the Profile

A profile likelihood asks how well the model can fit the data when a parameter of interest is held at a specified value. Other parameters are allowed to adjust to that constraint. This adjustment is essential in nonlinear mixed-effects (NLME) models: a change in clearance, for example, may be partly offset by changes in volume, residual-error parameters, or random-effect variability. The profile records the best fit available at each target value, not the fit obtained by keeping all other parameters at their full-model estimates.

The introductory material on profile likelihood establishes its use for parameter intervals. Here the central computational point is that every target value requires a new nuisance-parameter optimization. The likelihood must be the same likelihood used for the full fit, including the same treatment of random effects, data, and numerical approximation. If the NLME likelihood is evaluated by an approximation, use that same approximation at every profile point; changing the approximation partway through does not produce a coherent profile.

Definition: Let $\psi$ be the scalar parameter of interest and let $\lambda$ denote all remaining parameters. For log likelihood $\ell(\psi,\lambda)$, the profile log likelihood is $$ \ell_p(\psi)=\sup_{\lambda\in\Lambda(\psi)}\ell(\psi,\lambda), $$ where $\Lambda(\psi)$ is the set of nuisance-parameter values allowed when $\psi$ is fixed. Thus, at each $\psi$, nuisance parameters are re-estimated to maximize the likelihood subject to that fixed value. If $Q=-2\ell$, the profile objective is $Q_p(\psi)=\inf_{\lambda\in\Lambda(\psi)}Q(\psi,\lambda)$.

The full-model estimate maximizes $\ell$ over both $\psi$ and $\lambda$. At a profile point, $\psi$ is fixed and the nuisance parameters are optimized over their admissible values. The likelihood-ratio statistic is based on the difference between the full-model optimum and this constrained optimum. For example, using $Q=-2\ell$, the difference is $Q_p(\psi)-Q_{\min}$, where $Q_{\min}$ is the minimum over all parameters. The likelihood-ratio cutoff for an interval is supplied by the usual profile-interval procedure; it is not a reason to hold nuisance parameters fixed.

How Nuisance Reoptimization Changes Curvature

A local derivation makes clear why profiling can be less sharply curved than a calculation that freezes nuisance parameters. Near an interior optimum, approximate the negative log likelihood $Q$ by a quadratic. Write $\delta\psi$ and $\delta\lambda$ for displacements from that optimum, and let the blocks of its Hessian be $H_{\psi\psi}$, $H_{\psi\lambda}$, and $H_{\lambda\lambda}$. Assume $H_{\lambda\lambda}$ is positive definite, so the local quadratic has a unique nuisance minimum for each small $\delta\psi$.

Q(\delta\psi,\delta\lambda)-Q_{\min} \approx \frac{1}{2}\left( H_{\psi\psi}\delta\psi^2 +2\delta\psi H_{\psi\lambda}\delta\lambda +\delta\lambda^{T}H_{\lambda\lambda}\delta\lambda \right).

For a fixed $\delta\psi$, differentiate this quadratic with respect to $\delta\lambda$, set the derivative to zero, and solve. Substitution gives the profiled quadratic:

H_{\lambda\lambda}\delta\lambda +H_{\lambda\psi}\delta\psi=0 \quad\Longrightarrow\quad \delta\lambda^*=-H_{\lambda\lambda}^{-1}H_{\lambda\psi}\delta\psi, $$ $$ Q_p(\delta\psi)-Q_{\min} \approx \frac{1}{2}\delta\psi^2 \left(H_{\psi\psi} -H_{\psi\lambda}H_{\lambda\lambda}^{-1}H_{\lambda\psi}\right).

The subtracted term is nonnegative when the joint Hessian is positive semidefinite. The local profile curvature is therefore no greater than the curvature obtained by freezing nuisance parameters, $H_{\psi\psi}$. Intuitively, reoptimization lets nuisance parameters absorb some of the change in fit. If the target and nuisance parameters are locally uncorrelated, the cross-block is zero and the two curvatures agree. This is a local result near a regular interior optimum; boundaries, nonlinearity, and multiple optima can make the full profile quite different from a quadratic approximation.

Worked Example: Reoptimization Widens a Profile Interval

Consider a hypothetical two-parameter likelihood with target $\psi$, nuisance parameter $\lambda$, and $Q(\psi,\lambda)=(\psi-\lambda)^2+\lambda^2$. The full minimum is $Q_{\min}=0$ at $(\psi,\lambda)=(0,0)$. Suppose the likelihood-ratio cutoff is 3.84.

Profile calculation. At fixed $\psi$, minimize $Q$ over $\lambda$. Differentiation gives $2(\lambda-\psi)+2\lambda=0$, so $\hat{\lambda}_{\psi}=\psi/2$. Substituting, $Q_p(\psi)=(\psi-\psi/2)^2+(\psi/2)^2=\psi^2/2$. The interval endpoints solve $\psi^2/2=3.84$, giving $\psi=\pm\sqrt{7.68}$, or approximately $-2.771$ and $2.771$ in the units of $\psi$.

Incorrect fixed-nuisance calculation. If instead $\lambda$ is frozen at its full-fit estimate, zero, then $Q(\psi,0)=\psi^2$. Using the same cutoff gives endpoints $\psi=\pm\sqrt{3.84}$, or approximately $-1.960$ and $1.960$. This narrower range is not the profile interval: it ignores the improvement available by re-estimating $\lambda$ at each $\psi$.

Computing Each Constrained Fit

For a practical profile, choose a target value, impose that value as a constraint, and optimize all other parameters subject to their usual bounds and model constraints. Record the optimized nuisance estimates, the objective value, and diagnostics for that fit. Repeat for each target value. The nuisance estimates generally change from point to point; their movement is not a defect but part of the profile calculation.

In an NLME model, the likelihood often depends on population parameters and on the distribution of subject-level random effects. The nuisance optimization concerns the parameters remaining free in the chosen likelihood formulation. It does not mean that subject-specific random effects should automatically be treated as extra unrestricted parameters: follow the model's likelihood definition and the estimation method used for the full fit. A conditional objective and a marginal likelihood, for instance, are not interchangeable merely because both can be optimized.

Set the target.
Fix $\psi$ at one grid value, while retaining all parameter bounds, positivity constraints, and structural restrictions.

Optimize the nuisance parameters.
Use the same likelihood implementation as the full fit. Try informed starting values and, where needed, additional starts to assess whether the optimizer found the best constrained fit.

Check and record the result.
Save the objective value, nuisance estimates, convergence information, and relevant gradients or constraint diagnostics. Compare with nearby target values and alternative starts.

Refine consequential points.
Use a denser grid or direct constrained optimization near a suspected interval crossing, boundary, or abrupt change in the profile.

Worked Example: Locating an Approximate Clearance Interval

Suppose a hypothetical NLME analysis estimates clearance, $CL$, in L/h. At each fixed $CL$, the model re-estimates volume and residual-error parameters. The full fit has $Q_{\min}$, and the reported values below are constrained objective differences $Q_p(CL)-Q_{\min}$. Assume the cutoff is 3.84.

Fixed $CL$ (L/h)	Re-estimated volume (L)	Profile objective difference
3.5	31.8	4.60
4.0	33.0	2.70
5.0	35.0	0.00
6.0	37.4	2.40
6.5	38.9	4.30

The cutoff is crossed between 3.5 and 4.0 L/h on the lower side. With linear interpolation across that grid segment, the estimated crossing is $3.5+[(4.60-3.84)/(4.60-2.70)](4.0-3.5)=3.70$ L/h. On the upper side, the interpolated crossing is $6.0+[(3.84-2.40)/(4.30-2.40)](6.5-6.0)=6.379$ L/h. Thus the grid suggests an approximate interval from 3.70 to 6.379 L/h. These interpolated values are not exact endpoints: refine the grid and reoptimize near each crossing before reporting endpoints. The changing volume estimates also show why freezing volume at 35 L would answer a different question.

Optimization Failure Can Masquerade as Profile Shape

The required nuisance optimum is a global maximum of the constrained likelihood, or equivalently a global minimum of $Q$. A numerical optimizer may stop at a local optimum, fail to converge, or satisfy a stopping rule while still being far from the best constrained value. Such a fit has a likelihood that is too low and a $Q$ value that is too high relative to the true profile point. Because the error can vary across target values, it can create apparent curvature, move an interval crossing inward, or produce a gap in what should be a connected interval.

Worked Example: A Local Optimum Creates a False Gap

Return to the hypothetical objective $Q(\psi,\lambda)=(\psi-\lambda)^2+\lambda^2$, whose true profile is $Q_p(\psi)=\psi^2/2$. At $\psi=1.5$, the nuisance optimum is $\lambda=0.75$, and the correct profile value is $1.5^2/2=1.125$. With cutoff 3.84, this point belongs to the interval.

Suppose a failed optimization instead returns $\lambda=2.5$. Substitution gives $Q(1.5,2.5)=(1.5-2.5)^2+2.5^2=1+6.25=7.25$. The reported point is then incorrectly outside the interval because $7.25>3.84$. At $\psi=1$, the true profile value is $0.5$; at $\psi=2$, it is $2$, and both are below the cutoff. If this failed solution recurs over a band around 1.5, the computed accepted set can appear split into two pieces even though the exact profile is the smooth curve $\psi^2/2$. Re-optimizing from better or multiple starts recovers $\lambda=0.75$ at $\psi=1.5$ and removes the false gap.

A successful status flag alone is not enough to establish that a constrained optimum has been found. Inspect whether the nuisance estimates jump abruptly between adjacent target values, whether gradients are small at an interior solution, whether a parameter is unexpectedly pinned to a bound, and whether starts from neighboring profile points or the unconstrained fit reach better objective values. A sudden deterioration followed by an abrupt recovery deserves investigation rather than automatic interpretation as a feature of the model.

Alternatives, Limitations, and a Common Mistake

Calculation	What is optimized or approximated?	What it can tell you
Profile likelihood	Re-estimates nuisance parameters at each fixed target value.	Likelihood support for target values while allowing the rest of the model to adapt.
Fixed-nuisance likelihood	Holds nuisance parameters at selected values.	A conditional slice, not the profile likelihood; it can distort curvature and interval limits.
Wald interval	Uses local curvature or an estimated standard error near the full fit.	A local approximation; it does not perform constrained optimization across target values.
Bootstrap interval	Uses estimates from resampled or simulated data sets.	Sampling uncertainty under the chosen bootstrap design; it is a different calculation from profiling the observed-data likelihood.

Profiling can be computationally intensive because each target value requires a constrained fit. It cannot repair a misspecified model, guarantee parameter identifiability, or make an inaccurate likelihood approximation valid. Difficult boundaries and flat or irregular likelihoods require particular care; a numerical profile should be checked rather than treated as definitive simply because software returned values. Reoptimization is necessary for a valid profile, but it is not sufficient for every inferential assumption to hold.

Common mistake: “The full-model nuisance estimates are the best estimates, so I can keep them fixed while varying the target parameter.” They are best only when the target parameter is also free. Once the target is constrained to another value, the best-fitting nuisance values may change. The correction is to solve a new nuisance-parameter optimization for every fixed target value, and verify that it reached the constrained optimum.

Key takeaway: A profile point is the best likelihood achievable at its fixed target value. If nuisance parameters are not re-estimated—or are not adequately optimized—the resulting curve is not a reliable profile and its curvature, endpoints, and connectedness may be misleading.

Check Your Understanding

Answer each question, then compare with the brief solutions.

In a profile likelihood for clearance, what is fixed at each point and what is re-estimated?
For $Q(\psi,\lambda)=(\psi-\lambda)^2+\lambda^2$, find the nuisance optimum and profile objective at $\psi=2$.
Why can a failed nuisance optimization make an interval too narrow?
A profile objective rises sharply at one grid point, then falls at the next. Give two checks that could help determine whether this is a numerical failure.

Solutions. (1) Clearance is fixed at the selected value; all remaining model parameters are re-estimated subject to their constraints. (2) Differentiating gives $4\lambda-2\psi=0$, hence $\hat{\lambda}_{\psi}=1$; substitution gives $Q_p(2)=(2-1)^2+1^2=2$. (3) An unsuccessful constrained minimization can leave $Q$ too high, making target values appear to cross the interval cutoff sooner than they should. (4) Refit from multiple starts and compare objective values; also inspect convergence, gradients, bounds, and nuisance estimates at adjacent points.