Why a Boundary Estimate Needs a Diagnosis
A parameter at a bound is a signal to investigate, not an instruction to make a particular change. It may reflect a genuine feature of the fitted model, limited information in the observations, an unsuitable parameterization or constraint, misspecification, or an optimization problem. These possibilities call for different responses. Automatically deleting a random effect, fixing a parameter at its bound, or widening the bound can hide the underlying issue and produce a fit that appears more successful than it is.
Consider a standard NLME model in which observation \(y_{ij}\) for individual \(i\) at time \(t_{ij}\) has a model prediction \(f(t_{ij},\theta_i)\), and individual parameters depend on population parameters and random effects. A common specification is \(\theta_i=g(\theta,b_i)\), with \(b_i\sim N(0,\Omega)\), together with a specified residual-error model. Estimation depends on the model, the observed design and outcomes, the likelihood, and any constraints imposed on parameters. A bound is a numerical restriction in that estimation problem; it is not, by itself, a scientific conclusion.
The discussion below assumes that the model and likelihood have been specified, parameters and bounds are recorded on their actual estimation scales, and the estimation objective is reproducible. These conditions matter: a profile or comparison is informative only if it refers to the same model, data, objective, and parameter definition. Convergence messages, gradient and Hessian diagnostics, and objective reproducibility are useful evidence, but none alone establishes model adequacy or a scientifically appropriate bound.
Interpret the Constraint Before Changing It
For a variance component, nonnegativity is intrinsic: a variance cannot be less than zero. Thus an estimate of zero is a feasible constrained estimate. It says that, under the fitted model and available data, the likelihood optimization did not require a positive value to improve the fit. It does not prove that the population has no heterogeneity, nor does it prove that the random effect should always be removed. Information may be weak, or the model may not give the data enough leverage to separate that variability from residual error or other parameters.
Also check which quantity is bounded. A random-effect standard deviation, a variance, and a coefficient of variation are not interchangeable. For example, if a lognormal individual parameter is written \(\theta_i=\theta\exp(\eta_i)\) with \(\eta_i\sim N(0,\omega^2)\), then \(\omega\) is a standard deviation on the log scale, whereas \(\omega^2\) is its variance. The coefficient of variation on the original scale is \(\sqrt{\exp(\omega^2)-1}\). Bounds and interpretations must match the quantity the software actually estimates.
At an interior optimum of a smooth objective, its derivative with respect to a free parameter is zero. At a lower bound, the derivative need not be zero: moving into the allowed region must not lower the objective. For a negative log-likelihood \(Q\) minimized over \(\omega^2\geq0\), a local minimum at zero can have a nonnegative right derivative. Therefore, a nonzero ordinary gradient component at a constrained solution is not automatically evidence of optimizer failure; it must be interpreted with the active constraint and software diagnostics.
This condition is necessary for a local minimum at the lower boundary in a one-dimensional smooth profile, not proof of a global optimum or of a correct model. Check the objective over a meaningful range and assess the fit using the other available diagnostics.
Worked Example: A Variance Estimate at Zero
Suppose a hypothetical fit estimates the variance \(\omega^2\) of an individual log-clearance random effect, constrained to be nonnegative. The profiled negative log-likelihood \(Q\), with other parameters reoptimized at each value, is 120.00 at \(\omega^2=0\), 120.08 at 0.01, and 120.35 at 0.04. The variance is on the squared log scale.
The right-hand slope estimated from the first two values is \(\{120.08-120.00\}/(0.01-0)=8.00\) objective units per unit of variance. The objective increases as the variance moves inward from zero in this neighborhood. These values support a constrained minimum at zero over the examined neighborhood; they do not establish that the population variance is truly zero.
A suitable response is to inspect whether the design contains enough information to distinguish between-individual clearance variability and residual error, and to examine the fit and optimization diagnostics. If the fit is otherwise stable and the profile remains uninformative about positive variance, report that the component was estimated at its boundary and describe the resulting limitation. Do not infer that deleting the random effect is automatically justified: that changes the model and should be treated as a considered model comparison.
Diagnose Before Choosing a Remedy
A useful sequence is to verify what is constrained, assess whether the estimate is numerically credible, examine how the data support the parameter, and then consider model structure. This sequence separates a genuine boundary solution from a boundary caused by avoidable setup or optimization problems.
Confirm the estimated quantity, its units and scale, the exact lower and upper bounds, and whether the reported estimate is truly at the bound or merely close under rounding.
Review objective reproducibility, gradient behavior under the active constraint, Hessian or covariance-step messages, and results from suitable alternative starts. A boundary value accompanied by unstable or irreproducible optimization deserves further numerical investigation.
Ask whether the sampling times, dose information, and number and distribution of observations can inform this parameter separately from other parameters and residual error. A flat objective profile can indicate weak identification, not necessarily a bound that should be widened.
Check units, transformations, covariate definitions, random-effect placement, and whether the assumed structural or residual model can represent the observed data. Change one defensible feature at a time and compare the resulting fits.
A profile is especially useful for deciding whether a numerical bound is influential. Hold the parameter at a series of values and reoptimize the other parameters; then inspect how the objective and relevant predictions change. A steadily improving objective at the upper bound suggests that the imposed bound may be active in a consequential way, but could also reflect weak identification or model misspecification. A nearly flat profile means the data provide little precision over that range; extending it does not create information.
Worked Example: Investigating an Upper Bound
In a hypothetical model, \(\omega\) is the standard deviation of a log-clearance random effect and is constrained to \(0\leq\omega\leq0.20\). The estimate is at the upper bound. A profile of the negative log-likelihood gives \(Q(0.20)=100.0\), \(Q(0.30)=97.8\), \(Q(0.50)=96.9\), and \(Q(0.80)=96.8\), with other parameters reoptimized at each point.
The objective falls by \(100.0-96.8=3.2\) units between the original bound and \(\omega=0.80\). Thus the original upper bound restricts the best fit in this profile. However, the improvement from 0.50 to 0.80 is only \(96.9-96.8=0.1\) unit, suggesting a relatively flat region at larger values; the data may not locate \(\omega\) precisely.
On the original lognormal scale, \(\omega=0.80\) corresponds to a coefficient of variation of \(\sqrt{\exp(0.80^2)-1}=\sqrt{\exp(0.64)-1}\approx\sqrt{0.8965}\approx0.947\), or about 94.7%. That is a very large implied variability and warrants scrutiny of the data, model structure, and parameterization. The profile justifies investigating the restrictive bound; it does not justify simply accepting 0.80 as a scientifically credible estimate. Any revised constraint should be supported by the parameter's definition, the model, and the observed data, and the revised fit should be checked for stability and adequacy.
Common Responses That Can Make Matters Worse
The following actions may be appropriate in a particular analysis, but none is a default repair. Each changes either the optimization problem, the model, or the interpretation and therefore requires a reason and verification.
| Tempting response | Why it can mislead | More defensible response |
|---|---|---|
| Delete a random effect because its variance is estimated as zero | A boundary estimate may reflect weak information or model structure; deletion changes the model and can conceal uncertainty about the component. | Assess data support and fit, then compare a reduced model deliberately. Treat boundary inference with care rather than relying automatically on ordinary interior-parameter approximations. |
| Fix a parameter at its boundary so the fit will finish | Fixing it removes it from estimation and can make convergence appear easier without showing that the fixed value is appropriate. | Determine whether the constraint is scientifically and numerically justified; refit and verify that other estimates, objective, and predictions are understood. |
| Loosen a bound whenever the estimate reaches it | The expanded range may be weakly identified, implausible on the parameter's actual scale, or a symptom of misspecification. | Use a profile and review units, parameterization, data support, and model structure before considering a revised bound. |
| Call every boundary estimate optimizer failure | Constrained optima can occur at a boundary and need not have a zero unconstrained gradient. | Interpret diagnostics in light of the active constraint and check reproducibility and the objective profile. |
Deleting a random effect deserves particular care. If setting its variance to zero gives a nested model, that is a meaningful candidate for comparison, but it is not merely a numerical adjustment: the reduced model asserts that the component is absent from the fitted population distribution. At a variance boundary, standard likelihood-based tests can have nonstandard behavior, so ordinary interior-parameter reference distributions should not be applied automatically. The relevant uncertainty and comparison method depend on the analysis; a boundary estimate alone does not supply them.
Worked Example: Why “Fix It at the Bound” Is Not a Diagnosis
A hypothetical fit estimates a random-effect variance \(\omega^2\) at zero, with \(Q=120.00\). An analyst fixes \(\omega^2=0\), reruns the model, and obtains \(Q=120.00\). This equality is expected if the original fit was already at exactly zero and the fixed model is otherwise identical: fixing the parameter has not improved the objective or supplied new evidence.
Suppose instead the reported estimate was 0.002, rounded to 0.00 in a summary, and the fitted objective was \(Q=119.99\). Fixing the variance at zero and obtaining \(Q=120.00\) worsens the objective by \(120.00-119.99=0.01\). The difference is small in this hypothetical example, but its importance cannot be decided from the rounded output alone. The analyst should verify the unrounded estimate, profile and diagnostics, then state explicitly that fixing the variance changes the model. “The fit converged after fixing it” does not establish that zero is the right value.
Limits of Boundary-Based Conclusions
A boundary estimate is conditional on the chosen model, likelihood, data, parameterization, and numerical constraints. It cannot by itself determine whether a random effect exists in the population, whether the study design was informative, or whether the model is adequate. Profile shapes and optimizer diagnostics help localize the problem, but can be ambiguous: weakly identified parameters, correlations among parameters, and structural or residual misspecification can produce similar symptoms.
There is also no universal rule that bounds must be wide, narrow, or symmetric. A bound should respect the parameter's meaning and be broad enough not to truncate plausible values without justification. Conversely, an arbitrarily broad range may worsen numerical behavior or admit values that make the model scientifically nonsensical. If the parameter is re-expressed—for example, using a log scale to enforce positivity—confirm that the transformation and its bounds correspond to the intended model, rather than assuming that a different numerical scale changes the information in the data.
The practical goal is not to eliminate boundary estimates. It is to determine whether the estimate is a credible constrained solution, a consequence of limited information, or evidence that the numerical setup or model needs attention. Record the constraint, the diagnostic evidence, the change made, and whether the fit and interpretation changed. That record makes a rescue interpretable rather than merely successful in producing an output.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A nonnegative variance estimate is zero. Does this prove that the corresponding population random effect is absent? Explain.
- A parameter estimate reaches its upper bound, and the profiled objective decreases when the parameter is evaluated beyond that bound. What does this establish, and what does it not establish?
- Why can a nonzero gradient component be compatible with a constrained optimum at a lower bound?
- A random-effect standard deviation on the log scale is \(\omega=0.40\). Calculate the corresponding lognormal coefficient of variation using \(\sqrt{\exp(\omega^2)-1}\).
Solutions. (1) No. It establishes that the constrained fit did not obtain a positive variance estimate under the specified model and data; weak information or model structure may contribute, so absence is not proved. (2) It establishes that the imposed bound restricts the profiled fit over the evaluated range. It does not establish that a larger estimate is well identified, scientifically plausible, or consistent with an adequate model. (3) At a lower bound, only movement into the feasible region is allowed. For a minimized objective \(Q\), a nonnegative right derivative \(Q'(0^+)\) means a small allowed increase does not lower \(Q\), which is compatible with a boundary minimum. (4) \(\omega^2=0.16\), so the coefficient of variation is \(\sqrt{\exp(0.16)-1}=\sqrt{1.1735-1}=\sqrt{0.1735}\approx0.4165\), or about 41.7%.