Why a Zero Likelihood Is a Distinct Failure
A likelihood calculation can fail in more than one way. A contribution may be nonfinite because a computation produced a value such as “not a number”; it may also be exactly zero. The latter is not a harmless small value: its logarithm is negative infinity, so it can make the objective function unusable to an optimizer. Diagnosing the cause matters. A genuine zero says the model assigns no likelihood to the observation at the current parameter values. A numerical zero may instead mean that a positive contribution was lost during calculation.
For independent observations with likelihood contributions \(L_i\), the total likelihood is their product, and the log-likelihood is the sum of their logarithms. If any \(L_i=0\), the product is zero and the log-likelihood is negative infinity. In a pharmacometric model, an individual contribution may itself be a marginal likelihood, integrating over that individual’s random effects. Therefore, first identify the exact quantity being logged: a single observation-level contribution, an individual marginal contribution, or the full likelihood.
For a discrete observation, a likelihood contribution is a probability mass. For a continuous observation, it is a probability density evaluated at the observation; it is not the probability of observing that exact point, which is zero for a continuous distribution. This distinction is essential when investigating “zero probability” reports. For example, a Normal density with finite observation and mean and strictly positive finite standard deviation is positive, though it may be far too small for direct floating-point representation.
Structural Zeros: The Model Assigns No Support
A structural zero occurs when the model assigns exactly zero probability or density to the observed data. This can arise from a support restriction, a boundary prediction, or a deterministic rule in the model. A simple example is a Bernoulli observation \(Y\), where \(Y=1\) denotes an event and \(P(Y=1)=p\). If the model predicts \(p=0\) but the observed value is \(Y=1\), then the contribution is zero. The log-likelihood is negative infinity. This is not a small residual or an optimizer preference: at those parameter values, the observation is impossible under the specified model.
In pharmacometric applications, investigate how the model generated the probability or density before concluding that the data contradict the model. A probability may have been set to zero by a hard threshold; a prediction may have landed outside a distribution’s support; a concentration may have been clipped or rounded to zero before a likelihood was evaluated; or a boundary parameter may have made an event impossible. Each possibility points to a different correction. Do not change a zero to a small constant until its origin is understood.
Worked Example: An Observed Event With Predicted Probability Zero
Suppose a hypothetical binary endpoint indicates whether a concentration exceeds a threshold. A model implementation uses a hard rule that assigns \(p=0\) whenever the predicted concentration is below the threshold. For one subject, the endpoint is \(Y=1\), but the prediction falls below the threshold, so the code supplies \(p=0\). What is the likelihood contribution?
For a Bernoulli observation, the likelihood contribution is \(p^Y(1-p)^{1-Y}\). Substituting \(Y=1\) and \(p=0\) gives
Thus \(\log L=\log(0)=-\infty\). The calculation is consistent with the implemented model: its hard rule makes the observed event impossible. The diagnostic question is whether that rule is scientifically intended. If the endpoint is meant to have a gradual probability relationship with concentration, the hard threshold is a model-specification issue. If the rule is intended, the data and model cannot have a finite likelihood at these parameter values. Replacing \(p=0\) by an arbitrary small number would conceal this conflict rather than resolve it.
A structural zero can also be local to a component of a mixture rather than to the full likelihood. For a marginal contribution formed by averaging over latent effects or states, one zero component does not imply that the marginal contribution is zero: other components may assign positive likelihood to the observation. The marginal is zero only if the integrand is zero throughout the region that has positive integration weight, apart from sets of integration measure zero.
Worked Example: A Zero Component Does Not Force a Zero Marginal
Consider a hypothetical two-component mixture for a binary response, with component weights \(0.7\) and \(0.3\). The observed response is \(Y=1\). The first component assigns probability zero to \(Y=1\); the second assigns probability \(0.02\). Calculate the marginal contribution and its log.
The mixture likelihood is the weighted sum of component contributions:
Therefore, \(\log L=\log(0.006)\approx -5.116\). The marginal contribution is positive even though one component contributes zero. Treating the first component’s zero as the entire individual contribution would be an aggregation error. In a random-effects likelihood, the corresponding check is whether the integration procedure captures regions with positive contribution, not whether every point or quadrature node contributes positively.
Numerical Zeros: A Positive Value Is Lost
A numerical zero has a different cause. Floating-point arithmetic has a finite range. If a positive density is smaller than the smallest representable positive value, direct evaluation may return zero. Products of many positive contributions can underflow for the same reason. The correct log-likelihood can still be finite, because it is the logarithm of a positive mathematical quantity—not necessarily the logarithm of the rounded computer representation.
For a Normal observation with mean \(\mu\) and standard deviation \(\sigma>0\), the log density at \(y\) is
Evaluating this expression directly avoids first computing a tiny density and then taking its logarithm. It does not, however, make every failure numerical. If a prediction is invalid, a variance is not positive, or the model truly excludes the observation, a log-scale calculation cannot create a valid likelihood.
Worked Example: A Normal Density Underflows but Its Log Is Finite
Suppose a hypothetical observation is \(y=100\) concentration units, with predicted mean \(\mu=0\) concentration units and \(\sigma=1\) concentration unit. These values are intentionally extreme to illustrate numerical behavior; they are not presented as realistic data. The Normal log density is
This is a finite log density. The density itself is approximately \(\exp(-5000.919)\), far below the range of ordinary double-precision floating-point numbers, so a direct density calculation may return zero. If software then takes the logarithm of that computed zero, it reports negative infinity even though the log density formula gives a finite value. The diagnosis is numerical underflow in the density evaluation, not a mathematical zero. The enormous discrepancy also warrants checking the prediction and units; numerical stability does not establish that the model is scientifically appropriate.
There is another common route to underflow: multiplying moderate or small positive contributions before taking a logarithm. If the contributions are available separately, compute their logarithms and add them instead. When a contribution is a sum of positive terms, use a stable log-sum calculation. For \(a_j=\log L_j\), let \(m=\max_j a_j\). Then
This identity follows by writing \(L_j=\exp(a_j)\), factoring \(\exp(m)\) out of the sum, and taking the logarithm. Each shifted exponent \(a_j-m\) is at most zero, which avoids overflow in the exponentiation. The calculation still requires valid component log contributions; it cannot repair an incorrectly specified component likelihood.
Worked Example: Avoiding Underflow in a Product
Suppose a hypothetical record contains eight independent outcomes. Each observed outcome has likelihood contribution \(10^{-50}\) at the parameter values being checked. A direct product gives
This value is positive mathematically but typically underflows to zero in double-precision arithmetic. Instead, sum the log contributions:
The log-likelihood contribution is finite. This example differs from a structural zero: no individual factor is zero. Inspecting the factors separately and then evaluating the sum of logs distinguishes product underflow from an observation that the model actually assigns zero likelihood.
A Practical Trace From the Logarithm Back to the Prediction
When a log-likelihood becomes negative infinity, trace the calculation in the reverse direction. Locate the first zero, identify whether it is a probability, density, component weight, quadrature result, or already-aggregated product, and then inspect the inputs that produced it. This is more informative than changing optimizer settings before establishing whether the objective function is mathematically defined.
Record the subject, observation, prediction, distribution inputs, and the precise likelihood component that first equals zero. Do not infer the source solely from a final log-likelihood of negative infinity.
Ask whether the observed value is allowed by the distribution and whether a parameter or prediction has reached a boundary that assigns it zero probability or density. Check for hard thresholds, clipping, and inappropriate rounding.
Use a distribution’s log-density or log-probability formula, and sum log contributions rather than multiplying probabilities. For sums of components, use a stable log-sum calculation.
Compare the prediction and distribution inputs with a trusted calculation or higher numerical precision. Inspect units, transformed values, solver output, and any rounding or underflow before the likelihood call.
State whether the contribution is structurally zero, numerically underflowed, or invalid because an upstream calculation failed. Make and test a correction at that source, not by silently substituting an arbitrary positive floor.
This trace complements the earlier material on nonfinite individual likelihood contributions: a zero is a specific reason that a logarithmic objective can become nonfinite, but it should not be treated as interchangeable with every other nonfinite result. Diagnostics should also test nearby parameter values and inspect subject-level contributions. If only certain observations fail, their recorded values, support, and predictions deserve particular attention. If failures arise across many observations together, a shared transformation, prediction, or likelihood implementation may be responsible.
Alternatives, Diagnostics, and Limits
Direct probability-scale evaluation is convenient when contributions are comfortably representable and the calculation is simple. It is a poor choice for products of many contributions or very small densities. Log-scale evaluation is generally more stable, but it requires an appropriate log-probability or log-density calculation. For mixtures and other sums, a stable log-sum-exp calculation is preferable to exponentiating all component log contributions and summing them directly.
| Approach | Useful when | Main risk |
|---|---|---|
| Direct probability or density | Values are moderate and individual checks are needed | Underflow in small values or products |
| Log probability or log density | Computing individual log-likelihood contributions | Cannot make an impossible observation have positive likelihood |
| Stable log-sum calculation | Combining positive likelihood components | Does not fix missing components or an incorrect model |
| Adding a small floor to zero | Almost never a principled likelihood correction | Changes the objective and can hide a model or coding defect |
A useful diagnostic is to compare the direct result, if representable, with the log-scale result on a small test case. Also inspect whether a positive contribution becomes zero only after multiplication, exponentiation, transformation, or integration. For an integral-based marginal likelihood, distinguish a genuinely zero integral from a numerical approximation that misses the region where the integrand is positive. Increasing integration accuracy may help diagnose the latter; it does not alter the model’s support.
A common mistake is to replace every zero likelihood by a small constant, such as \(10^{-12}\), and continue fitting. The reasoning is that the optimizer only needs a finite objective. The correction is that this substitution changes the likelihood, often in a data-dependent way, and may conceal an impossible observation, an incorrect prediction, or underflow. First establish which of those causes applies. Use stable log calculations for positive quantities that are too small to represent, and correct the model or its inputs when the zero is genuine or generated upstream.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A Bernoulli observation is \(Y=0\), and the model predicts \(p=1\) for \(Y=1\). What is its likelihood contribution and log contribution?
- A positive Normal density is too small to represent and the software reports zero. What computation should be checked, and what does the zero alone establish?
- Two mixture components have positive weights. One gives a zero likelihood contribution and the other gives a positive contribution. Must the marginal likelihood be zero? Explain.
- Why should a small positive constant not automatically replace a zero contribution?
- Eight independent contributions are each \(10^{-50}\). Give the log of their product to three decimal places.
Solutions. 1. The contribution is \(p^0(1-p)^1=1(0)=0\), so its log is negative infinity. 2. Evaluate the log-density directly and inspect its inputs; the reported zero alone does not establish a structural zero because underflow may have occurred. 3. No. The marginal is a weighted sum, and the positive component gives a positive term. 4. A floor changes the likelihood and can hide a support, prediction, or numerical defect; diagnose the source first. 5. The log product is \(8\log(10^{-50})=-400\log(10)\approx -921.034\).