Censoring Under Proportional Residual Error

Why Proportional Error Changes BQL Probabilities

A below-quantification-limit (BQL) result is an observation known to fall below an assay threshold, not a measured concentration that should be treated as an exact value. Its likelihood contribution therefore depends on the probability assigned to the region below the threshold by the observation model. Under proportional residual error, the residual standard deviation changes with the model-predicted concentration. Consequently, the BQL probability is not calculated in the same way as it is under additive error.

Let \(f_{ij}=f(t_{ij},\theta,\eta_i)\) denote the model-predicted concentration for individual \(i\) at observation time \(t_{ij}\), conditional on population parameters \(\theta\) and individual random effects \(\eta_i\). Let \(L_{ij}>0\) be the known LLOQ applying to that observation. The derivation below is conditional on \(\eta_i\); the population likelihood must also account for the distribution of the random effects.

This model assumes that the residual errors have the specified normal distribution and, when the usual conditionally independent observation likelihood is used, are independent across observations given the individual effects and model predictions. The prediction must be positive for the standard deviation \(\sigma f_{ij}\) to be positive. The model also uses one relative error scale \(\sigma\); it does not describe data whose relative variability changes systematically with concentration unless that feature is modeled separately.

Deriving the Conditional BQL Probability

A BQL result records the event \(Y_{ij}<L_{ij}\). Under the proportional model, divide both sides by the positive prediction \(f_{ij}\), then standardize the residual error. The resulting probability is a normal cumulative probability.

Here \(\Phi\) is the standard normal cumulative distribution function. The last expression is also obtained by taking the conditional normal distribution of \(Y_{ij}\) and standardizing its threshold using its standard deviation, \(\sigma f_{ij}\). With a continuous normal distribution, using \(Y_{ij}\le L_{ij}\) instead of \(Y_{ij}<L_{ij}\) gives the same probability.

The formula is conditional: it gives the BQL probability at a particular value of \(\eta_i\). In a nonlinear mixed-effects model, the individual effects are generally not known. The subject-level likelihood therefore averages the conditional likelihood over their distribution; substituting a population-average prediction into the conditional formula is not generally equivalent.

Likelihood Contributions and Random Effects

A quantified result contributes a density, whereas a BQL result contributes the probability of the censored region. For a quantified concentration \(y_{ij}\), the conditional density under proportional error is

If \(b_{ij}=1\) denotes a BQL record and \(b_{ij}=0\) a quantified record, the conditional contribution can be written as

The subject’s conditional likelihood is the product of these contributions over that subject’s observations when residual errors are conditionally independent. If \(g_\eta(\eta_i\mid\Omega)\) is the random-effects distribution, the marginal subject likelihood is the integral of that conditional likelihood against \(g_\eta\). This distinction matters: an observed BQL record contributes its conditional probability at each possible individual effect, averaged according to the model, rather than a probability evaluated only at an average effect.

Low-Concentration Behavior and Comparison with Additive Error

For proportional error and a positive LLOQ, the standardized threshold can be written as \(z=(L/f-1)/\sigma\). As \(f\) approaches zero from above, \(L/f\) grows without bound, so \(z\) approaches infinity and the BQL probability approaches one. This follows from the normal CDF limit \(\Phi(z)\to1\) as \(z\to\infty\). At \(f=L\), the probability is exactly \(0.5\); for a fixed positive \(L\), it decreases as \(f\) increases.

Under additive normal error with standard deviation \(\sigma_a>0\), the observation model is \(Y\mid\eta\sim N(f,\sigma_a^2)\). Its BQL probability is \(\Phi((L-f)/\sigma_a)\). As \(f\) approaches zero, this tends to \(\Phi(L/\sigma_a)\), which is less than one for finite positive \(L\) and \(\sigma_a\). Thus additive error retains a fixed absolute spread at low predictions, while proportional error’s spread shrinks with the prediction.

Conditions, Diagnostics, and Common Mistakes

The proportional normal model is mathematically a normal distribution for any finite positive \(f\) and \(\sigma\), but its suitability for concentration data requires attention. A normal distribution has support over the entire real line. Under the proportional model, the probability of a negative observation is

which does not depend on \(f\). For example, with \(\sigma=0.300\), this probability is \(\Phi(-3.333)\approx0.000429\). If the fitted relative error is large enough that negative values are not negligible, a normal proportional model may be a poor description of a nonnegative assay measurement. Also, at exactly \(f=0\), the model \(Y=f(1+\epsilon)\) collapses to a point mass at zero: it no longer has the ordinary positive-variance normal density used for quantified observations. The standard likelihood calculations therefore require strictly positive predictions for observations modeled with a continuous proportional normal distribution.

Check that the prediction is positive at every observation and that the LLOQ supplied to the likelihood is the threshold that actually applies to that record. Examine whether a single relative error scale is plausible across the concentration range and whether the implied negative-value probability is acceptable. A fit or implementation that produces nonpositive predictions, a zero residual scale, or an invalid standard deviation cannot use the stated density and BQL formula as written.

A common mistake is to calculate the proportional-error probability as \(\Phi((L-f)/\sigma)\). That expression uses \(\sigma\) as though it were an additive standard deviation in concentration units. Under proportional error, the standard deviation is \(\sigma f\), so the correct expression is \(\Phi((L-f)/(\sigma f))\). A related mistake is to treat the LLOQ itself as the observed concentration; the BQL contribution is the probability of the entire event below the LLOQ, not a density evaluated at the threshold.

The main alternative considered here is additive residual error. It is appropriate to compare the implied absolute variability and BQL probabilities over the concentration range, while recognizing that the two models make different assumptions about how variability scales. Neither model should be selected solely because one gives a preferred BQL rate at a single prediction. Proportional error can be unsuitable when absolute assay variability does not shrink at low concentrations; additive error can be unsuitable when variability is primarily relative. The error model is an assumption about the conditional observation distribution, not a device for changing the observed censoring status.