Below-LLOQ Probabilities Under Residual Error Models

Why a Prediction Does Not Determine BQL Status

A model prediction gives the expected observation under a specified set of conditions; it does not determine whether a particular assay result will be below the lower limit of quantification (LLOQ). Residual error represents the variation of an observation around that prediction. Consequently, even when a prediction is above the LLOQ, there may be a nonzero probability that the observation falls below it. Conversely, a prediction below the LLOQ does not mean every observation must be below it.

The relevant probability depends on both the distance from the prediction to the LLOQ and the residual-error model. This tutorial calculates that probability conditional on a prediction. It is distinct from the probability of a censored observation used in a censored likelihood, although the same model probability can supply that likelihood contribution.

Definition: Let \(Y\) be an observation, \(f\) its model prediction conditional on the relevant time, covariates, individual effects, and model parameters, and \(L\) the LLOQ on the same measurement scale as \(Y\) and \(f\). The conditional below-LLOQ probability is \(P(Y<L\mid f)\), calculated from the specified residual-error distribution. If the model parameters are estimated, a plug-in calculation uses their fitted values and does not by itself account for their uncertainty.

The formulas below assume a continuous observation distribution, so the probability of exactly \(Y=L\) is zero. They also assume that the LLOQ and the model prediction are expressed in compatible units. The standard normal cumulative distribution function is written \(\Phi(z)=P(Z\leq z)\), where \(Z\sim N(0,1)\).

Conditional Probabilities for Three Residual Models

Under an additive normal residual-error model, the residual has a constant standard deviation in the units of the observation. Under a proportional model, the residual standard deviation scales with the prediction. A combined model includes both sources of variation. The following expressions are conditional on \(f\); thus, they describe residual variation for a fixed prediction, not variation in predictions across individuals.

Model definitions: For additive error, \(Y=f+\epsilon_{\text{add}}\), where \(\epsilon_{\text{add}}\sim N(0,\sigma_{\text{add}}^2)\). For proportional error, \(Y=f(1+\epsilon_{\text{prop}})\), where \(\epsilon_{\text{prop}}\sim N(0,\sigma_{\text{prop}}^2)\). For combined error, \(Y=f+\epsilon_{\text{add}}+f\epsilon_{\text{prop}}\), where the two error terms are independent and normally distributed as just specified.

The proportional error parameter \(\sigma_{\text{prop}}\) is dimensionless; \(\sigma_{\text{add}}\) has the units of \(Y\). These models are centered at \(f\), so \(E(Y\mid f)=f\). For positive \(f\), the conditional standard deviations are \(\sigma_{\text{add}}\) for the additive model and \(f\sigma_{\text{prop}}\) for the proportional model. The combined-model variance follows from independence: the variance of a sum of independent terms is the sum of their variances.

\begin{aligned} P(Y<L\mid f,\text{ additive}) &=\Phi\!\left(\frac{L-f}{\sigma_{\text{add}}}\right),\\ P(Y<L\mid f,\text{ proportional}) &=\Phi\!\left(\frac{L-f}{f\sigma_{\text{prop}}}\right),\qquad f>0,\\ P(Y<L\mid f,\text{ combined}) &=\Phi\!\left(\frac{L-f}{\sqrt{\sigma_{\text{add}}^2+(f\sigma_{\text{prop}})^2}}\right). \end{aligned}

Each expression standardizes the distance from the prediction to the threshold by the conditional standard deviation. For example, in the additive model, \((Y-f)/\sigma_{\text{add}}\sim N(0,1)\); therefore \(Y<L\) is equivalent to \((Y-f)/\sigma_{\text{add}}<(L-f)/\sigma_{\text{add}}\), giving the first formula. The proportional formula follows by standardizing \(Y-f=f\epsilon_{\text{prop}}\), and the combined formula follows because the sum of independent normal errors is normal with variance equal to the sum of their variances.

If \(f=L\), the probability is \(0.5\) for each of these continuous, centered normal models. If \(f\) is above \(L\), the probability is below \(0.5\); if it is below \(L\), the probability is above \(0.5\). These checks are useful, but they do not replace evaluating the formula.

Worked Example: Additive Error With a Prediction Above the LLOQ

Suppose a hypothetical observation has prediction \(f=0.140\) mg/L, the LLOQ is \(L=0.100\) mg/L, and the additive residual standard deviation is \(\sigma_{\text{add}}=0.030\) mg/L. Assume the conditional residual is normal, centered at zero, and has the stated constant standard deviation. Then

P(Y<L\mid f)= \Phi\!\left(\frac{0.100-0.140}{0.030}\right) =\Phi(-1.333333\ldots) \approx 0.0912.

Thus, under this model and at this fixed prediction, the probability of an observation below the LLOQ is about 9.12%. The prediction is \(0.040\) mg/L above the limit, or about 1.333 residual standard deviations above it. This is a probability for a future observation conditional on the specified prediction and residual model, not a statement that the prediction itself has a 9.12% chance of being below the limit.

Worked Example: Proportional Error

For a hypothetical concentration, let \(f=1.20\) mg/L, \(L=0.80\) mg/L, and \(\sigma_{\text{prop}}=0.25\). Assume \(f>0\) and a centered normal proportional error on the original measurement scale. The residual standard deviation at this prediction is \(f\sigma_{\text{prop}}=(1.20)(0.25)=0.300\) mg/L. Therefore,

P(Y<L\mid f) =\Phi\!\left(\frac{0.80-1.20}{(1.20)(0.25)}\right) =\Phi(-1.333333\ldots) \approx 0.0912.

The below-LLOQ probability is about 9.12%. Although the proportional error parameter is 0.25, it is not a standard deviation of \(0.25\) mg/L: it is a relative, dimensionless standard deviation. At this prediction it corresponds to a residual standard deviation of \(0.300\) mg/L. At a different prediction, the residual standard deviation and, generally, the below-LLOQ probability would change.

Worked Example: Combined Additive and Proportional Error

Suppose a hypothetical concentration has \(f=0.120\) mg/L and \(L=0.100\) mg/L. Let \(\sigma_{\text{add}}=0.015\) mg/L and \(\sigma_{\text{prop}}=0.20\). Assume both components are centered, normally distributed, and independent. The proportional component contributes a standard deviation of \((0.120)(0.20)=0.024\) mg/L. The combined variance and standard deviation are

\begin{aligned} \operatorname{Var}(Y\mid f) &=(0.015)^2+(0.024)^2\\ &=0.000225+0.000576=0.000801\;(\text{mg/L})^2,\\ SD(Y\mid f)&=\sqrt{0.000801}=0.0283019\ \text{mg/L}. \end{aligned}

The standardized threshold is \((0.100-0.120)/0.0283019=-0.70667\), so

P(Y<L\mid f)=\Phi(-0.70667)\approx 0.2399.

The conditional probability is about 24.0%. The combined standard deviation is not the sum \(0.015+0.024\): variances, not standard deviations, add for independent components. Here the prediction is only about 0.707 combined standard deviations above the LLOQ, which explains why the below-limit probability is appreciably larger than in the first two examples.

Individual Predictions and Population-Average Probabilities

In an NLME model, a prediction can depend on an individual's random effects. Conditional on those effects and the model parameters, the residual-error formulas give an individual's below-LLOQ probability. A population-average probability is a different quantity: it averages the conditional probability over the distribution of individual predictions. In general, it is not correct to substitute an average prediction into the conditional formula, because the probability is a nonlinear function of \(f\), and proportional and combined error scales also depend on \(f\).

Worked Example: Why Averaging Predictions Can Mislead

Consider a hypothetical population with two equally common prediction groups: half have \(f=0.080\) mg/L and half have \(f=0.160\) mg/L. Let \(L=0.100\) mg/L and assume additive normal residual error with \(\sigma_{\text{add}}=0.020\) mg/L for both groups. The conditional probabilities are

\begin{aligned} P(Y<L\mid f=0.080) &=\Phi\!\left(\frac{0.100-0.080}{0.020}\right) =\Phi(1)\approx0.8413,\\ P(Y<L\mid f=0.160) &=\Phi\!\left(\frac{0.100-0.160}{0.020}\right) =\Phi(-3)\approx0.00135. \end{aligned}

The population-average probability is the equally weighted average of those conditional probabilities:

$$ 0.5(0.8413)+0.5(0.00135)=0.4213. $$

The average prediction is \(0.5(0.080)+0.5(0.160)=0.120\) mg/L. Applying the formula only at that average prediction gives \(\Phi((0.100-0.120)/0.020)=\Phi(-1)\approx0.1587\), which is not the population-average probability of 0.4213. In an actual NLME analysis, the population average is obtained by integrating the conditional probability over the random-effects distribution, or by an appropriately weighted numerical approximation.

Assumptions, Diagnostics, and Alternatives

Before interpreting a calculated probability, verify the assumptions that determine its error scale and distribution:

Correct residual model: the residuals have the specified conditional mean and distribution. The formulas above require normal errors on the stated observation or relative-error scale.
Correct scale and units: \(f\), \(L\), and additive error must use the same measurement units. The proportional error scale is dimensionless.
Combined-error independence: the combined formula assumes the additive and proportional components are independent. If they are correlated, a covariance term is needed in the variance.
Fixed conditional prediction: the formula conditions on \(f\) and fitted model quantities. It does not automatically include uncertainty in parameter estimates or average over random effects.
Positive prediction for proportional error: the usual proportional model and its standardization require \(f>0\). At \(f=0\), this model has zero residual variation and cannot represent an additive measurement error.

Diagnostics should assess whether the chosen residual model is plausible over the prediction range, not merely whether a single calculated probability seems reasonable. Examine residual patterns against predictions and time, check whether residual spread is roughly constant for additive error or increases with prediction for proportional error, and assess whether the fitted distribution captures the tails relevant to the LLOQ. Compare observed BQL frequencies with model-predicted frequencies in suitable prediction or time groups, while recognizing that observed frequencies also vary by chance. A systematic mismatch can indicate an unsuitable error model, an incorrect scale, an incorrectly recorded LLOQ, or model misspecification in the predictions.

The normal formulas can assign probability to negative observations. For a concentration, that may be physically unrealistic, but it does not invalidate the algebra; rather, it may signal that the assumed normal error model is inappropriate in the relevant range. If the residual distribution is not normal, replace \(\Phi\) with the cumulative distribution function of the specified standardized error, using the appropriate location and scale. A simulation approach can approximate the same probability by generating residual errors at fixed \(f\) and calculating the fraction of simulated observations below \(L\). Simulation is useful for more complicated distributions, but it does not repair a misspecified model and introduces Monte Carlo error.

Approach	What it calculates	Key qualification
Conditional formula	Below-LLOQ probability for a fixed prediction	Requires a specified residual distribution and scale
Population integration	Average probability over the distribution of predictions or random effects	Must average probabilities, not generally evaluate at the mean prediction
Simulation	Numerical estimate of the probability from generated observations	Depends on the simulation model and has finite-simulation error
Prediction-versus-LLOQ comparison	Whether the prediction itself is above or below the limit	Does not calculate the probability of an observation being below the limit

Common Mistakes and Practical Interpretation

A frequent error is to use \(f-L\) in the numerator when calculating \(P(Y<L\mid f)\). The correct standardized threshold is \(L-f\), because the event of interest is below the limit. If \(f>L\), this numerator is negative, and the resulting probability must be below 0.5 under a centered normal model. This sign check can catch an arithmetic or coding error.

Another common mistake is to use the proportional coefficient directly as an absolute standard deviation. The correct proportional standard deviation is \(f\sigma_{\text{prop}}\). For combined independent errors, use \(\sqrt{\sigma_{\text{add}}^2+(f\sigma_{\text{prop}})^2}\), not the sum of standard deviations. Finally, do not interpret \(P(Y<L\mid f)\) as the probability that \(f<L\), or as a probability that a parameter is true. It is a conditional model probability for an observation, under the assumptions stated.

These calculations are useful for understanding how residual error and prediction jointly determine the chance of an unquantifiable result. Their value is greatest when the model, measurement scale, and LLOQ are aligned and the conditional interpretation is kept explicit.

Check Your Understanding

For each question, calculate or explain the requested quantity. Complete solutions follow each question.

An additive normal-error model has \(f=0.090\) mg/L, \(L=0.100\) mg/L, and \(\sigma_{\text{add}}=0.020\) mg/L. What is the below-LLOQ probability?
A proportional-error model has \(f=2.00\) mg/L, \(L=1.50\) mg/L, and \(\sigma_{\text{prop}}=0.10\). Calculate the conditional probability and identify the residual standard deviation.
For an independent combined model, \(f=0.050\) mg/L, \(L=0.040\) mg/L, \(\sigma_{\text{add}}=0.010\) mg/L, and \(\sigma_{\text{prop}}=0.20\). Calculate the combined standard deviation and below-LLOQ probability.
Why is evaluating the conditional probability at the average prediction generally not a valid way to obtain a population-average probability?

Solutions. (1) \(z=(0.100-0.090)/0.020=0.500\), so \(P(Y<L\mid f)=\Phi(0.500)\approx0.6915\), or about 69.15%. (2) The residual standard deviation is \(f\sigma_{\text{prop}}=(2.00)(0.10)=0.200\) mg/L. Thus \(z=(1.50-2.00)/0.200=-2.500\), and the probability is \(\Phi(-2.500)\approx0.00621\), or about 0.621%. (3) The proportional standard deviation is \((0.050)(0.20)=0.010\) mg/L. The combined standard deviation is \(\sqrt{(0.010)^2+(0.010)^2}=0.01414\) mg/L. Then \(z=(0.040-0.050)/0.01414\approx-0.7071\), giving \(\Phi(-0.7071)\approx0.2398\), or about 23.98%. (4) The conditional probability is a nonlinear function of the prediction, and with proportional or combined error its scale also changes with the prediction. Therefore, average conditional probabilities over the prediction distribution; in general, that average is not the probability evaluated at the average prediction.