Censoring Under Combined Residual Error

Why Combine Additive and Proportional Error?

Residual variability can behave differently across a concentration profile. At higher concentrations, measurement or model error may increase with the predicted concentration; near zero, a nonzero amount of variability may remain even when the prediction is small. A combined additive-and-proportional residual error model represents both patterns. It also gives a direct way to calculate the probability that an observation will fall below the assay's lower limit of quantification (LLOQ).

A BQL result records an event, not an exact concentration: the observation is below the LLOQ. The probability of that event depends on the model-predicted concentration, the LLOQ, and the residual variability. Under a combined model, both error components enter that probability through the total residual standard deviation.

Definition: Let $Y_{ij}$ be the observation for subject $i$ at sample $j$, and let $f_{ij}$ be the model-predicted concentration for that observation, conditional on the subject's model parameters. Under a combined additive-and-proportional normal residual error model, $$ Y_{ij}=f_{ij}+\epsilon_{ij}, \qquad \epsilon_{ij}\sim N(0,\sigma_{\mathrm{add}}^2+\sigma_{\mathrm{prop}}^2 f_{ij}^2). $$ Here $\sigma_{\mathrm{add}}$ has concentration units, while $\sigma_{\mathrm{prop}}$ is dimensionless. Equivalently, the residual can be written as the sum of independent additive and proportional normal components. The conditional BQL probability at LLOQ $L$ is $$ P(Y_{ij}<L\mid f_{ij})= \Phi\left(\frac{L-f_{ij}}{\sqrt{\sigma_{\mathrm{add}}^2+\sigma_{\mathrm{prop}}^2 f_{ij}^2}}\right), $$ where $\Phi$ is the standard normal cumulative distribution function.

Assumptions and the Probability Calculation

The formula applies when the residual, conditional on the model prediction and subject parameters, is normally distributed with mean zero and the stated variance. It assumes the additive and proportional error components are independent if the model is expressed as their sum. It also assumes the assay limit $L$ is known and that a BQL record means exactly that the measured observation fell below this threshold. The model prediction and both error parameters must use compatible concentration units.

The probability is conditional on $f_{ij}$. In an NLME model, that prediction may depend on subject-specific parameters. If the question concerns a particular subject with specified parameters, use that subject's prediction. A population-marginal probability instead averages the conditional probability over the distribution of subject-specific parameters; it is not generally obtained by simply inserting a population-average prediction into the conditional formula.

To derive the expression, write the conditional standard deviation as $s_{ij}=\sqrt{\sigma_{\mathrm{add}}^2+\sigma_{\mathrm{prop}}^2f_{ij}^2}$. Standardizing the observation gives $Z=(Y_{ij}-f_{ij})/s_{ij}\sim N(0,1)$. Therefore,

\begin{aligned} P(Y_{ij}<L\mid f_{ij}) &=P\left(\frac{Y_{ij}-f_{ij}}{s_{ij}}< \frac{L-f_{ij}}{s_{ij}}\ \middle|\ f_{ij}\right)\\ &=\Phi\left(\frac{L-f_{ij}}{s_{ij}}\right). \end{aligned}

The numerator indicates where the prediction lies relative to the LLOQ; the denominator scales that distance by residual variability. If $f_{ij}=L$, the probability is $0.5$ under this symmetric normal model. If $f_{ij}<L$, the probability exceeds $0.5$; if $f_{ij}>L$, it is below $0.5$. These comparisons assume the same conditional normal model and do not imply that an individual observation must fall on the same side of the limit as its prediction.

Worked Example: Prediction Above the LLOQ

Suppose a hypothetical sample has prediction $f=0.80$ mg/L, assay limit $L=0.50$ mg/L, additive error $\sigma_{\mathrm{add}}=0.10$ mg/L, and proportional error $\sigma_{\mathrm{prop}}=0.20$. Calculate the conditional BQL probability.

First calculate the proportional contribution to the standard deviation: $\sigma_{\mathrm{prop}}f=0.20(0.80)=0.16$ mg/L. The total variance and standard deviation are $$ \sigma_{\mathrm{add}}^2+(\sigma_{\mathrm{prop}}f)^2 =(0.10)^2+(0.16)^2=0.0356\ (\mathrm{mg/L})^2, \qquad s=\sqrt{0.0356}=0.1887\ \mathrm{mg/L}. $$

Then standardize the LLOQ relative to the prediction: $$ z=\frac{L-f}{s}=\frac{0.50-0.80}{0.1887}=-1.589. $$ Thus $P(Y<L\mid f)=\Phi(-1.589)\approx0.0560$. Under the stated hypothetical model, the observation has about a 5.60% conditional probability of being BQL, despite its prediction being above the limit.

For comparison, additive error alone would give $z=(0.50-0.80)/0.10=-3.000$, with probability about $0.00135$. Proportional error alone would give $s=0.16$ mg/L, $z=-1.875$, and probability about $0.0304$. The combined variance is larger than either component's variance alone, so here the combined model gives the largest BQL probability.

How Each Component Affects Predictions Near the Limit

The additive component contributes the same variance, $\sigma_{\mathrm{add}}^2$, at every prediction. It therefore prevents residual variability from shrinking to zero as the predicted concentration approaches zero. The proportional component contributes variance $\sigma_{\mathrm{prop}}^2 f^2$, so its standard deviation is $\sigma_{\mathrm{prop}}f$ for positive predictions. Its contribution grows with concentration and diminishes as the prediction falls.

Neither component always increases the BQL probability. Adding variability moves a probability toward $0.5$ when the prediction is on one side of the threshold: it raises the BQL probability if $f>L$, and lowers it if $f<L$. This is why the same increase in residual variability can have different effects depending on the prediction's position relative to the LLOQ.

Worked Example: Prediction Just Below the LLOQ

Consider a hypothetical prediction $f=0.45$ mg/L, LLOQ $L=0.50$ mg/L, $\sigma_{\mathrm{add}}=0.10$ mg/L, and $\sigma_{\mathrm{prop}}=0.20$. The proportional standard deviation is $0.20(0.45)=0.090$ mg/L, so $$ s=\sqrt{(0.10)^2+(0.090)^2} =\sqrt{0.0181}=0.1345\ \mathrm{mg/L}. $$ The standardized threshold is $$ z=\frac{0.50-0.45}{0.1345}=0.3716, $$ and $P(Y<L\mid f)=\Phi(0.3716)\approx0.6449$.

With additive error alone, the standard deviation would be $0.10$ mg/L and the probability would be $\Phi(0.50)=0.6915$. With proportional error alone, the standard deviation would be $0.090$ mg/L and the probability would be $\Phi(0.5556)\approx0.7107$. Both components together produce more variability than either alone, moving the BQL probability closer to $0.5$ because the prediction lies below the LLOQ.

Worked Example: Comparing Error Components Very Near the LLOQ

Suppose a hypothetical prediction is $f=0.48$ mg/L with $L=0.50$ mg/L, $\sigma_{\mathrm{add}}=0.080$ mg/L, and $\sigma_{\mathrm{prop}}=0.25$. The proportional standard deviation is $0.25(0.48)=0.120$ mg/L. Combining the components gives $$ s=\sqrt{(0.080)^2+(0.120)^2} =\sqrt{0.0208}=0.1442\ \mathrm{mg/L}. $$ Therefore, $$ z=\frac{0.50-0.48}{0.1442}=0.1387,\qquad P(Y<L\mid f)=\Phi(0.1387)\approx0.5552. $$

For additive error alone, $z=0.020/0.080=0.2500$, giving $\Phi(0.2500)\approx0.5987$. For proportional error alone, $z=0.020/0.120=0.1667$, giving $\Phi(0.1667)\approx0.5662$. Both components make the total uncertainty larger, so the combined probability is nearer $0.5$ than either one-component probability. Since the prediction is below the limit, this larger spread lowers the probability of falling below it.

Using the Probability for a Censored Observation

When a sample is recorded as BQL, the data identify the event $Y_{ij}<L$, not a numerical value below $L$. Under the combined normal model, the event's likelihood contribution is the probability just calculated. For example, the first worked example contributes approximately $0.0560$ for a BQL record at the stated prediction and parameters. A quantified observation contributes a normal density evaluated at its observed concentration, with mean $f_{ij}$ and variance $\sigma_{\mathrm{add}}^2+\sigma_{\mathrm{prop}}^2f_{ij}^2$. The censored-likelihood material establishes this event-versus-value distinction; the key point here is that the combined error variance determines the probability assigned to the censored event.

This is different from replacing a BQL record by $L/2$ or by $L$. Substitution creates a numerical observation where the assay supplied only an inequality, and the substituted value is not the conditional probability of that inequality. Additive-only and proportional-only models are also alternatives, but each omits one source of variability. A different observation distribution can be considered when normal residuals are scientifically unsuitable; the normal formula above does not automatically apply to such a model.

Checks, Diagnostics, and Common Pitfalls

Before using a calculated BQL probability, check the inputs and the model. A negative or implausible prediction, incompatible units, or an incorrectly entered LLOQ can invalidate the calculation. Confirm that $\sigma_{\mathrm{add}}$ is expressed in concentration units and that $\sigma_{\mathrm{prop}}$ is dimensionless. Check that the two variance contributions, rather than the two standard deviations, are added. For a quick numerical check, the computed probability should be near $0.5$ when $f$ is near $L$, and should be below $0.5$ when $f>L$.

Compare predicted BQL frequencies with observed BQL patterns across time or prediction ranges as a model check. Systematic underprediction or overprediction of BQL records may indicate a poor concentration model, an unsuitable residual-error structure, an incorrect assay limit, or an unmodeled feature of the data. Such a discrepancy is a reason to investigate, not by itself proof that one particular error component is wrong. In particular, a single conditional probability does not account for between-subject variability unless that variability is integrated over for a population-marginal prediction.

Common mistake: Adding the additive and proportional standard deviations directly, or treating the proportional coefficient as a concentration-scale standard deviation. The combined model adds variances: $\sigma_{\mathrm{add}}^2+(\sigma_{\mathrm{prop}}f)^2$. The proportional contribution to the standard deviation is $\sigma_{\mathrm{prop}}f$, in concentration units. Another mistake is to assume that more residual variability always means more BQL probability. It raises the probability when the prediction is above the LLOQ but lowers it when the prediction is below the LLOQ, moving the probability toward $0.5$ in either case.

The combined model is useful when a constant-scale component and a concentration-dependent component are both plausible. Its limitations follow from its assumptions: conditional normality permits negative observations, and a single pair of error parameters may not describe every concentration range or assay condition. The BQL probability is therefore a model-based calculation, not a direct property of the assay independent of the model. The result should be interpreted alongside the residual-error specification, the prediction, and the threshold used to define BQL.

Check Your Understanding

Use the combined normal residual error model. Each problem uses hypothetical values; solutions are included below.

For $f=1.00$ mg/L, $L=0.80$ mg/L, $\sigma_{\mathrm{add}}=0.10$ mg/L, and $\sigma_{\mathrm{prop}}=0.15$, calculate the conditional BQL probability.
If $f=L$ and the combined residual standard deviation is positive, what is the conditional BQL probability under the model?
At a prediction below the LLOQ, what direction does increasing residual variability move the BQL probability, and why?
Why must a population-marginal BQL probability generally account for subject-specific parameter variability rather than simply use a population-average prediction?

Solutions. (1) The variance is $(0.10)^2+(0.15\times1.00)^2=0.0325\ (\mathrm{mg/L})^2$, so $s=0.1803$ mg/L. Then $z=(0.80-1.00)/0.1803=-1.109$, giving $\Phi(-1.109)\approx0.1337$. (2) The standardized threshold is zero, so $\Phi(0)=0.5000$. (3) It moves the probability toward $0.5$. Since $f<L$, the original probability exceeds $0.5$; increasing the standard deviation reduces the positive standardized distance $(L-f)/s$, lowering the probability. (4) The conditional probability depends on the subject-specific prediction, and averaging that probability over subjects is generally not the same as evaluating it at an average prediction because the formula is nonlinear in $f$.