Checking Predicted BQL Frequencies

Why Check Predicted BQL Frequencies?

A censored-data model assigns a probability to each BQL record: given the model and the sampling conditions, how likely is a concentration to fall below the assay’s lower limit of quantification? Checking predicted BQL frequencies asks whether those probabilities agree, in aggregate, with how often BQL results are actually observed. The check is useful because a model can produce plausible parameter estimates yet systematically underpredict or overpredict censoring in a particular part of the design.

The comparison is most informative when organized by relevant features of the data: sampling time, dose, and assay, for example. A model might match the overall fraction of BQL results while missing the terminal phase, or while predicting too few BQL results for one assay. Such patterns can point to where the model’s predictions deserve closer review. They do not, by themselves, identify the cause of a discrepancy.

Definition: For record \(i\), let \(B_i=1\) if the result is BQL and \(B_i=0\) otherwise. Let \(p_i\) be the model-predicted probability of a BQL result under that record’s covariates, sampling time, assay LLOQ, and the specified model parameters. For a stratum \(g\) containing \(n_g\) records, the observed BQL proportion is \(\hat p_{BQL,g}=O_g/n_g\), where \(O_g=\sum_{i\in g}B_i\). The expected BQL count is \(E_g=\sum_{i\in g}p_i\).

What Probability Is Being Compared?

The prediction must correspond to the measurement process that generated the BQL indicator. Suppose a positive concentration \(C_i\) has a model-predicted distribution and the applicable assay has LLOQ \(L_i\). Then, under the convention that a result is BQL when it is below the LLOQ,

p_i=P(C_i<L_i\mid\text{model and record conditions}).

For a lognormal observation model, one possible specification is \(\log C_i\sim N(\log f_i,\sigma^2)\), where \(f_i\) is the model’s typical concentration for the specified individual and conditions, and \(\sigma\) is the residual standard deviation on the log scale. In that case,

p_i=\Phi\left(\frac{\log L_i-\log f_i}{\sigma}\right),

where \(\Phi\) is the standard normal cumulative distribution function. This expression assumes that the stated lognormal distribution is the relevant distribution for the BQL event. If individual random effects are part of the population prediction, the probability must also average over their population distribution. If residual error is defined on a different scale or has a different distribution, the corresponding probability calculation changes.

Use the LLOQ applicable to each record, not a single pooled limit when assays have different limits. Predictions should also use the same conventions for concentration scale, residual error, and random effects as the model being checked. For a population-level check, probabilities generally represent population predictions with random effects integrated over their distribution. A prediction conditioned on an individual’s estimated random effect answers a different question and may be influenced by the same observed concentrations being checked.

Expected Counts and Their Uncertainty

The expected count \(E_g\) is the sum of the record-specific probabilities; it need not be an integer. If all records in a stratum have the same predicted probability \(p\), then \(E_g=n_gp\), and the expected proportion is \(p\). With different probabilities, the expected proportion is \(E_g/n_g\), not necessarily the probability calculated at an “average” covariate value.

A simple uncertainty calculation is available when BQL indicators are independent Bernoulli outcomes with their specified probabilities. Because a Bernoulli variable with probability \(p_i\) has mean \(p_i\) and variance \(p_i(1-p_i)\), summing the indicators gives the following result.

Result: Under conditional independence of the BQL indicators, \(E(O_g)=E_g=\sum_{i\in g}p_i\) and \(\operatorname{Var}(O_g)=\sum_{i\in g}p_i(1-p_i)\). A descriptive standardized discrepancy is \(R_g=(O_g-E_g)/\sqrt{\sum_{i\in g}p_i(1-p_i)}\), provided the denominator is positive and the independence assumption is reasonable.

This standardized discrepancy is a diagnostic scale, not an automatic pass-or-fail rule. A value near zero means observed and expected counts are close relative to this model-based standard deviation. A larger positive value means more BQL results were observed than expected; a larger negative value means fewer. Formal reference distributions and thresholds require additional assumptions, especially when many strata are examined.

Repeated concentrations from the same person are commonly dependent through shared random effects. In that setting, the variance of the total also contains covariance terms:

\operatorname{Var}(O_g) =\sum_{i\in g}p_i(1-p_i) +2\sum_{i<j,\ i,j\in g}\operatorname{Cov}(B_i,B_j).

Ignoring positive within-person covariance can make the independent-record standard deviation too small, so a discrepancy may appear more striking than it is. The expected count remains the sum of the marginal probabilities, but uncertainty should reflect the dependence structure when an inferential comparison is intended. In routine diagnostics, report the aggregation and its limitations clearly rather than presenting an independence-based standardized value as definitive evidence.

Worked Example: Translating a Concentration Prediction Into a BQL Probability

Hypothetical data. In a specified sampling stratum, a model predicts a typical concentration \(f=0.080\) mg/L. Assume the lognormal observation model \(\log C\sim N(\log f,0.50^2)\), and the assay LLOQ is \(L=0.050\) mg/L. The model parameters and this distribution are treated as fixed for this calculation.

The predicted probability is

p=P(C<0.050) =\Phi\left(\frac{\log(0.050)-\log(0.080)}{0.50}\right) =\Phi\left(\frac{\log(0.625)}{0.50}\right) =\Phi(-0.9400) \approx 0.1736.

Thus the model predicts a BQL probability of about 17.36% for each record with these same conditions. If 30 independent hypothetical records have this probability, the expected number of BQL results is \(30(0.1736)=5.208\). Under the independence assumption, the variance is \(30(0.1736)(1-0.1736)=4.304\), and the standard deviation is \(\sqrt{4.304}=2.075\).

Suppose 11 of the 30 records are observed BQL. The observed proportion is \(11/30=0.3667\), or 36.67%, and the descriptive standardized discrepancy is \((11-5.208)/2.075=2.791\). This is a notable excess relative to the stated independent-record calculation, but it does not establish why the excess occurred. If the 30 records include repeated samples from fewer individuals, that calculation does not account for within-person dependence.

Stratify Before Interpreting the Overall Fit

A pooled comparison can hide offsetting discrepancies. For instance, too few predicted BQL results in one time range and too many in another may yield a satisfactory total. Stratification should follow the scientific and design questions: time windows that distinguish absorption from terminal sampling, dose groups, and assay groups with different LLOQs are common choices. A stratum needs enough records to make its observed proportion interpretable; very small groups can show large percentage changes from only one or two records.

When comparing strata, show both observed and expected quantities, preferably counts as well as proportions. Counts reveal the amount of information behind a percentage, while the expected count makes clear that predictions may differ from record to record. If several dimensions are relevant, begin with transparent groupings and inspect whether discrepancies persist after separating plausible sources of variation. Excessively fine stratification can create sparse, noisy summaries.

Worked Example: Finding an Assay-Specific Discrepancy

Hypothetical data. Consider two groups of 30 records sampled in the same terminal-phase window at the same dose. Assay A has a model-predicted BQL probability of 0.60 for each record; assay B has probability 0.35, reflecting its own LLOQ and the model predictions. There are 20 observed BQL results with assay A and 19 with assay B. For this illustrative calculation, assume independent BQL indicators within each group.

For assay A, \(E_A=30(0.60)=18\), and \(\hat p_{BQL,A}=20/30=0.6667\). Its variance is \(30(0.60)(0.40)=7.200\), so the standard deviation is \(\sqrt{7.200}=2.683\). The standardized discrepancy is \((20-18)/2.683=0.745\).

For assay B, \(E_B=30(0.35)=10.50\), and \(\hat p_{BQL,B}=19/30=0.6333\). Its variance is \(30(0.35)(0.65)=6.825\), giving a standard deviation of \(\sqrt{6.825}=2.612\). The standardized discrepancy is \((19-10.50)/2.612=3.254\).

The assay A result is close to its expected count on this descriptive scale, whereas assay B has substantially more BQL results than predicted. Pooling the groups would give 39 observed BQL results out of 60, versus an expected total of 28.50; the pooled summary would indicate an excess but obscure that it is concentrated in assay B. The pattern warrants checking whether the assay-specific LLOQ and record assignments are represented correctly and whether the model’s predictions are adequate for those records. It does not prove that the assay is at fault.

Unequal Probabilities and a Short Justification

A stratum may combine records with different predicted probabilities because concentrations, covariates, doses, or LLOQs differ. Averaging probabilities is appropriate for the expected proportion, but the uncertainty calculation should retain the individual probabilities. The reason follows directly from the variance of a sum of independent indicators.

Represent each record.
For each record, \(B_i\) is Bernoulli with \(P(B_i=1)=p_i\), so \(E(B_i)=p_i\) and \(\operatorname{Var}(B_i)=p_i(1-p_i)\).

Sum the indicators.
The total count is \(O_g=\sum_{i\in g}B_i\), so linearity of expectation gives \(E(O_g)=\sum_{i\in g}p_i\).

Use independence for the variance.
If the indicators are independent, the variance of their sum is the sum of their variances: \(\operatorname{Var}(O_g)=\sum_{i\in g}p_i(1-p_i)\).

Worked Example: Avoiding an Average-Probability Shortcut

Hypothetical data. A stratum contains 10 independent records: five have predicted BQL probability 0.10 and five have probability 0.50. Six records are observed BQL. The average predicted probability is \((5(0.10)+5(0.50))/10=0.30\).

The expected count is \(5(0.10)+5(0.50)=3.00\). The correct independent-record variance is \(5(0.10)(0.90)+5(0.50)(0.50)=0.450+1.250=1.700\), so the standard deviation is \(\sqrt{1.700}=1.304\). The standardized discrepancy is \((6-3.00)/1.304=2.301\).

If all records were incorrectly treated as having probability 0.30, the variance would be \(10(0.30)(0.70)=2.100\), and the standardized discrepancy would instead be \(3.00/\sqrt{2.100}=2.070\). The expected count happens to be unchanged, but the uncertainty calculation is not. Keeping the record-specific probabilities preserves the heterogeneity that the model actually predicts.

Reading Patterns and Recognizing Failure Modes

A discrepancy is most useful when interpreted by its direction, location, and persistence. More observed BQL results than predicted in late samples could be consistent with concentrations declining faster than the model predicts, a mismatch in residual variability, or an issue in how observations or LLOQs were assigned. Fewer observed BQL results than predicted could indicate the opposite pattern. These are hypotheses to investigate, not conclusions established by the frequency check.

Consider the following features when reviewing results:

Time pattern: repeated underprediction late in the profile differs from an isolated difference during early absorption. Check whether the pattern is consistent across subjects and dose groups.
Dose pattern: a discrepancy confined to one dose group may reflect dose-related model misspecification or differences in the data represented there; a pooled proportion cannot distinguish these possibilities.
Assay pattern: verify that each record was compared with its applicable LLOQ. A wrong assay assignment can change predicted BQL probabilities even if the concentration model is unchanged.
Repeated records: identify whether multiple observations from the same individual are treated as independent in any uncertainty calculation. Shared random effects can induce association among BQL indicators.
Sparse groups: present counts alongside proportions. A change from one BQL result to two can look dramatic in a very small stratum without providing stable evidence of miscalibration.
Fitted-data optimism: when parameters are estimated from the same records, agreement in those records is not an independent validation. The check remains descriptive and should be interpreted with that limitation in mind.

Frequency summaries complement, rather than replace, other diagnostics. A single pooled observed-versus-expected count is easy to report but can conceal structure. A table of stratified proportions is more interpretable for categorical dimensions, while a plot of observed and expected frequencies across ordered time windows can make trends visible. Record-level residual diagnostics can help localize patterns, but they answer related rather than identical questions. No summary alone establishes that the full concentration distribution is correct: matching the probability of crossing the LLOQ does not guarantee accurate predictions among quantified concentrations.

Common Mistake and Practical Limits

Common pitfall: “The pooled observed BQL percentage is close to the pooled predicted percentage, so the model is calibrated everywhere.” This does not follow. Opposing errors across times, doses, or assays can cancel in the pooled total. Compare observed and expected frequencies within meaningful strata, and inspect the direction of each discrepancy before drawing an overall conclusion.

There is no universal cutoff that turns a stratified discrepancy into proof of model failure. The reliability of a comparison depends on group size, uncertainty in the fitted model, the way repeated observations are handled, and how many strata are inspected. A large difference may reflect a meaningful systematic pattern, but it may also arise from sparse data or unaccounted dependence. Conversely, close observed and expected counts can occur even when other parts of the concentration model are inadequate.

The practical goal is therefore calibrated interpretation: state the observed count and proportion, the expected count or proportion, the stratum definition, and the assumptions behind any uncertainty summary. Then describe what the pattern suggests checking next without claiming that the frequency comparison identifies a unique cause.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

A group has 25 independent records, each with predicted BQL probability 0.20. If 8 are observed BQL, calculate the expected count, observed proportion, and independent-record variance.
Why should an assay-specific LLOQ be used when calculating a predicted BQL probability?
A group contains repeated observations from the same individuals. What assumption is needed for the simple variance formula, and what may happen if positive within-person dependence is ignored?
Two time strata have observed counts that are respectively above and below their expected counts, while the pooled count matches expectation. What does the pooled match establish?

Solutions. (1) \(E=25(0.20)=5.00\); the observed proportion is \(8/25=0.3200\); the variance under independence is \(25(0.20)(0.80)=4.000\). (2) The BQL event is defined relative to the limit for the assay that measured that record, so using a different limit gives the wrong event probability. (3) The simple variance formula assumes independent BQL indicators. Ignoring positive within-person dependence can underestimate the variance and make a discrepancy look more extreme. (4) It establishes only that the aggregate observed count matches the aggregate expected count; it does not show that predictions are calibrated in either time stratum.