Residual Variance From Assay Coefficients of Variation

From Assay Precision to Residual Variance

Assay performance is often reported as a coefficient of variation (CV), such as “the assay CV is 8% at 2 mg/L.” In an NLME model, however, the residual component is usually specified as a variance conditional on the model-predicted concentration. Converting a CV into that variance is straightforward once the scale and reference concentration are clear. The harder question is whether the reported assay CV describes the residual variation that the likelihood is meant to represent.

Let \(Y_{ij}\) be the measured concentration for subject \(i\) at observation \(j\), and let \(f_{ij}\) be the model-predicted concentration for that observation, conditional on the subject’s random effects. For a CV defined on the original concentration scale, the basic conversion uses the mean concentration as the denominator. In a model, the conditional mean \(f_{ij}\) supplies that reference value.

The conversion gives a variance function; it does not, by itself, specify a probability distribution. For example, a Gaussian residual model might use \(Y_{ij}\mid b_i\sim N(f_{ij},v(f_{ij}))\), where \(b_i\) denotes the subject-level random effects. That likelihood also assumes that the conditional mean is correctly specified and that the residual distribution is appropriate. A CV summary alone establishes neither Gaussianity nor independence.

Deriving and Using a CV-Based Function

When the assay CV varies with concentration, write it as a function \(q(c)\). Under an unbiased assay model, the conditional standard deviation at concentration \(c\) is \(q(c)c\). Squaring gives the residual variance function used in the likelihood. In an NLME model, this function is typically evaluated at the individual prediction \(f_{ij}\), not at the observed value \(Y_{ij}\).

This substitution treats the prediction as the conditional mean concentration. It is appropriate only if the assay CV applies to the relevant assay, concentration range, sample matrix, and laboratory conditions, and if the measurement error represented by that CV is the residual component being modeled. For a Gaussian likelihood, the residuals are additionally assumed to be conditionally Gaussian and independent across observations, unless the model specifies another dependence structure.

A useful special case is a power-shaped CV curve. To keep units explicit, choose a reference concentration \(c_0\), and express the curve as \(q(c)=q_0(c/c_0)^{-b}\), where \(q_0\) and \(b\) are dimensionless. The implied variance is then:

The exponent matters: a CV that falls as concentration increases does not imply that absolute residual variance must fall. Since the standard deviation is the CV multiplied by concentration, absolute variability can still increase, decrease, or remain roughly constant depending on the rate at which the CV changes.

Assumptions for Use in an NLME Likelihood

A residual variance function enters the conditional likelihood for each observation. For a Gaussian specification, the contribution depends on both the residual \(Y_{ij}-f_{ij}\) and the variance \(v(f_{ij})\). The model’s random effects are then accounted for according to the NLME likelihood. Consequently, a variance function calibrated from assay data is suitable only if the assay information is compatible with the model’s conditional measurement process.

• Comparable precision conditions. The CV must be relevant to the assay method, concentration range, sample matrix, and handling conditions represented in the modeled data. Precision from a different context may not transfer.
• Correct scale and definition. Confirm whether the CV is the arithmetic standard deviation divided by the arithmetic mean on the concentration scale. A CV defined after a logarithmic transformation is not converted by the same formula.
• Suitable mean reference. The conversion assumes that the assay mean at concentration \(c\) is \(c\), or that any assay bias has been handled. A systematic offset or concentration-dependent bias affects the mean model; it is not corrected by changing the variance alone.
• Appropriate residual interpretation. Assay precision must describe the observation-level error being modeled. It should not be assumed to represent every source of unexplained variability in the data.
• Distribution and dependence assumptions. A variance specifies only a second moment. A Gaussian likelihood additionally assumes conditional normality; independent residual contributions additionally require conditional independence. These assumptions need scientific and data-based support.

If the CV curve is estimated from a finite validation data set, its estimates are themselves uncertain. Treating an estimated curve as fixed in the NLME likelihood ignores that uncertainty. This may be a reasonable approximation when validation is extensive and precision estimates are stable, but it should be recognized as an approximation. A model that estimates residual parameters from the pharmacometric data instead answers a different question and may not reproduce the external assay information exactly.

Short Justification and Alternatives

The variance conversion follows directly from the definition of CV. If the conditional mean is \(c\) and the arithmetic CV is \(q(c)\), then \(q(c)=\operatorname{SD}(Y\mid c)/c\). Multiplying both sides by \(c\) gives \(\operatorname{SD}(Y\mid c)=q(c)c\). Variance is the square of standard deviation, so \(v(c)=[q(c)c]^2\). The conversion therefore requires a positive concentration reference; at or near zero, a CV may be undefined or unstable, and an additive component or another measurement model may be needed.

A CV-derived variance function is one option among several. Constant additive error specifies a variance that does not depend on prediction; constant-CV proportional error specifies a standard deviation proportional to prediction; combined additive and proportional error includes both a baseline and a prediction-dependent component. A power error model allows a more flexible concentration pattern. A directly specified assay CV curve is attractive when reliable validation data describe precision over the relevant range, but may be less stable if those data are sparse. The choice should reflect the measurement process and supported evidence, rather than a preference for a more elaborate formula.

The formula above is for an arithmetic-scale CV and variance. If assay precision is instead summarized on a log scale, or if a lognormal residual model is intended, the observation model and parameterization must be matched to that scale. Simply squaring an arithmetic CV and inserting it as a log-scale variance is generally incorrect. Likewise, an observed value should not ordinarily be substituted for \(f_{ij}\) to make the variance depend on the outcome itself: that changes the likelihood’s dependence on the data and does not represent the stated conditional measurement model.

Common Mistakes and Practical Checks

• Using a percentage as a proportion. Substituting 8 instead of 0.08 makes the standard deviation 100 times too large and the variance 10,000 times too large.
• Forgetting to square. The residual variance is \([q(c)c]^2\), not \(q(c)c\). The unsquared product is the standard deviation and has concentration units.
• Assuming a falling CV means falling variance. The variance depends on the square of the product \(q(c)c\); inspect that product rather than the CV alone.
• Treating a CV as a complete likelihood. A CV determines a scale, not the distribution shape, conditional independence, or absence of assay bias.
• Applying validation precision without checking comparability. A reported CV may not apply at the modeled concentrations or to the model’s samples and procedures.

A practical check is to compute \(q(f_{ij})f_{ij}\) at several representative predictions, verify the resulting units, and compare the implied residual standard deviations with assay knowledge. Values that are implausibly small, large, or sharply changing can signal a percentage conversion error, a poor interpolation or extrapolation, or a mismatch between the validation conditions and the modeled data. These checks do not replace assessment of the full observation model.