Separating Assay Replicate Error From Biological Variability

Why Replicates Matter

A measured concentration can vary because subjects differ biologically, because the same subject differs across occasions, and because the laboratory measurement is imprecise. Those sources have different implications: repeated assays of one specimen can quantify assay error, but they cannot reveal how much the subject or occasion itself varied. Conversely, repeated biological samples without replicate assays may confound laboratory error with biological variation.

The key design feature is therefore a hierarchy: several subjects, multiple biological occasions per subject, and multiple assays of the same specimen within an occasion. The last level provides a comparison in which the biological state is shared and only the assay measurement is repeated. This tutorial develops that decomposition on a specified response scale, such as concentration in mg/L.

A Hierarchical Observation Model

Let \(i\) index subjects, \(j\) index occasions within subject \(i\), and \(k\) index replicate assays of the specimen collected at that occasion. After accounting for the fitted mean response under the structural model and relevant covariates, write the response-scale deviation as

Y_{ijk} = \mu_{ij} + b_i + c_{ij} + e_{ijk}.

Here \(\mu_{ij}\) is the expected response for that subject-occasion sampling condition; \(b_i\) is a subject-level biological deviation; \(c_{ij}\) is a biological occasion deviation; and \(e_{ijk}\) is laboratory assay error for replicate \(k\). In an NLME application, subject and occasion effects may enter through model parameters and a nonlinear prediction rather than as literal additive response shifts. The additive expression is a variance-decomposition representation on the stated response scale, often applied after accounting for the fitted mean or using a suitable transformed scale.

Definition: In this hierarchical decomposition, subject variability is represented by \(b_i\), occasion variability by \(c_{ij}\), and assay replicate error by \(e_{ijk}\). A standard variance-components model assumes \(E(b_i)=E(c_{ij})=E(e_{ijk})=0\), mutually independent components, and variances \(\sigma_B^2\), \(\sigma_O^2\), and \(\sigma_A^2\), respectively. Assay errors from distinct replicates are independent and have the same variance when the assay conditions and response scale are comparable.

Under these assumptions, the total variance of one observation is \(\sigma_B^2+\sigma_O^2+\sigma_A^2\). The covariance depends on what two observations share. Two replicate assays of the same specimen share both subject and occasion effects, so their covariance is \(\sigma_B^2+\sigma_O^2\). Observations from different occasions of the same subject share only the subject effect, so their covariance is \(\sigma_B^2\). Observations from different subjects have covariance zero under the model.

This covariance pattern is the source of identification. Replicate assays separate the within-specimen measurement noise from the shared biological state; repeated occasions then distinguish occasion-level variation from persistent subject differences. If the mean response differs systematically by time, treatment, or other design features, those effects must be represented in \(\mu_{ij}\) before interpreting residual variation as these components.

Why Replicate Differences Isolate Assay Error

For two replicates \(k\) and \(l\) of the same specimen, subtract their observations. Every term shared within that specimen cancels:

Y_{ijk}-Y_{ijl} =(\mu_{ij}+b_i+c_{ij}+e_{ijk}) -(\mu_{ij}+b_i+c_{ij}+e_{ijl}) =e_{ijk}-e_{ijl}.

If the replicate errors are independent with variance \(\sigma_A^2\), the difference has mean zero and variance \(2\sigma_A^2\). Thus, across \(m\) independently replicated specimens, the mean squared replicate difference estimates twice the assay variance. This result relies on replicates genuinely sharing the same biological specimen or state; separately collected biological samples are not interchangeable with technical replicates.

E\left[\frac{1}{m}\sum_{r=1}^{m}(Y_{r1}-Y_{r2})^2\right] =2\sigma_A^2, \qquad \widehat{\sigma_A^2} =\frac{1}{2m}\sum_{r=1}^{m}(Y_{r1}-Y_{r2})^2.

The estimate above is a method-of-moments estimate for paired replicates. A variance estimate based on the sample variance of the replicate differences can instead be used when estimating their common mean rather than imposing a zero mean. With more than two replicates per specimen, within-specimen variation across all replicates can be used. In either case, the inferential target is the assay variance on the scale used for analysis.

Worked Example: Estimating Assay Variance From Paired Replicates

Suppose four hypothetical specimens are each assayed twice, in mg/L. Their paired results are \(4.8, 5.0\); \(7.1, 6.9\); \(3.6, 4.0\); and \(8.2, 7.8\). Assume each pair shares the same specimen state, and assay errors are independent with a common variance.

The replicate differences, first assay minus second assay, are \(-0.2,\ 0.2,\ -0.4,\ 0.4\) mg/L. Using the paired-difference estimator,

\widehat{\sigma_A^2} =\frac{(-0.2)^2+(0.2)^2+(-0.4)^2+(0.4)^2}{2(4)} =\frac{0.40}{8} =0.0500\;(\text{mg/L})^2.

The estimated assay standard deviation is \(\sqrt{0.0500}=0.224\) mg/L. This quantifies replicate-level assay imprecision for this hypothetical setting. It does not estimate between-subject or between-occasion variability, because those shared effects cancelled in every paired difference.

Separating Subject, Occasion, and Assay Components

In a balanced nested design, suppose there are \(n\) subjects, \(J\) occasions per subject, and \(K\) assay replicates per occasion. After accounting for the mean structure, calculate mean squares for three levels: replicate variation within a subject-occasion specimen, occasion variation within subject, and subject variation. Under the independent, equal-variance model, their expected values are

\begin{aligned} E(MS_A) &= \sigma_A^2,\\ E(MS_O) &= \sigma_A^2 + K\sigma_O^2,\\ E(MS_S) &= \sigma_A^2 + K\sigma_O^2 + JK\sigma_B^2. \end{aligned}

The coefficients \(K\) and \(JK\) arise because occasion and subject differences are represented in averages that each contain, respectively, \(K\) and \(JK\) observations. Subtracting the expected mean squares removes the lower-level components. This gives the balanced-design method-of-moments estimates:

\widehat{\sigma_A^2}=MS_A,\qquad \widehat{\sigma_O^2}=\frac{MS_O-MS_A}{K},\qquad \widehat{\sigma_B^2}=\frac{MS_S-MS_O}{JK}.

These estimates are derived by equating observed mean squares to their expectations; they are not guaranteed to be nonnegative in a finite sample. A negative difference is evidence that the data provide little support for a positive component at that level, not evidence of a physically negative variance. Restricted maximum likelihood (REML) or another likelihood-based mixed-model fit can estimate components while respecting nonnegative variances and accommodating unbalanced designs, but it still cannot recover a component the design does not identify.

Worked Example: Decomposing Balanced Nested Mean Squares

Consider a hypothetical balanced study with \(J=3\) occasions per subject and \(K=2\) replicate assays per occasion. Suppose the mean squares, after fitting the mean structure, are \(MS_A=0.040\), \(MS_O=0.280\), and \(MS_S=1.480\), all in \((\text{mg/L})^2\). Assume the nested random-effects model and its equal-variance and independence assumptions are appropriate.

The assay variance estimate is \(\widehat{\sigma_A^2}=MS_A=0.0400\;(\text{mg/L})^2\). The occasion variance estimate is

\widehat{\sigma_O^2} =\frac{0.280-0.040}{2} =0.120\;(\text{mg/L})^2.

The between-subject variance estimate is

\widehat{\sigma_B^2} =\frac{1.480-0.280}{(3)(2)} =\frac{1.200}{6} =0.200\;(\text{mg/L})^2.

The resulting estimated total variance for a single observation is \(0.040+0.120+0.200=0.360\;(\text{mg/L})^2\). Its estimated standard deviation is \(\sqrt{0.360}=0.600\) mg/L. The three estimates are nonnegative and sum consistently. The calculation depends on the balanced design and correct mean structure; the mean squares alone do not establish that these assumptions hold.

What Replicate Averaging Does—and Does Not Do

If \(K\) independent assays are averaged for one specimen, the average assay error has variance \(\sigma_A^2/K\). The shared subject and occasion components do not average away. Therefore the variance of that specimen’s replicate mean is

\operatorname{Var}(\bar{Y}_{ij\cdot}) =\sigma_B^2+\sigma_O^2+\frac{\sigma_A^2}{K}.

This distinction matters when deciding whether extra assays are useful. Replication improves precision against laboratory error, but it cannot reduce genuine biological variation. In a nonlinear model, replicate measurements should be represented at the assay level or summarized using a model that retains their known precision; treating each technical replicate as a separate subject would falsely inflate the biological sample size.

Worked Example: Benefit of Assaying the Same Specimen Four Times

Use the hypothetical variance estimates from the previous example: \(\widehat{\sigma_B^2}=0.200\), \(\widehat{\sigma_O^2}=0.120\), and \(\widehat{\sigma_A^2}=0.040\), in \((\text{mg/L})^2\). Compare one assay with the mean of four independent assays on the same specimen.

With one assay, the estimated variance is \(0.200+0.120+0.040=0.360\;(\text{mg/L})^2\), with standard deviation \(0.600\) mg/L. With four assays, only the assay component is divided by four:

\widehat{\operatorname{Var}}(\bar{Y}_{ij\cdot}) =0.200+0.120+\frac{0.040}{4} =0.330\;(\text{mg/L})^2.

The replicate-mean standard deviation is \(\sqrt{0.330}=0.574\) mg/L. The reduction is modest because most of the modeled variance is biological and remains shared by the four measurements. The result assumes independent assay errors; repeated readings with a common instrument drift or shared preparation error may not deliver the \(1/K\) reduction.

Assumptions, Diagnostics, and Limits

A variance decomposition is only as credible as its design and assumptions. Before interpreting estimates, check the following:

Replication is at the intended level. Technical replicates should be repeated assays of the same specimen. Independent collections from one subject are biological observations, not assay replicates.
The mean structure is adequate. A time trend, dose effect, or other systematic difference left out of \(\mu_{ij}\) can appear as excess occasion or residual variation.
Assay errors are comparable and independent. Plots of replicate differences against concentration, run, and specimen can reveal changing precision, drift, or outliers. A constant \(\sigma_A^2\) is not appropriate if the assay variance changes materially with concentration.
The response scale is appropriate. A constant additive assay variance on the concentration scale differs from a constant coefficient of variation or a constant variance on the log scale. The chosen scale must match the measurement process and the model.
The design contains enough independent units. Multiple assay replicates without repeated occasions identify assay variation but not separate subject and occasion variation. Multiple occasions without assay replication generally confound assay error with lower-level variability.

In the balanced case, negative method-of-moments estimates or estimates near zero can occur from sampling variation. They can also flag weak separation of components, too few subjects or occasions, or a mismatch between the assumed hierarchy and the data. Examine replicate differences, within-subject occasion contrasts, and subject-level summaries at their corresponding levels rather than relying on a single total residual variance.

The principal alternatives are to estimate components directly from replicate contrasts, use nested ANOVA mean squares for a balanced design, or fit a mixed-effects model by likelihood or REML. Replicate contrasts give a transparent assay-error estimate but do not by themselves separate subject from occasion variability. The balanced ANOVA formulas make the hierarchy visible but are inconvenient for imbalance and can yield negative estimates. Likelihood-based mixed models handle more general designs and mean structures, but require a defensible distributional model and sufficient replication. None of these approaches can distinguish components that the sampling design has confounded.

Common Pitfall

Common mistake: “The spread among all measurements is the assay error, so I can use it as the residual variance.” This is wrong when observations include different subjects or biological occasions: their spread contains biological variation as well. Correction: Estimate assay error from variation among replicate measurements of the same specimen, then use the nested structure and repeated biological occasions to separate higher-level components.

A related mistake is to count every replicate assay as an independent biological observation. Four assays of one specimen provide more precise measurement of that specimen, not four independent subjects or occasions. The appropriate analysis preserves the shared biological levels.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Two assays of the same specimen differ by 0.30 mg/L. Under independent, equal-variance assay errors, what does the squared difference estimate on average?
Why can replicate assays of one specimen estimate assay variance but not between-subject variance?
In a balanced design with \(K=3\), suppose \(MS_A=0.06\) and \(MS_O=0.42\), in \((\text{mg/L})^2\). Calculate the occasion variance estimate.
What happens to the subject and occasion variance components when \(K\) independent assays are averaged for one specimen?
A study has many assay replicates but only one occasion per subject. Can it separately estimate subject and occasion variance? Explain.

Solutions. 1. The squared difference is \(0.30^2=0.0900\;(\text{mg/L})^2\). Its expectation is \(2\sigma_A^2\), so its value estimates twice the assay variance; the corresponding estimate of \(\sigma_A^2\) from this one pair is \(0.0900/2=0.0450\;(\text{mg/L})^2\). 2. Both assays share the same subject and occasion effects, which cancel in their difference; only their assay errors remain. 3. \(\widehat{\sigma_O^2}=(0.42-0.06)/3=0.120\;(\text{mg/L})^2\). 4. Neither biological component is divided by \(K\); the averaged assay-error variance becomes \(\sigma_A^2/K\). 5. No. With only one occasion per subject, the design has no repeated occasions within a subject to separate subject-level variation from occasion-level variation. Replicates identify assay error, but not those two biological components separately.