Companion Diagnostic Statistical Validation

where \(c\) is the classification cutoff.

Cutoff selection is therefore a major statistical issue.

Cutoff Validation Principles

A cutoff should not simply be selected because it produces the most favorable validation results in a convenient dataset.

Important considerations include:

• Clinical rationale
• Analytical characteristics
• Biological plausibility
• Prespecified development strategy
• Independent or appropriately controlled evaluation
• Expected clinical consequences of misclassification

ROC Curves

When a continuous biomarker is used to classify patients, receiver operating characteristic analysis may be useful during development.

The ROC curve displays:

across possible thresholds.

The area under the curve, or AUC, summarizes discrimination over the range of thresholds.

However, the AUC does not automatically establish that a selected cutoff is clinically appropriate.

Clinical Concordance

Clinical validation asks a different question from analytical validation.

The central issue is whether the diagnostic identifies the population in which the therapeutic benefit demonstrated in clinical development applies.

For example, suppose a pivotal clinical trial enrolled patients based on biomarker status determined using an investigational assay.

A commercial CDx may need to demonstrate appropriate concordance with that clinical-trial classification.

Bridging a Clinical-Trial Assay to a CDx

A common development problem is that the assay used during clinical development differs from the final commercial diagnostic.

The statistical bridge may involve comparing:

• Clinical-trial assay results
• Candidate CDx results
• Reference or orthogonal assay results
• Archived clinical specimens

The key question is whether patients classified as biomarker-positive or negative by the final assay correspond appropriately to the population supporting the therapeutic indication.

Positive and Negative Percent Agreement in Bridging

Suppose the clinical-trial assay identifies 150 positive specimens and the candidate CDx identifies 143 of those as positive.

If there are 7 discordant positive specimens:

The analysis should then investigate the discordant specimens rather than treating the aggregate percentage as the end of the analysis.

Discordant Analysis

Discordant specimens can provide important information.

For every clinically important discordance, consider whether the difference could arise from:

• Specimen quality
• Low analyte concentration
• Heterogeneity
• Pre-analytical handling
• Assay failure
• Cutoff differences
• Biological differences
• True disagreement between measurement technologies

A good validation report should not simply list discordances without investigation.

Borderline Specimens

Specimens near the assay cutoff are particularly informative.

Suppose a continuous assay has a positive cutoff of:

A specimen with:

may be classified positive while another measurement of the same specimen could be negative.

This makes near-cutoff precision and reproducibility especially important.

Clinical Consequences of Misclassification

A useful validation framework distinguishes statistical performance from clinical consequences.

The relative importance of these errors depends on the clinical context.

Predictive Values

Positive predictive value is:

Negative predictive value is:

Unlike PPA and NPA, predictive values depend strongly on the prevalence of the biomarker in the tested population.

Prevalence Effects

Consider an assay with:

If biomarker prevalence is low, the number of false positives can be large relative to true positives.

Therefore, the same assay can have substantially different PPV and NPV in different populations.

Invalid Rates

A CDx validation should explicitly evaluate invalid, indeterminate, or uninterpretable results when such outcomes are possible.

For example:

An assay with excellent PPA among valid results may still have an important clinical limitation if a substantial fraction of specimens produce no valid result.

First-Pass Success Rate

Another useful metric can be the percentage of specimens producing an interpretable result on the initial testing attempt.

This can be particularly relevant for assays involving complex specimen processing.

Missing Data and Unevaluable Specimens

Diagnostic validation studies can lose specimens because of:

• Insufficient specimen quantity
• Degraded material
• Failed extraction
• Instrument failure
• Invalid control results
• Missing comparator results
• Inadequate specimen quality

The analysis should distinguish:

• Not tested
• Test attempted but invalid
• Valid negative
• Valid positive
• Unevaluable for a particular endpoint

These categories should not be silently collapsed into a single "missing" category.

Repeat Testing

Repeated testing after an initial invalid result introduces another important statistical issue.

Suppose:

• 100 specimens are initially tested.
• 10 produce invalid results.
• 8 of the 10 become valid after repeat testing.

Reporting only the final valid results could make the assay appear to have 100% valid performance among the retained specimens.

The initial invalid rate remains clinically relevant.

Robustness

Robustness examines whether modest, realistic variations in operating conditions materially affect assay performance.

Examples include:

• Incubation time
• Temperature
• Input amount
• Extraction conditions
• Storage duration
• Reagent handling
• Instrument settings

The purpose is not to intentionally break the assay.

The purpose is to determine whether reasonable deviations within the intended operating environment cause unacceptable performance changes.

Specimen Stability

Specimen stability can be evaluated at clinically relevant storage conditions.

For example:

Lot-to-Lot Consistency

If reagent lots can change over time, the validation strategy should evaluate whether lot-to-lot variation remains within predefined requirements.

For a quantitative assay, this may involve estimating:

For a qualitative assay, classification agreement may be more appropriate.

Reader Agreement

Some CDx technologies involve subjective interpretation.

Immunohistochemistry is a common example where trained readers may assign categorical scores.

Possible analyses include:

• Raw agreement
• Positive agreement
• Negative agreement
• Inter-reader agreement
• Intra-reader agreement
• Kappa statistics where appropriate

Kappa Statistics

For categorical assessments, Cohen's kappa is:

where:

• \(P_o\) = observed agreement
• \(P_e\) = agreement expected by chance under the specified model

Kappa can be useful, but it should not replace clinically meaningful agreement measures.

In situations with highly imbalanced prevalence, kappa can behave in ways that make interpretation difficult.

Multi-Category CDx Results

Not all CDx classifications are binary.

An assay might classify specimens as:

• Negative
• Low positive
• Intermediate
• High positive

For ordered categories, weighted agreement methods may be appropriate.

The analysis should preserve the ordinal structure rather than automatically reducing everything to positive versus negative.

NGS-Based Companion Diagnostics

Next-generation sequencing introduces additional statistical considerations.

Potential validation dimensions include:

• Variant detection
• Variant type
• Allele fraction
• Coverage
• Read quality
• Sample quality
• False-positive variant calls
• False-negative variant calls
• Pipeline reproducibility
• Reference genome and bioinformatics configuration

The statistical unit may differ depending on whether the endpoint is:

• Specimen-level classification
• Variant-level detection
• Patient-level biomarker status

Variant-Level vs. Patient-Level Agreement

Nested Data and Correlation

When multiple measurements come from the same specimen or patient, the data are clustered.

For example:

Observations within the same hierarchy are not necessarily independent.

Possible analytical approaches include:

• Mixed-effects models
• Variance-component models
• Generalized estimating equations
• Cluster-aware confidence intervals
• Specimen-level summaries

The method should follow the estimand and experimental design.

Analytical Sensitivity vs. Clinical Sensitivity

These terms are often confused.

A test can have excellent analytical sensitivity while still having imperfect clinical sensitivity because clinical specimens introduce biological heterogeneity and pre-analytical variation.

Pre-Analytical Variables

The assay does not operate in isolation.

The full diagnostic workflow can be represented as:

Validation should consider the portions of this pathway included in the intended use.

Sample Quality

For tissue-based diagnostics, sample quality can materially influence performance.

Potential factors include:

• Tumor content
• Necrosis
• DNA or RNA integrity
• Fixation quality
• Specimen age
• Section thickness
• Extraction yield

These variables may need to be incorporated into analytical validation or eligibility criteria.

Statistical Acceptance Criteria

A validation study should not be designed around vague goals such as "demonstrate good performance."

Instead, each critical performance characteristic should have a predefined acceptance criterion.

Point Estimate vs. Confidence-Bound Criterion

Suppose the acceptance criterion is:

A study might observe:

But if the lower confidence bound is below 90%, the prespecified success rule may not be met.

Therefore:

The inferential rule must be defined in advance.

Multiple Performance Characteristics

A CDx validation program can contain many endpoints.

For example:

• PPA
• NPA
• Precision
• LoD
• Linearity
• Interference
• Stability
• Reproducibility

Not every performance characteristic should automatically be treated as a formal hypothesis test requiring multiplicity adjustment.

The statistical framework should distinguish:

• Descriptive performance characterization
• Formal acceptance criteria
• Primary validation claims
• Secondary supporting analyses

Study Design Matrix

A useful planning tool is a validation matrix.

Validation Hierarchy

A useful conceptual hierarchy is:

Clinical Cutoff vs. Analytical Cutoff

A subtle but important distinction is the difference between an analytical detection threshold and a clinical classification threshold.

For example, an assay might reliably detect a molecular signal at:

while the clinically validated positive classification cutoff may be:

These are not contradictory.

The assay can detect lower levels than are required for the clinical classification.

Precision Near the Clinical Cutoff

Near-cutoff precision is often more consequential than precision far from the cutoff.

Suppose:

and repeated measurements are:

The assay may have excellent precision in absolute terms but still produce classification switching around the decision boundary.

Therefore, validation should include specimens representative of the decision region.

Classification Concordance

For a binary CDx, classification concordance can be expressed as:

But concordance should be accompanied by the directional components.

Reference Method Limitations

A comparator method is not automatically a perfect gold standard.

This is particularly important for biomarkers where biological truth cannot be directly observed.

If two assays use different technologies, a disagreement may reflect:

• Technical error by one assay
• Technical error by the other assay
• Differences in analytical sensitivity
• Different target definitions
• Biological heterogeneity

Therefore, an orthogonal method or adjudication strategy may sometimes be useful for discordant cases.

Orthogonal Testing

An orthogonal assay uses a different analytical principle to investigate a result.

For example, an NGS finding could potentially be investigated using an independent molecular method where appropriate.

The purpose is not necessarily to declare one assay correct automatically.

Instead, orthogonal evidence can help determine whether a discordance reflects a genuine biological signal or an analytical limitation.

Clinical Specimen Diversity

A strong validation program should reflect the intended clinical population and specimen environment.

Depending on the indication, this may include variation in:

• Disease stage
• Specimen type
• Tumor content
• Biomarker abundance
• Demographic and clinical characteristics
• Specimen age
• Collection site

The precise sampling strategy depends on the intended-use population and development program.

Rare Biomarkers

Rare biomarker populations create a special statistical challenge.

Suppose the biomarker prevalence is only 5%.

A general population sample of 1,000 patients would contain approximately:

positive patients on average.

If the precision of PPA is important, the study may therefore require intentional enrichment or another appropriate specimen-acquisition strategy, provided that the resulting analysis remains valid for the intended claim.

Enrichment Does Not Change the Mathematics of PPA

If the study intentionally enriches positive specimens, PPA and NPA remain defined using their respective denominators.

However, predictive values estimated from the enriched sample would generally not represent the predictive values expected in a routine population unless appropriately adjusted.

Statistical Analysis Plan

The statistical analysis plan for a CDx validation study should define at least:

• Analysis populations
• Primary endpoints
• Secondary endpoints
• Specimen eligibility
• Handling of invalid results
• Handling of repeat testing
• Comparator definition
• Cutoff definition
• Confidence interval methodology
• Missing-data handling
• Outlier handling
• Acceptance criteria
• Subgroup analyses
• Discordant-result investigations

Analysis Populations

A useful framework may distinguish:

Different endpoints may use different denominators.

That is acceptable as long as the definitions are explicit.

Subgroup Analysis

CDx performance can vary across clinically relevant specimen characteristics.

Potential subgroups may include:

• Specimen type
• Tumor content
• Biomarker abundance
• Site
• Operator
• Instrument
• Reagent lot

Subgroup analyses should generally be prespecified where they are intended to support a validation claim.

Small Subgroups

A subgroup with only a few positive specimens can produce an apparently extreme PPA.

For example:

does not mean the subgroup's true performance is known with certainty.

The confidence interval is essential.

Bootstrapping and Resampling

For some complex validation statistics, resampling methods may provide useful uncertainty estimates.

However, resampling should respect the data hierarchy.

If five replicates come from one specimen, randomly resampling individual replicates as though they were independent patient observations can be misleading.

A cluster-aware bootstrap may instead resample at the specimen level.

Power and Precision

Validation studies are often framed around acceptance criteria rather than traditional superiority hypotheses.

Consequently, sample-size planning may focus on the width of a confidence interval.

For example, the study team may require:

where \(W\) is the maximum acceptable uncertainty.

This is particularly intuitive for agreement estimates.

Example of Precision-Based Planning

Suppose investigators expect:

and want a reasonably narrow confidence interval around the estimate.

The number of positive specimens required will generally be driven much more by the desired precision than by the total number of specimens alone.

This illustrates a useful planning principle:

Validation of a Quantitative Biomarker

Suppose the CDx reports a quantitative biomarker value.

A comprehensive analytical validation might examine:

1. Accuracy
2. Precision
3. Linearity
4. Reportable range
5. Analytical sensitivity
6. Analytical specificity
7. Interference
8. Stability
9. Carryover
10. Robustness

The final clinical classification may then be based on a prespecified cutoff.

Validation of a Qualitative Biomarker

For a qualitative CDx, the core package may emphasize:

• Positive agreement
• Negative agreement
• Overall agreement
• Precision
• Reproducibility
• LoD
• Interference
• Cross-reactivity
• Invalid rate
• Clinical concordance

The statistical methods should match the categorical nature of the endpoint.

Example CDx Validation Dataset

SPECIMEN   CDX   COMP   RESULT
001        +     +      Concordant Positive
002        +     +      Concordant Positive
003        +     -      Discordant
004        -     -      Concordant Negative
005        -     +      Discordant
006        -     -      Concordant Negative
007        +     +      Concordant Positive
008        -     -      Concordant Negative

From the table, the analysis can derive the four classification cells.

In real validation programming, the data structure would generally contain additional variables for specimen source, assay run, lot, operator, site, instrument, biomarker level, validity status, and other protocol-defined attributes.

R Implementation: PPA and NPA

tab <- table(
  CDx = data$CDX,
  Comparator = data$COMP
)

TP <- tab["Positive", "Positive"]
FP <- tab["Positive", "Negative"]
FN <- tab["Negative", "Positive"]
TN <- tab["Negative", "Negative"]

PPA <- TP / (TP + FN)
NPA <- TN / (TN + FP)
OPA <- (TP + TN) / sum(tab)

results <- data.frame(
  PPA = PPA,
  NPA = NPA,
  OPA = OPA
)

In production programming, explicit handling of missing cells, zero denominators, factor levels, and invalid specimens should be added.

R Implementation: Wilson Confidence Interval

wilson_ci <- function(x, n, z = 1.96) {

  p <- x / n

  denominator <- 1 + z^2 / n

  center <- (
    p + z^2 / (2 * n)
  ) / denominator

  half_width <- (
    z * sqrt(
      p * (1 - p) / n +
      z^2 / (4 * n^2)
    )
  ) / denominator

  c(
    lower = center - half_width,
    upper = center + half_width
  )
}

wilson_ci(TP, TP + FN)

For a regulated validation package, the chosen confidence-interval method should be specified in the statistical analysis plan and implemented consistently.

R Implementation: LoD Detection Probability

If replicate results are coded as detected versus not detected, a logistic model can be useful for exploratory estimation of the detection curve.

lod_model <- glm(
  detected ~ log10(concentration),
  data = lod_data,
  family = binomial()
)

summary(lod_model)

pred <- predict(
  lod_model,
  type = "response"
)

The exact LoD estimation approach should be selected according to the assay, study design, protocol, and intended claim.

R Implementation: Bland-Altman Data

data$mean_method <- (
  data$reference +
  data$candidate
) / 2

data$difference <- (
  data$candidate -
  data$reference
)

mean_difference <- mean(
  data$difference,
  na.rm = TRUE
)

sd_difference <- sd(
  data$difference,
  na.rm = TRUE
)

upper_loa <-
  mean_difference +
  1.96 * sd_difference

lower_loa <-
  mean_difference -
  1.96 * sd_difference

The resulting values can be plotted against the paired means.

Validation Programming QC

Statistical validation requires more than checking whether the final percentages look plausible.

Independent QC should verify:

• Specimen inclusion
• Comparator classification
• Candidate classification
• Cutoff derivation
• Invalid handling
• Repeat-testing logic
• Numerators
• Denominators
• Confidence intervals
• Precision estimates
• LoD derivations
• Subgroup definitions
• Figure values

Patient-Level Traceability

Every summary statistic should be traceable back to specimen-level records.

For example:

Common Statistical Mistakes

1. Calling a comparator the gold standard without justification. Agreement with a comparator is not necessarily equivalent to truth.
2. Reporting only overall agreement. PPA and NPA can reveal asymmetric performance hidden by OPA.
3. Ignoring confidence intervals. A point estimate does not quantify uncertainty.
4. Treating replicates as independent patients. Repeated measurements within a specimen are correlated.
5. Ignoring invalid results. Invalid rates can have direct clinical relevance.
6. Using post hoc cutoff optimization. This can create optimistic estimates of performance.
7. Confusing LoD with clinical sensitivity. Analytical detection capability is not the same as patient-level diagnostic performance.
8. Using correlation as a substitute for agreement. High correlation can coexist with systematic bias.
9. Ignoring rare positive populations. A large total sample does not guarantee precise PPA.
10. Ignoring discordant specimens. Discordances can reveal important assay limitations.
11. Pooling heterogeneous specimen types without justification. Different matrices can behave differently.
12. Using inappropriate denominators. The denominator must correspond to the defined estimand.

Regulatory Perspective

Companion diagnostic development is closely linked to the therapeutic development program.

The statistical validation strategy should therefore be consistent with:

• The intended use of the diagnostic
• The biomarker definition
• The therapeutic indication
• The clinical-trial enrollment strategy
• The clinical evidence supporting the treatment
• The analytical characteristics of the final assay

In the United States, FDA guidance concerning in vitro companion diagnostic devices emphasizes the close relationship between the diagnostic and the corresponding therapeutic product.

The FDA has also issued guidance addressing analytical validation considerations for certain next-generation sequencing-based in vitro diagnostics.

CDx and the Drug Development Program

The diagnostic and therapeutic programs should generally be considered as an integrated development strategy.

The relationship can be represented as:

Bridging Studies

A bridging study may be necessary when the assay used to select patients in clinical development differs from the final commercial CDx.

The statistical question is whether the final assay reproduces the clinically meaningful classification sufficiently well.

Potential endpoints include:

• PPA
• NPA
• OPA
• Classification consistency
• Discordance rates
• Clinical outcome concordance where applicable

Clinical Outcome Concordance

In some programs, the strongest question is not merely:

but rather:

This is a fundamentally different level of evidence.

A diagnostic can agree extremely well with another assay while neither assay adequately identifies the population that benefits from treatment.

Diagnostic Performance vs. Clinical Utility

Subgroup Performance

A CDx can have excellent overall performance while exhibiting weaker performance in a clinically important subgroup.

For example, performance could differ by:

• Low versus high biomarker abundance
• Specimen type
• Fresh versus archived tissue
• Primary versus metastatic tissue
• Different collection sites

Subgroup analyses should therefore be planned around plausible sources of heterogeneity.

Interaction Between Biomarker Level and Error

Diagnostic error is often not constant across the measurement range.

For example:

The probability of correct classification may be highest for very strong positives and strong negatives, while specimens near the cutoff may have greater uncertainty.

Visualizing Performance Across the Measurement Range

Clinical Specimen Testing Strategy

A practical validation program should include specimens spanning the relevant range of biomarker status.

For a binary biomarker, this may include:

• Strong positives
• Moderate positives
• Near-cutoff positives
• Near-cutoff negatives
• Strong negatives

This design helps distinguish broad assay performance from decision-boundary performance.

Precision and Reproducibility Near LoD

Low-level specimens can simultaneously challenge:

• Analytical sensitivity
• Precision
• Classification stability

Therefore, LoD studies and precision studies may provide complementary evidence.

Carryover

For some assays, carryover from a high-positive specimen into a subsequent negative specimen can create false-positive results.

A typical design may alternate high-positive and negative specimens.

The key question is whether the negative specimens remain negative under the specified conditions.

Cross-Reactivity

Cross-reactivity evaluates whether related analytes or sequences produce inappropriate positive signals.

This can be particularly important for molecular assays targeting closely related variants or organisms.

Analytical Validation Summary Matrix

Clinical Validation Summary Matrix

Validation Reporting

A final validation report should allow an independent reviewer to understand:

• What was tested
• Why it was tested
• How specimens were selected
• How results were classified
• Which specimens were excluded
• How invalids were handled
• How repeat testing was handled
• Which statistical methods were used
• Which acceptance criteria were prespecified
• Whether the criteria were met
• What limitations remain

Traceability From Requirement to Result

A strong validation package can be organized as:

Example Validation Table

The numerical values above are illustrative rather than regulatory requirements.

How to Interpret a Validation Failure

Not every failure means that the entire assay is unusable.

A failed acceptance criterion should trigger investigation.

Potential explanations include:

• Unexpected specimen variability
• Insufficient sample size
• Incorrect design assumptions
• Assay instability
• Unexpected interference
• Lot effects
• Operator effects
• Inappropriate cutoff
• Data-processing error

The appropriate response depends on the specific failure and its clinical importance.

Statistical Validation Is Not the Same as Statistical Significance

A common misconception is that validation means obtaining statistically significant p-values.

That is often not the central objective.

The real question may be whether performance is sufficiently precise and reliable to support a predefined claim.

For example:

can be more informative than:

for an assay-performance claim.

What Makes a Strong CDx Statistical Package?

A strong package has five characteristics.

1. Prespecified

The endpoints, estimands, acceptance criteria, and analysis methods are defined before examining the final validation results.

2. Representative

The specimens and operating conditions represent the intended use.

3. Quantitative

Performance is expressed with appropriate estimates and uncertainty.

4. Traceable

Every summary result can be traced to the underlying specimen-level data.

5. Clinically connected

Analytical performance is ultimately connected to the clinical decision the diagnostic is intended to support.

Practical CDx Validation Checklist

Common Reporting Language

A concise statistical conclusion might state:

What the Final Statistical Report Should Contain

Final Perspective

Companion diagnostic statistical validation sits at the intersection of laboratory science, biostatistics, clinical development, and regulatory science.

The central statistical challenge is not simply calculating a sensitivity, specificity, or coefficient of variation.

The deeper question is whether the evidence establishes that the diagnostic can make the intended clinical classification reliably enough for its purpose.

That requires a chain of evidence.

Each link answers a different question.

An assay may be analytically precise but clinically irrelevant.

An assay may agree well with a comparator but perform poorly near the therapeutic decision threshold.

An assay may have excellent PPA but an unacceptable invalid rate.

A large study may produce a precise overall estimate while providing too few positive specimens to establish a reliable PPA.

These distinctions are why companion diagnostic validation requires careful statistical planning before the laboratory study begins.

References

U.S. Food and Drug Administration. In Vitro Companion Diagnostic Devices: Guidance for Industry and Food and Drug Administration Staff. 2014.

U.S. Food and Drug Administration. Considerations for Design, Development, and Analytical Validation of Next Generation Sequencing (NGS)-Based In Vitro Diagnostics (IVDs) Intended to Aid in the Diagnosis of Cancer.

U.S. Food and Drug Administration. Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.

Clinical and Laboratory Standards Institute. Evaluation of Precision of Quantitative Measurement Procedures. CLSI EP05.

Clinical and Laboratory Standards Institute. Evaluation of Detection Capability for Clinical Laboratory Measurement Procedures. CLSI EP17.

Clinical and Laboratory Standards Institute. Measurement Procedure Comparison and Bias Estimation Using Patient Samples. CLSI EP09.

Clinical and Laboratory Standards Institute. Evaluation of Qualitative, Binary Output Examination Performance. CLSI EP12.

Bland, J.M. and Altman, D.G. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 1986.

McHugh, M.L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.