Replication Criteria Beyond P-Value Matching

Why Replication Needs More Than a P-Value Match

A discovery analysis and an independent validation analysis can produce different p-values even when their estimated effects point in the same direction and are similar in size. Sample sizes, measurement precision and cohort composition all affect uncertainty. Requiring both p-values to fall on the same side of 0.05 therefore makes a threshold crossing—not the scientific pattern of interest—the definition of replication.

A more informative assessment states in advance what pattern would count as replication. Depending on the scientific question, that may involve a matching direction, an effect at least as large as a meaningful minimum, an uncertainty bound that rules out an important effect in the opposite direction, or sufficiently close effects across cohorts. These are distinct claims. A criterion should specify which claim is being assessed, how it will be calculated and what threshold will be used.

Let \(\hat{\beta}_D\) and \(\hat{\beta}_V\) denote the discovery and validation estimates of an effect, respectively, on the same scale and with the same orientation. Their standard errors are \(SE_D\) and \(SE_V\). For a directional criterion, the expected sign should be specified before examining the validation estimate. A minimum meaningful effect, if used, is denoted by \(\delta\), in the units of the effect. A tolerance for comparing two effects is denoted by \(M\), also in effect units.

The estimates need to represent comparable quantities for these comparisons to have their intended meaning. The effect definition, outcome, exposure or contrast, covariate adjustment, and direction coding must be aligned. If, for example, a positive coefficient denotes higher expression in one cohort but lower expression in another because the contrast was reversed, apparent sign disagreement is not an interpretable biological finding. A locked protocol and the earlier material on cross-platform measurement comparability help define what is being compared; the criterion itself cannot repair a mismatch in estimands or measurement.

Constructing a Criterion

Start from the scientific claim rather than from a convenient significance cutoff. If the claim is that the validation effect is positive, a directional criterion is appropriate. If the claim is that it is large enough to matter, specify a magnitude threshold. If the claim is that the two cohorts have sufficiently similar effects, specify an allowable difference. One analysis may report several of these dimensions, but it should not silently substitute one for another.

A confidence interval communicates which effect values remain compatible with the data under the model and confidence procedure. A bound can therefore be tied directly to the claim. For example, if a positive effect is expected and the question is whether the validation effect exceeds zero, a one-sided lower confidence bound is relevant. If the question is whether it exceeds a meaningful threshold \(\delta\), the lower bound must exceed \(\delta\) to support that stronger claim. Merely observing \(\hat{\beta}_V>\delta\) says the point estimate is large enough; it does not establish that the underlying effect exceeds \(\delta\).

Here \(z_{1-\alpha}\) is the \(1-\alpha\) quantile of the standard normal distribution. If \(L_V>0\), the data support a positive effect at the chosen one-sided error level under the model. If \(L_V>\delta\), they support an effect exceeding the specified positive threshold. The corresponding upper-bound rule applies for a prespecified negative direction. These statements depend on the estimator's sampling distribution and the validity of its standard error; a bound is not automatically reliable just because a formula produced it.

Magnitude Bounds and Cross-Cohort Agreement

Magnitude criteria require a scientifically defensible scale. A minimum effect \(\delta\) might be chosen because effects smaller than that amount would have little biological or practical relevance. Its value and units must be stated before inspecting the validation result. A threshold chosen after seeing the estimate can make a weak result appear to satisfy a rule tailored to it.

There are at least two distinct ways to use such a threshold. One can require the validation point estimate to exceed \(\delta\), which describes the estimated magnitude, or require a confidence bound to exceed \(\delta\), which provides stronger evidence that the underlying effect exceeds the threshold. The protocol should say which rule is intended. When multiple features are assessed, the error-rate plan also matters: prespecified thresholds do not by themselves address the multiplicity of testing many features. The earlier material on multiplicity in multi-omics replication is relevant to how thresholds and error rates are managed across a family of claims.

A different question is whether the discovery and validation effects are close enough to count as practically similar. For independent cohorts estimating the same effect on the same scale, define the difference \(\hat{\Delta}=\hat{\beta}_V-\hat{\beta}_D\). If the estimates are independent, its standard error is

A two-sided equivalence rule with prespecified margin \(M>0\) requires the chosen confidence interval for \(\Delta\) to lie wholly within \((-M,M)\). This asks whether differences larger than the margin in either direction are inconsistent with the data at the selected confidence level. It is not enough for the estimated difference to be small, and failure to establish equivalence is not proof that the effects differ. The margin must represent a tolerable difference for the application; exact equality is neither expected nor usually a useful target.

Justifying the Uncertainty Rule

The lower-bound rule follows directly from the approximate sampling model. Assume the estimator is unbiased or suitably centered for the target effect \(\beta_V\), and that \((\hat{\beta}_V-\beta_V)/SE_V\) is approximately standard normal. Then the one-sided lower bound is constructed so that

Consequently, if the observed bound is above zero, values at or below zero are excluded by this one-sided confidence procedure at level \(\alpha\). Replacing zero with \(\delta\) gives the corresponding test of whether the effect exceeds \(\delta\). This justification explains why the estimate alone is not enough: uncertainty, through \(SE_V\), changes the bound even when the point estimate is unchanged.

The approximation can be poor with very small samples, sparse outcomes, boundary estimates or misspecified models. In those cases, use an interval procedure appropriate to the estimator and data, and state it. For a multi-feature analysis, the reported uncertainty should also be interpreted alongside the prespecified multiplicity procedure. A nominal interval for one selected feature does not automatically provide simultaneous coverage for a large set of features.

Diagnostics, Alternatives, and Limits

Before applying any criterion, check the feature and effect coding, the outcome and exposure definitions, the model specification, sample size, missingness and the standard-error method. Inspect the estimate and interval, not just a pass/fail label. A wide interval signals that the data leave a broad range of effects plausible; a narrow interval can rule out effects that would otherwise be important. Examine whether influential observations or model violations drive the estimate, and check that the validation analysis followed its prespecified plan.

The main alternatives answer different questions:

Pooling estimates or assessing between-cohort heterogeneity can address broader questions about combined effects and variation across cohorts; they are not interchangeable with a feature-specific replication rule. A criterion also cannot establish causation, guarantee transportability to every population, or fix a flawed validation design. It makes the evidential target explicit, but the conclusion remains conditional on the data, model and design.

Common Pitfall

A common mistake is to declare “no replication” whenever the discovery p-value is below 0.05 and the validation p-value is above 0.05. That reasoning treats a cutoff crossing as evidence that the effects conflict. The correction is to compare the estimates, their directions and their uncertainty using a rule set in advance. A validation p-value above 0.05 does not establish no effect; conversely, a matching direction alone does not establish a meaningful or sufficiently precise effect. Report what the chosen criterion supports and what remains uncertain.