Why Compare Effects, Not Just Significance?
A replication question is about whether an effect observed in one cohort is also present, in the same direction and at a similar magnitude, in another cohort. A significant result in both cohorts is not enough to answer that question: two estimates can both be significant but differ greatly in size, or one can be non-significant despite being close to the first estimate. The estimates themselves, interpreted on a shared scale, are the central quantities to compare.
In multi-omics work, this comparison requires care. Cohorts may use different assay platforms, encode groups in opposite orders, or report effects in different units. Before comparing numbers, specify exactly what each estimate represents and transform them, where possible, to the same contrast and scale. This tutorial focuses on that comparison, not on combining estimates across cohorts.
Define the Contrast Before Comparing Estimates
An effect estimate is meaningful only with its contrast, outcome, model and scale. For instance, a coefficient may describe the adjusted mean difference in a molecular feature between cases and controls, with the contrast defined as cases minus controls. Reversing that order changes the coefficient's sign. Similarly, an effect per one-year increase in age is not numerically the same as an effect per ten-year increase, even though the underlying relationship may be identical.
Write the contrast in words and, where useful, as a subtraction. For a two-group comparison, state which group is first and which is the reference. For a continuous predictor, state the increment in predictor units. Also record whether the effect is a raw difference, a log-scale difference, a ratio, or another parameter. Do not compare coefficients merely because they have the same variable name.
A valid direct comparison relies on several conditions. Each cohort should use the same target definition and contrast, or a defensible transformation to that definition. Outcome and predictor coding must be aligned. Measurement units and any standardization must be understood. The models should adjust for covariates in a way that supports the same interpretation. Finally, the feature should be measured comparably enough that the estimated effects refer to the same biological quantity. These conditions are not guaranteed by having similar cohort labels.
Worked Example: Comparing Estimates on the Same Scale
Suppose two hypothetical independent cohorts estimate the adjusted difference in a molecular feature between cases and controls. Both define the contrast as cases minus controls, measure the feature in log2 expression units, and use the same adjustment variables. Cohort A reports \(\hat{\theta}_A=0.40\) log2 units with \(SE=0.16\); Cohort B reports \(\hat{\theta}_B=0.38\) log2 units with \(SE=0.25\). Assume the fitted models are suitable for these estimates and their standard errors.
Both estimates are positive, so they agree in direction. Their absolute magnitudes differ by \[ |\hat{\theta}_B-\hat{\theta}_A|=|0.38-0.40|=0.02 \] log2 units. Relative to Cohort A's estimate, Cohort B's estimate is \[ \frac{0.38}{0.40}=0.95, \] or 95% of its magnitude. This is a descriptive comparison, not proof that the population effects are identical. Whether a difference of 0.02 log2 units is negligible should be judged against a prespecified scientific tolerance.
For context only, the cohort-specific Wald statistics are \(z_A=0.40/0.16=2.50\) and \(z_B=0.38/0.25=1.52\). Under a two-sided standard-normal Wald test, the approximate p-values are 0.0124 and 0.1285, respectively. Thus one estimate crosses a conventional 0.05 threshold and the other does not, even though the estimates are close in direction and size. The p-value of 0.1285 is not the probability that the null hypothesis is true, nor does it show that Cohort B has no effect.
Harmonize Sign and Units
A sign disagreement can be biological, but it can also be a bookkeeping error. Check group order, reference categories, allele or feature coding, and whether a reported coefficient has already been reversed. If Cohort A reports treated minus control and Cohort B reports control minus treated, the estimates must not be compared as written. Re-express one estimate using the same contrast first.
Unit conversion is similarly important. If one cohort reports an effect per year and another per decade, multiply the per-year coefficient by 10 to express it per decade. This follows directly from a linear coefficient: if the predictor increment is multiplied by \(k\), the effect per increment is multiplied by \(k\). More generally, if the estimate is \(\hat{\theta}_{old}\) per one old unit and one new unit equals \(k\) old units, then \(\hat{\theta}_{new}=k\hat{\theta}_{old}\). This preserves the modeled relationship while changing its reporting scale.
Worked Example: Correcting a Reversed Contrast
In a hypothetical treatment study, Cohort A reports a feature difference of \(+0.30\) standard-deviation units for treated participants minus controls. Cohort B reports \(-0.28\) standard-deviation units for controls minus treated participants. The feature has been standardized comparably in both cohorts, and the models otherwise target the same adjusted contrast.
The estimates appear to have opposite signs, but Cohort B used the reverse subtraction. To express its estimate as treated minus controls, reverse its sign: \[ \hat{\theta}_{B,\;treated-control}=-(-0.28)=+0.28 \] standard-deviation units. Both harmonized estimates are positive. Their magnitude difference is \(|0.30-0.28|=0.02\) standard-deviation units, and Cohort B's estimate is \(0.28/0.30=0.933\), or about 93.3% of Cohort A's magnitude.
This apparent agreement is interpretable only if the standardization is comparable. If each cohort divided by its own substantially different standard deviation, the estimates are cohort-specific standardized effects, not necessarily the same raw-scale biological difference. The contrast and scale check is therefore part of the conclusion, not a clerical afterthought.
Worked Example: Converting a Predictor Increment
Consider a hypothetical association between age and a molecular marker. Cohort A reports an adjusted change of \(0.12\) log2 units per 10 years. Cohort B reports \(0.010\) log2 units per year. Assume both use the same linear effect definition and covariate interpretation.
Convert Cohort B to the per-decade scale: \[ \hat{\theta}_{B,\;10\ years}=10(0.010)=0.10 \] log2 units per 10 years. The estimates are both positive. The absolute difference is \(|0.12-0.10|=0.02\) log2 units per decade; Cohort B's magnitude is \(0.10/0.12=0.833\), or 83.3% of Cohort A's estimate.
This conversion is justified because the stated coefficient is linear in the age increment. It would not by itself harmonize estimates from different nonlinear models, such as coefficients applying over different ranges on a curved age-response relationship. In that situation, compare predictions or contrasts over a common age interval instead of multiplying a single coefficient.
A Practical Comparison and Its Limits
A transparent report distinguishes direction, magnitude and uncertainty. Direction asks whether both nonzero estimates have the same sign after harmonization. Magnitude asks how far apart the estimates are in common units. Uncertainty describes how precisely each cohort estimates its effect. These are related but distinct questions. A small difference between point estimates can be imprecise, and similar p-values do not imply similar effects.
For numerical summaries, the signed difference \(\hat{\theta}_B-\hat{\theta}_A\) retains which estimate is larger. The absolute difference \(|\hat{\theta}_B-\hat{\theta}_A|\) gives their separation regardless of order. A ratio of magnitudes, \(|\hat{\theta}_B|/|\hat{\theta}_A|\), can be useful when the reference estimate is clearly away from zero and signs agree. It is unstable or undefined when the reference is near or equal to zero, so it should not be used mechanically. Any threshold for an acceptable difference should be chosen for the scientific setting and preferably specified before inspecting the validation result.
These summaries describe the observed estimates; they do not establish that the underlying cohort effects are equal. A formal assessment of uncertainty around replication effects is a separate inferential task. Likewise, combining cohort estimates answers a different question from checking their individual directions and magnitudes. Keep those tasks distinct when reporting results.
| Approach | What it answers | What it cannot establish alone |
|---|---|---|
| Compare harmonized estimates | Whether observed directions and sizes are similar on a shared scale | That population effects are exactly equal |
| Compare significance decisions | Whether each test crossed its selected threshold | Whether effect sizes agree; decisions depend on precision and sample size |
| Compare signs only | Whether nonzero estimates point in the same direction | Whether their magnitudes are close or scientifically important |
| Compare raw coefficients without harmonization | Usually only the reported numbers | Meaningful cross-cohort agreement when contrasts or units differ |
Diagnostics and Common Failure Modes
When a comparison looks surprising, revisit the data and analysis definitions before interpreting it biologically. A useful diagnostic sequence is to verify cohort and sample independence, confirm the feature mapping and assay direction, check the group reference and predictor increment, and inspect transformations and covariate adjustments. Differences in assay calibration, normalization, missingness or eligible participants may change the estimand or scale even when the feature name is identical.
- Opposite signs: check whether one cohort reversed the contrast or coding before concluding that biological directions conflict.
- Large apparent size difference: check raw versus standardized units, predictor increments, outcome transformations and model specifications.
- Matching significance but different estimates: report the estimates and scales; threshold agreement does not show magnitude replication.
- Different significance decisions: do not infer that effects differ solely from one p-value being below a threshold and the other above it. Precision and sample size affect those decisions.
- Zero or near-zero reference estimate: do not use a magnitude ratio as if it were stable. Report signed and absolute differences, with uncertainty handled separately.
A key limitation is that harmonization cannot repair a mismatch in the scientific target. If the cohorts measure different molecular features, define different outcomes, or estimate differently adjusted contrasts, a numerical conversion may create the appearance of comparability without making the estimands the same. State such limitations explicitly and avoid stronger replication claims than the data support.
Check Your Understanding
Answer each question, then check the brief solutions below.
- Cohort A reports \(+0.24\) units for exposed minus unexposed; Cohort B reports \(-0.20\) units for unexposed minus exposed. After harmonization, do the estimates agree in direction, and what is their absolute difference?
- One cohort reports \(0.015\) marker units per year, while another reports \(0.13\) marker units per decade. Convert the first estimate to a per-decade scale and compare the directions.
- Why does a significant result in both cohorts not, by itself, demonstrate magnitude agreement?
- If \(\hat{\theta}_A=0\), what is wrong with reporting \(|\hat{\theta}_B|/|\hat{\theta}_A|\) as a magnitude-agreement measure?
Solutions. (1) Reverse Cohort B's contrast: \(-(-0.20)=+0.20\). The estimates agree positively; their absolute difference is \(|0.24-0.20|=0.04\) units. (2) Convert the first estimate: \(10(0.015)=0.15\) marker units per decade. Both are positive; their difference is \(0.15-0.13=0.02\) units per decade. (3) Significance depends on both estimated effect and its precision, so two significant tests can have substantially different effect sizes. Compare the harmonized estimates directly. (4) The ratio divides by zero and is undefined. If the reference estimate is near zero, the ratio is also unstable; use a difference and describe uncertainty separately.