Why Test Between-Study Heterogeneity?
A fixed-effect meta-analysis summarizes several cohort-specific genetic association estimates by assuming that they estimate one common underlying effect. Cochran’s Q provides a test of whether the observed estimates differ more than expected from their sampling variation under that assumption. It is a test about consistency of effects across cohorts, not a test of whether the genetic variant is associated with the outcome at all.
For example, a genetic association may be strongly supported overall even if the cohort estimates are not fully consistent. Conversely, estimates may be consistent with one another while all being close to zero. Keeping these questions separate is essential: the association test asks about the common effect’s value, whereas Q asks whether a common effect is a reasonable description of the cohort-specific effects.
The estimate \(\hat{\beta}_{FE}\) is the weighted center against which each cohort estimate is compared. A cohort contributes more to Q when its estimate is far from that center relative to its standard error. The terms are squared, so departures in opposite directions both contribute positively. Because effects and standard errors use the same scale, each weighted squared departure, and therefore Q, is dimensionless.
Assumptions and Reference Distribution
The usual chi-square reference distribution applies under a specific null model. Each cohort must estimate the same effect, the estimates must be independent, and their sampling distributions must be approximately normal with correctly estimated variances. In notation, under the null hypothesis of a common effect \(\beta\), assume \(\hat{\beta}_i\) are independent and approximately \(N(\beta,SE_i^2)\). Here \(SE_i\) is treated as known, or as an adequate estimate of the sampling standard deviation. In large genetic association studies, normal approximations are often reasonable, but they are not automatic for every estimate or analysis.
The association effects also need to be comparable. For instance, a variant’s effect estimates should refer to the same harmonized effect allele, outcome definition, and effect scale. A coefficient per effect allele on a log-odds scale cannot be compared directly with a coefficient on a different scale. Differences in the estimand can create apparent inconsistency that Q will detect, but that inconsistency does not necessarily represent genuine variation in one well-defined biological effect.
If these conditions hold, Q has approximately a chi-square distribution with \(K-1\) degrees of freedom when all cohort effects are equal. One degree of freedom is used in estimating the weighted common effect from the \(K\) estimates. For a chosen significance level \(\alpha\), reject the common-effect null when Q exceeds the upper-tail \(\alpha\) critical value from \(\chi^2_{K-1}\), or equivalently when its p-value is below \(\alpha\).
Why the Chi-Square Distribution Appears
The result follows by expressing Q as a squared residual distance after estimating one common mean. Let \(y\) be the vector of cohort estimates, \(W\) the diagonal matrix with entries \(w_i\), and \(\mathbf{1}\) the vector of \(K\) ones. Under the null, \(y\) is approximately multivariate normal with mean \(\beta\mathbf{1}\) and covariance matrix \(W^{-1}\), because the cohort estimates are assumed independent. The weighted least-squares estimate of the common effect is \(\hat{\beta}_{FE}\).
To see the distribution, standardize the estimates by setting \(z=W^{1/2}y\). Then \(z\) has identity covariance and mean \(\beta W^{1/2}\mathbf{1}\). Estimating the common effect is equivalent to projecting \(z\) onto the one-dimensional vector \(W^{1/2}\mathbf{1}\). The residual is the projection onto the perpendicular subspace. This residual projection has rank \(K-1\), and under the null its coordinates in an orthonormal basis of that subspace are independent standard normal variables. Q is the sum of their squares, so it has a \(\chi^2_{K-1}\) distribution.
This derivation also shows why estimating the common effect costs one degree of freedom. If there is only one cohort, \(K-1=0\): there is no between-cohort comparison and no Q test. When the true effects differ, the statistic tends to be larger, but a large observed Q is not a diagnosis of the cause of those differences.
Calculating and Interpreting Q
Worked Example: A Q Statistic With Similar Cohort Estimates
Suppose three independent hypothetical cohorts estimate a variant’s log odds ratio per effect allele as \(0.10\), \(0.20\), and \(0.30\), each with standard error \(0.10\). Assume that the effect allele and outcome definition have been harmonized and that each standard error is adequate.
Each weight is \(w_i=1/(0.10)^2=100\). Since the weights are equal, the fixed-effect estimate is the ordinary average:
The Q statistic is
There are \(K-1=2\) degrees of freedom. For two degrees of freedom, the upper-tail probability is \(P(\chi^2_2\geq Q)=e^{-Q/2}\), so the p-value is \(e^{-1}=0.3679\). At \(\alpha=0.05\), this does not provide evidence against equal cohort effects. It does not prove that the effects are identical: with only three cohorts, differences of this size could readily occur through sampling variation.
Worked Example: Unequal Precision and a Small Q-Test p-Value
Consider three independent hypothetical estimates, all on the same log-odds-ratio-per-effect-allele scale: \(0.12\) with \(SE=0.04\), \(0.08\) with \(SE=0.08\), and \(0.48\) with \(SE=0.10\). Assume compatible effect definitions and reliable standard errors. The inverse-variance weights are \(625\), \(156.25\), and \(100\), respectively.
The weighted estimate is
Using this value in the Q formula gives
With two degrees of freedom, the p-value is \(e^{-12.21/2}\approx0.0022\). This is evidence against the common-effect model at the 0.05 level. Most of Q comes from the third estimate’s departure from the weighted center. The test does not establish why that cohort differs; possible explanations include real effect differences, incompatible analysis choices, or a data or harmonization problem.
Worked Example: A Non-Significant Test With Only Two Cohorts
Two independent hypothetical cohorts report log odds ratios of \(0.10\) and \(0.20\), each with \(SE=0.15\). Suppose the same effect allele and outcome definition were used. With equal weights, the fixed-effect estimate is \((0.10+0.20)/2=0.15\). Each weight is \(1/(0.15)^2=44.44\).
Therefore,
There is one degree of freedom. Equivalently, \(\sqrt{Q}=0.4714\), and the chi-square upper-tail p-value is approximately \(2[1-\Phi(0.4714)]=0.637\). This does not indicate detectable heterogeneity, but it is weak evidence about consistency: only two estimates are being compared, and both are imprecise. The non-significant result should not be reported as confirmation that the true effects are equal.
Power, Interpretation, and Failure Modes
The non-significant result in the two-cohort example illustrates a central limitation of Q. The test’s ability to detect heterogeneity depends on the number and precision of cohorts as well as on the size of the effect differences. Few cohorts leave few residual degrees of freedom, and imprecise cohort estimates can obscure meaningful differences. Therefore, a large p-value means that the data do not provide strong evidence against a common effect; it does not show that between-cohort variation is absent or negligible.
The test also has limits as a summary. A small p-value indicates that the observed spread is difficult to explain under the assumed common-effect model, but it does not measure the magnitude of heterogeneity in an interpretable effect-size unit. Nor does Q identify which cohort, study feature, or biological mechanism explains the spread. A useful interpretation reports the statistic, degrees of freedom, p-value, and the cohort estimates with their standard errors, while describing the result as evidence for or against consistency under the stated assumptions.
Before relying on the reference distribution, check whether the inputs satisfy the model. Overlapping participants can make estimates correlated; then the diagonal covariance assumed by the usual formula is wrong, and the usual chi-square calibration may not apply. Incompatible effect alleles, outcome scales, or phenotype definitions can likewise make the common-effect null inappropriate. Poorly estimated standard errors or non-normal estimates can also distort the test. These issues should be investigated rather than treated as evidence of biological heterogeneity.
Cochran’s Q is distinct from the fixed-effect association test, which assesses whether the pooled association effect differs from zero. It is also distinct from simply inspecting whether cohort estimates have the same sign: Q uses both their distances from the pooled estimate and their precision. A random-effects analysis addresses a different modeling aim by allowing underlying effects to vary; Q is a test of the common-effect restriction, not a substitute for choosing or fitting an alternative model.
Practical Calculation Sequence
A reliable calculation keeps the estimate, its uncertainty, and the interpretation connected to the genetic association being analyzed.
Check that all cohort estimates target the same effect, use harmonized alleles and compatible outcome scales, and come from independent cohorts for the standard test.
For each cohort, set \(w_i=1/SE_i^2\), then calculate \(\hat{\beta}_{FE}=\sum_i w_i\hat{\beta}_i/\sum_i w_i\).
Sum \(w_i(\hat{\beta}_i-\hat{\beta}_{FE})^2\), use \(K-1\) degrees of freedom, and obtain the upper-tail chi-square p-value.
State whether the test provides evidence against a common effect, and do not treat a non-significant p-value as proof of homogeneity.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What null hypothesis does Cochran’s Q test, and how is it different from the null hypothesis in a test of the pooled genetic association?
- For four independent cohort estimates with compatible effects and valid standard errors, what are the degrees of freedom for the usual Q test?
- Two cohorts have estimates \(0.30\) and \(0.10\), each with standard error \(0.10\). Calculate their fixed-effect estimate and Q.
- Why can a non-significant Q test be especially uninformative when only a few cohorts are available?
- Name two conditions that can make the usual chi-square calibration inappropriate.
Solutions. (1) Q tests whether the underlying cohort effects are equal; a pooled association test asks whether the common effect differs from zero. (2) The degrees of freedom are \(K-1=3\). (3) The equal weights are \(1/(0.10)^2=100\), so \(\hat{\beta}_{FE}=[100(0.30)+100(0.10)]/200=0.20\). Then \(Q=100(0.30-0.20)^2+100(0.10-0.20)^2=1+1=2.00\). With one degree of freedom its p-value is about \(0.157\), so it is not significant at 0.05. (4) A small number of cohorts provides little information about the spread of true effects, and imprecise estimates may conceal differences; failure to reject is not evidence that effects are identical. (5) Examples include overlapping participants that induce correlated estimates, incompatible effect alleles or outcome scales, and inaccurate standard errors or poor normal approximations.