Heterogeneity Testing With Cochran's Q

Why Test Heterogeneity?

With multiple genetic variants as instruments, each variant can yield a variant-specific causal estimate. If the variants are compatible with one common causal effect, their estimates will differ because of sampling error, but should not be more dispersed than their standard errors predict. Cochran’s Q summarizes that dispersion relative to the estimated uncertainty.

In Mendelian randomization (MR), a large Q can signal that a single shared-effect description does not fit the variant-specific estimates well. It does not identify why they disagree. Horizontal pleiotropy is one possible explanation, but neither a large nor a small Q proves that pleiotropy is present or absent.

Model and Assumptions

The test concerns whether the variant-specific estimates are consistent with a common value, not whether that value is zero. A convenient model under the null is \(\hat{\theta}_j=\theta+\varepsilon_j\), where \(\theta\) is the common effect and the errors are independent and approximately normal with mean zero and variance \(s_j^2\). The standard errors must adequately represent the sampling uncertainty in the estimates.

In MR, \(\hat{\theta}_j\) is often a Wald ratio: the variant-outcome association divided by the variant-exposure association. Its standard error must be appropriate for that ratio. If exposure-association uncertainty is ignored, the familiar outcome-association-based approximation relies on the exposure associations being sufficiently precise. With weak instruments, ratio estimates can be poorly approximated by normal distributions and their reported standard errors may not support the usual Q reference distribution.

The ordinary formula also assumes the variant-specific estimates are independent. Clumping is intended to reduce dependence from linkage disequilibrium, but correlated estimates can remain, for example because of residual linkage disequilibrium or overlapping samples. If the estimates have a known covariance matrix \(V\), the independent-estimate formula is not generally appropriate; a covariance-aware residual statistic is needed. For the standard Q test, the common-effect model, reliable uncertainty estimates, approximate normality, and independence are the working conditions.

Why the Null Distribution Has \(K-1\) Degrees of Freedom

The degrees of freedom account for estimating the common effect from the same \(K\) variant-specific estimates. The following argument shows why, under the stated normal model and known variances, \(Q\) has a chi-square distribution with \(K-1\) degrees of freedom.

Let \(y_j=\hat{\theta}_j/s_j\), and let \(a\) be the vector with entries \(a_j=(1/s_j)/\sqrt{\sum_{\ell=1}^{K}1/s_\ell^2}\). The vector \(a\) has length one. Under the null, the standardized estimates have the form \(y=\theta(1/s_1,\ldots,1/s_K)^\mathsf{T}+\epsilon\), where \(\epsilon\) has independent standard normal entries. The weighted residuals remove the component of \(y\) in the direction \(a\), so

The second equality holds because \(I-aa^\mathsf{T}\) removes the common-effect component, which is in the direction \(a\). This matrix is an idempotent projection: it retains the \(K-1\) directions perpendicular to \(a\) and removes one direction. The squared length of a standard normal vector projected onto a \(K-1\)-dimensional subspace has a \(\chi^2_{K-1}\) distribution. Hence, under these assumptions, \(Q\sim\chi^2_{K-1}\). In applications, estimated standard errors and approximate ratio normality make this reference distribution an approximation.

The p-value is the probability, under the common-effect null model and its assumptions, of obtaining a Q statistic at least as large as the observed value. A small p-value indicates that the observed dispersion is difficult to explain by the specified sampling variation alone. It is not the probability that the common-effect model is true.

Weights Matter

Q measures dispersion relative to uncertainty, not raw differences alone. A discrepancy involving a precise variant contributes more than the same discrepancy involving an imprecise variant. This is why Q uses inverse-variance weights rather than an unweighted sum of squared deviations.

What a Large or Small Q Can Tell You

A large Q is evidence of statistical heterogeneity: the variant-specific estimates do not fit the common-effect model well relative to their estimated precision. In MR, one possible explanation is that some variants have direct effects on the outcome not mediated by the exposure. Other explanations include different causal effects across the genetic perturbations, errors in harmonization, poor uncertainty estimates, residual dependence among variants, or problems with the normal approximation for weak-instrument ratios. The test does not distinguish these explanations.

Conversely, a small Q is not proof of homogeneity. Q has limited power when there are few variants, when estimates are imprecise, or when departures from a common effect are modest relative to sampling error. It can also fail to flag some forms of pleiotropy. For instance, if direct effects shift variant-specific estimates in a similar direction and amount, the estimates may still cluster around a common value. As discussed in the earlier material on direct and correlated pleiotropy, consistency across instruments is not the same as proof that every instrument satisfies the exclusion restriction.

Diagnostics, Comparisons, and Limitations

A Q p-value is best considered alongside the variant-specific estimates, their standard errors, and their directions. A forest-style display can reveal whether one estimate is unusually far from the others or whether dispersion is spread across several variants. That inspection describes the pattern; it does not, by itself, establish the cause. Check allele harmonization, instrument selection, and whether dependence between estimates has been adequately handled before interpreting a large statistic.

Q is a test of fit around the inverse-variance weighted common estimate, not a separate causal-effect estimator. A fixed-effect IVW analysis uses that common estimate and assumes a shared underlying effect for its standard interpretation. A multiplicative random-effects IVW analysis allows extra dispersion to affect uncertainty estimates, but it does not identify or remove the source of heterogeneity. Neither approach turns Q into a test that specifically diagnoses pleiotropy.

The ordinary Q formula is also not a substitute for accounting for correlated variants. When estimates are dependent, their covariance affects the expected residual dispersion; treating them as independent can miscalibrate the test. More generally, the chi-square p-value is only as credible as the model and uncertainty estimates supporting it. Q summarizes inconsistency, but gives no direct estimate of its biological cause and may have low power in a small instrument set.