Falsification Tests for Causal Claims

What a Falsification Test Can Establish

A causal explanation makes claims beyond the size of its estimated treatment effect. It may also imply that treatment cannot affect an outcome measured before treatment, cannot affect a particular unrelated outcome, or cannot produce an effect larger than a mechanistic limit. A falsification test checks one of these implications. A clear violation can challenge the proposed explanation or the analysis that supports it.

The test is not a direct measurement of unmeasured confounding. Rejection may indicate residual confounding, but it may also reflect a faulty placebo choice, selection, measurement problems, model misspecification, or chance. Conversely, not rejecting a falsification restriction does not show that confounding is absent: the test may be imprecise, or the chosen placebo may fail to share the relevant sources of bias.

Let \(A\) denote treatment, \(Y\) the outcome of interest, \(L\) measured pre-treatment covariates, and \(Y(a)\) the potential outcome under treatment level \(a\). For a binary treatment, one common target is the average treatment effect, \(E[Y(1)-Y(0)]\). The usual causal interpretation of an adjusted estimate requires assumptions such as consistency, positivity, and conditional exchangeability given \(L\). A falsification test does not make these assumptions true; it probes implications of a proposed design and causal account.

A useful test begins with a precise restriction and a reason it should hold. The null might be a zero adjusted contrast for a placebo outcome, or that a causal effect cannot exceed a specified bound. The estimate and its uncertainty should be calculated using a method appropriate to the study design, including clustering or repeated measurements where relevant. A test result is only as credible as this setup and the justification for the restriction.

Placebo-Time Tests

A placebo-time test asks whether treatment appears to affect an outcome measured before treatment could have acted. Let \(Y^{-}\) be an outcome measured before treatment assignment or initiation. If treatment truly begins after that measurement, the causal effect of the later treatment on \(Y^{-}\) must be zero by temporal order. An adjusted association between \(A\) and \(Y^{-}\) can therefore reveal a problem with the proposed comparison, such as remaining differences between treated and untreated groups.

This interpretation requires accurate timing and a clearly defined treatment start. If treatment assignment or anticipation could change behavior before the recorded start date, the placebo period may not be truly pre-treatment. The analysis should also avoid adjusting for variables caused by treatment. A pre-treatment outcome can be especially informative when it is measured comparably across groups and has a plausible relationship to the later outcome of interest.

A related design uses several pre-treatment measurements to check whether treated and comparison groups were already changing differently before treatment. Such checks can challenge a proposed causal account, but a visually or statistically similar pre-period pattern does not prove that the groups would have followed the same pattern afterward. The pre-treatment record is evidence about the design, not a guarantee about unobserved counterfactual outcomes.

Placebo-Outcome Tests

A placebo-outcome test replaces \(Y\) with an outcome \(Y^{pl}\) that the treatment cannot cause, while seeking an outcome that may share relevant confounding or measurement processes with \(Y\). The expected causal effect of \(A\) on \(Y^{pl}\) is zero. If an adjusted treatment contrast remains, that association can be a warning that the adjustment or study design has not removed all relevant differences.

The choice of placebo matters. An outcome unrelated to treatment and unrelated to the confounding structure may be uninformative: no association with treatment would be expected even if the main analysis were confounded. Conversely, a supposed placebo may actually be affected by treatment through a less obvious pathway, such as altered health-care contact or outcome detection. Investigators should explain both why a causal pathway is absent and why the placebo is exposed to sources of bias similar to those affecting the main outcome.

Impossible-Effect Tests and Mechanistic Bounds

Some causal explanations imply more than a zero placebo effect. A known constraint may limit the size of an effect, even if an effect is possible. An impossible-effect test compares the estimated effect with such a limit. The constraint must be justified by substantive knowledge and stated precisely; a convenient numerical threshold chosen after seeing the estimate is not a valid mechanistic bound.

For example, suppose \(Y\) is binary and a treatment can affect the outcome only in a pre-specified subgroup \(S\). Assume that no individual outside \(S\) can have their outcome changed by treatment, and let \(q=P(S)\). Then the absolute average treatment effect cannot exceed \(q\). This follows because an individual binary-outcome effect is between \(-1\) and \(1\), and effects outside \(S\) are zero:

Here \(I(S)\) equals 1 for members of \(S\) and 0 otherwise. The first equality uses the assumption that the individual effect is zero outside \(S\); the last inequality uses that \(Y(1)\) and \(Y(0)\) are binary, so their difference has absolute value at most 1. The bound is about the average effect on the risk-difference scale, not a bound on an odds ratio or a regression coefficient.

Planning and Interpreting the Tests

A well-designed falsification test states the restriction before examining the result, gives a substantive reason for it, and uses an analysis that matches the data structure. The following sequence makes those decisions explicit.

Multiple falsification tests can be useful, but they raise interpretation issues. If many tests are run, at least one may reject by chance even when all restrictions hold. Report which tests were planned and consider the multiplicity when making a joint claim. A cluster of coherent violations may be more concerning than one isolated result, but the tests are not automatically independent or equally informative.

Common Mistakes

• Calling a non-significant placebo result proof of no confounding. A large p-value may reflect low precision or a placebo that does not share the relevant bias. Examine the interval and the rationale for the placebo.
• Assuming every association with a placebo proves confounding. A rejected restriction can also reflect chance, a flawed placebo, selection, measurement, or model error. It signals a problem to investigate, not a unique diagnosis.
• Using an outcome that treatment can influence indirectly. For example, treatment may change health-care contact and therefore the chance that an outcome is recorded. Explain why the full measurement process is not affected.
• Choosing a bound after seeing the estimate. A mechanistic limit must be independently justified. If its assumptions are not credible, exceeding it does not falsify the causal claim.
• Treating the test as a correction. A falsification test does not supply a bias-adjusted causal effect. It assesses a restriction; it does not quantify or remove the bias in the main estimate.