Multiplicity in Families of Falsification Tests

Why a Family of Falsification Tests Needs a Plan

A set of negative-control analyses is most informative when its members are considered together. If many controls are tested and only the smallest p-value is highlighted, a result that appears striking may simply reflect the number of opportunities for a small p-value to occur. The issue is not that any one test is necessarily calculated incorrectly. It is that the interpretation changes when several hypotheses are examined and results are selected afterward.

The relevant planning decision is which hypotheses belong to the same family: the collection of tests whose results will jointly support a particular inferential claim or decision. For falsification analyses, a family might include the prespecified associations between a treatment and several negative-control outcomes. Its members should be chosen based on the scientific purpose and analysis plan, not on which tests later appear significant. The earlier material on triangulating evidence across multiple controls helps motivate examining patterns across controls; multiplicity procedures address the additional risk of false-positive findings within a defined family.

For each control, the null hypothesis might state that the treatment association with that control outcome equals zero on a prespecified scale. The alternative might be two-sided, \(H_a:\theta_j\ne 0\), or one-sided if a directional test was scientifically justified and specified in advance. Here, \(\theta_j\) denotes the association parameter for the \(j\)th control test. The falsification-test interpretation still depends on whether the control is appropriate and on the assumptions underlying the test. Adjusting p-values does not establish those conditions.

Bonferroni and Holm Adjustments

The Bonferroni procedure controls the FWER by testing each of \(m\) hypotheses at level \(\alpha/m\). Equivalently, it multiplies each raw p-value by \(m\), capped at 1, and compares the result with \(\alpha\). This bound is valid whether or not the tests are independent, provided each individual p-value is valid under its null hypothesis.

Here, \(\mathcal{H}_0\) is the set of true null hypotheses and \(m_0\) is its size, so \(m_0\le m\). The first inequality is the union bound: the probability of at least one event is no greater than the sum of their probabilities. This proves the familywise guarantee without requiring independence. Bonferroni can be conservative, particularly when tests are strongly correlated or many hypotheses are included.

Holm's step-down procedure usually offers at least as much power while also controlling the FWER under arbitrary dependence. Order the raw p-values from smallest to largest, \(p_{(1)}\le \cdots \le p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\). If it passes, compare \(p_{(2)}\) with \(\alpha/(m-1)\), continuing with \(p_{(k)}\) compared with \(\alpha/(m-k+1)\). Stop at the first failure; that hypothesis and all hypotheses with larger ordered p-values are not rejected. The procedure's adjusted p-value for rank \(k\) is the maximum of the values \( (m-i+1)p_{(i)}\) for \(i\le k\), capped at 1. The maximum makes the adjusted values nondecreasing in rank, as required for consistent step-down decisions.

To see why Holm controls the FWER, suppose there are \(m_0\) true null hypotheses and consider the first true null in the ordered list, at rank \(k\). Every hypothesis before rank \(k\) is false, so \(m-k+1\ge m_0\). If Holm rejects that true null, its p-value must satisfy \(p_{(k)}\le\alpha/(m-k+1)\le\alpha/m_0\). Therefore, a false rejection requires at least one of the \(m_0\) true-null p-values to be no greater than \(\alpha/m_0\). By the union bound, that probability is at most \(m_0(\alpha/m_0)=\alpha\). If the first true null is not rejected, the step-down rule stops before any later true null can be rejected.

Prespecify the Family Before Seeing Results

A useful plan names the family, the hypotheses, the effect scale, the testing direction, the multiplicity procedure, and the familywise level. It also states how missing or infeasible control analyses will be handled. This prevents the analyst from quietly redefining the family after learning which results are favorable. If the scientific question is whether any of a set of controls shows an association, treating all those tests as one family is generally more coherent than reporting only the one with the smallest p-value.

Not every analysis must belong to one enormous family. Distinct families can be defensible when they correspond to distinct prespecified claims or decisions. But dividing a set into smaller families merely to avoid adjustment does not preserve the stated error rate for the broader claim. If five tests are reported as one coordinated search for evidence of residual bias, their error rate should be considered together even if they concern different control outcomes.

The procedure should also match the intended interpretation. Bonferroni and Holm control the chance of one or more false rejections in the family. A false discovery rate (FDR) procedure instead targets the expected proportion of false rejections among all rejections, under its applicable conditions. FDR control can be useful for explicitly exploratory screening, but it does not provide the same guarantee as FWER control. Selecting a procedure because it gives more discoveries, without specifying what error rate the analysis is intended to control, makes the result difficult to interpret.

Report Effect Sizes Alongside Adjusted Evidence

Multiplicity adjustment changes the threshold for a decision or the p-value used for that decision. It does not change the estimated association itself. A report that lists only adjusted p-values conceals whether an association is small or large, in which direction it points, and how uncertain its estimate is. For each control, give the estimated association on its stated scale, a measure of uncertainty such as a confidence interval, the raw p-value, and the adjusted result. Do not imply that a nonsignificant adjusted test means the estimated association is exactly zero.

Ordinary confidence intervals for individual tests are typically marginal intervals: each interval has its stated coverage considered on its own. A collection of such intervals does not automatically provide simultaneous coverage for all parameters in the family. If simultaneous interval coverage is needed, use an appropriate simultaneous procedure, such as Bonferroni-adjusted intervals, and state that choice. This distinction matters when readers compare several intervals as a set.

Diagnostics, Comparisons, and Limitations

Before interpreting adjusted results, check that the family was actually prespecified, that every reported test is accounted for, and that each raw p-value is valid for its null under the analysis model. Dependence among tests is common when controls share participants or confounding pathways. It does not invalidate Bonferroni or Holm, but dependence can make Bonferroni conservative. More importantly, a multiplicity correction cannot repair model misspecification, invalid standard errors, selective omission of outcomes, or a control that fails its substantive assumptions.

A useful diagnostic is to compare the stated family with the set of analyses that could have been selected: outcomes, exposure definitions, subgroups, models, and testing directions. If many versions were tried and only the most favorable result is disclosed, adjusting a narrow family of reported tests does not account for that larger search. Report deviations from the plan and treat unplanned analyses as exploratory. Avoid interpreting an adjusted p-value as the probability that a null hypothesis is true; it is a tail probability computed under a null model and procedure, not a posterior probability.

The main alternatives answer different questions. An unadjusted test is appropriate when a single hypothesis was genuinely the sole prespecified test, but can understate false-positive risk when many related tests inform one claim. Bonferroni is simple and robust but may lose power; Holm retains FWER control and is never less powerful than Bonferroni for the same family and level. An omnibus test can test a joint null about a set of associations, but by itself may not identify which controls depart from their null values. FDR methods may suit broad exploratory screening when the intended error criterion is the expected false-discovery proportion, rather than the probability of any false rejection.

No multiplicity procedure makes a family of falsification tests a complete diagnostic of unmeasured confounding. A rejection can arise from a genuine association with a control, a violated control assumption, chance, or a problem in the analysis. A failure to reject may reflect limited power. The adjusted evidence should therefore be read together with effect sizes, uncertainty, design knowledge, and the substantive interpretation of the controls.