Why Multiplicity Matters in Exposure–Response
An exposure–response analysis can involve several plausible exposure summaries, such as area under the concentration–time curve, maximum concentration, or a time-weighted measure. Investigators may also examine multiple outcomes, subgroups, time windows, or functional forms. Each additional analysis can be scientifically reasonable; the statistical problem arises when the most favorable result is selected for emphasis after the results are known.
If a null hypothesis is true, a valid test can still produce a small p-value by chance. Testing many hypotheses therefore increases the chance of at least one false rejection. This risk is particularly relevant when a selected exposure metric or subgroup result is presented as if it were the sole planned analysis. Multiplicity methods do not establish that an exposure causes an outcome, correct confounding, or repair a misspecified model. They address the error rate created by testing a defined family of hypotheses.
The family should follow the intended claim, not merely the table or figure in which results appear. For example, a claim that “exposure is related to clinical response” may draw on tests across several exposure metrics, endpoints, and model forms if those analyses were all searched and the strongest result selected. Defining families requires scientific judgment, but making that judgment before examining results reduces the opportunity to define a conveniently narrow family after the fact.
Multiplicity is separate from the validity of each individual test. A p-value can be invalid because of model misspecification, unaccounted data-dependent selection, or an inappropriate sampling distribution; adjusting it for the number of tests does not fix those problems. Conversely, even individually valid tests can produce a high familywise false-positive risk when many are run.
Quantifying the Risk and Controlling It
Suppose \(m\) null hypotheses are all true, and test \(j\) rejects with probability \(q_j\), where validity at level \(\alpha\) ensures \(q_j\leq\alpha\). If the tests are independent, the probability of no false rejection is \(\prod_{j=1}^{m}(1-q_j)\), so:
The upper bound is an equality when each test rejects with probability exactly \(\alpha\) under its null. For example, independent uniform p-values with the rule \(p\leq\alpha\) have this property. Validity alone does not imply equality: if every p-value is always 1, each test is valid but the FWER is 0. In exposure–response analyses, tests are often correlated because they use the same participants, related exposure metrics, or related endpoints. Independence should not be assumed merely because the hypotheses are distinct.
Worked Example: Eight Independent Exposure–Response Tests
Hypothetical data. A team conducts eight tests at \(\alpha=0.05\), one for each of eight prespecified combinations of exposure metric and outcome. Assume that all eight null hypotheses are true, all p-values are valid, and the tests are independent. What can be said about the chance of at least one false rejection?
Validity gives \(q_j\leq0.05\) for each test. Independence therefore gives the upper bound:
Thus the FWER is no greater than 0.3366 under these assumptions. It equals 0.3366 only if all eight tests have exact size 0.05, such as when each null p-value is uniform. If some tests are conservative, the actual probability can be lower; if all p-values are always 1, it is 0. The calculation concerns false rejections under the stated joint null, not the probability that any particular exposure–response association is real.
A simple control that does not require independent tests is the Bonferroni procedure. For \(m\) tests, test each hypothesis at \(\alpha/m\), or equivalently compare its valid raw p-value with \(\alpha/m\). The justification is the union bound: the probability of any false rejection is no greater than the sum of the probabilities of false rejection for the true null hypotheses.
where \(\mathcal{T}\) is the set of true null hypotheses in the family. This establishes strong FWER control: the bound holds whether all nulls are true or only some are true. Bonferroni can be conservative, particularly when many tests are strongly correlated, but it is transparent and valid under arbitrary dependence when each individual p-value is valid.
Worked Example: Bonferroni and Holm for Candidate Exposure Metrics
Hypothetical data. Investigators prespecify five tests of the exposure–response association for a binary safety outcome, using five candidate exposure summaries. The raw p-values are 0.004, 0.018, 0.031, 0.044, and 0.120. Treat these as valid p-values for their respective null hypotheses. The familywise significance level is \(\alpha=0.05\).
Bonferroni. The per-test cutoff is \(0.05/5=0.010\). Only the p-value 0.004 is at or below 0.010, so Bonferroni rejects only that null hypothesis. Its Bonferroni-adjusted p-value is \(5(0.004)=0.020\); the other adjusted values are 0.090, 0.155, 0.220, and 0.600.
Holm step-down. Order the p-values from smallest to largest and compare the \(i\)th p-value with \(\alpha/(m-i+1)\), stopping at the first failure. The comparisons are:
The first null is rejected, but the procedure stops at the second comparison; it does not reject the remaining hypotheses. Holm therefore reaches the same set of rejections here. In another data set it can reject more hypotheses than Bonferroni while still controlling FWER under arbitrary dependence. Neither procedure identifies which metric is scientifically most appropriate; that choice needs scientific justification rather than the smallest p-value.
The Holm procedure controls FWER by testing the smallest p-value first at \(\alpha/m\), then the next at \(\alpha/(m-1)\), continuing in this way. If any comparison fails, all hypotheses at that rank and later ranks are retained. To see why the control holds, suppose there are \(r\) true null hypotheses. If the procedure falsely rejects any of them, it must reject the smallest true-null p-value in the ordered list. At that step, at most \(m-r\) false nulls precede it, so its cutoff is no larger than \(\alpha/r\). The probability that any of the \(r\) true-null p-values is at most \(\alpha/r\) is no greater than \(r(\alpha/r)=\alpha\), by validity and the union bound. This argument does not require independence.
Prespecification and Hierarchical Testing
The most direct way to limit avoidable multiplicity is to define the primary exposure metric, endpoint, population, analysis method, and decision rule before examining outcome results. Prespecification does not mean that only one analysis can ever be conducted. It means the confirmatory claim is tied to the planned analysis, while additional analyses are identified as secondary or exploratory rather than silently substituted when they produce more attractive results.
When several claims have a scientific order, a fixed-sequence procedure can preserve FWER without dividing \(\alpha\) across every test. The order and hypotheses must be specified in advance. Test the first hypothesis at \(\alpha\); proceed to the next only if the first is rejected, and continue in order. Stop at the first non-rejection. This controls FWER under arbitrary dependence: if any true null is falsely rejected, all hypotheses before the first true null must have been rejected, and the first true null is tested at level \(\alpha\). The chance of reaching and falsely rejecting it is therefore no greater than \(\alpha\).
Worked Example: A Prespecified Efficacy Testing Sequence
Hypothetical data. A development team prespecifies three ordered claims: first, that the primary clinical endpoint is associated with the primary exposure metric; second, that a key secondary endpoint is associated with that metric; and third, that a specified subgroup has an exposure–response association. The tests are valid, and the fixed sequence is tested at \(\alpha=0.05\). Their p-values in the planned order are 0.012, 0.028, and 0.041.
The first p-value is below 0.05, so the primary null is rejected and testing proceeds. The second is also below 0.05, so the key secondary null is rejected and testing proceeds. The third is below 0.05, so it too is rejected under the prespecified sequence. FWER is controlled at 0.05 for these ordered claims, without requiring independence.
If instead the first p-value were 0.080, testing would stop there. The second and third p-values could still be reported descriptively, but they would not support confirmatory rejections under this sequence, even if one were very small. The order is part of the procedure, not something to rearrange after seeing the p-values.
Selective Reporting, Alternatives, and Diagnostics
Selective reporting occurs when the analysis search is wider than the set of results disclosed—for example, testing several exposure summaries, functional forms, subgroups, or endpoints, then reporting only the most favorable association without explaining the search. The nominal p-value for the selected analysis does not account for that selection. Even if the underlying individual tests are valid before selection, presenting the selected result as if it were the only test can mislead readers about the strength of evidence.
If analyses were exploratory, the appropriate response is not to conceal them or to label a post hoc choice as prespecified. Report the candidates considered, the selection process, the analyses that were conducted, and the fact that the findings are exploratory. Provide effect estimates and uncertainty intervals, not only selected p-values, and distinguish hypotheses generated for future evaluation from confirmatory conclusions. An exploratory finding may be scientifically useful without having confirmatory error-rate control.
Bonferroni and Holm control the probability of one or more false rejections. A different objective is to control the false discovery rate (FDR), the expected proportion of false rejections among all rejections, defined as zero when there are no rejections. FDR procedures, such as the Benjamini–Hochberg procedure under its stated dependence conditions, can be useful for a larger exploratory screen where some false discoveries are tolerable. FDR control is not the same as FWER control and should not be presented as such. The choice depends on the consequences of false claims and the purpose of the analysis.
Useful diagnostics are primarily about the analysis process and its assumptions. Check that the stated family includes all analyses that informed the claim; verify that the test-level p-values are valid for the fitted models and planned comparisons; and confirm that the adjustment or hierarchy was chosen before results were selected. Discrepancies between a protocol and the reported analysis, unexplained switching among exposure metrics, or a subgroup highlighted only after its result is known are warnings of selection. Documenting a complete analysis inventory makes these issues visible.
A multiplicity-adjusted result is not automatically clinically important. The exposure contrast, effect size, uncertainty, biological plausibility, and limitations of the exposure–response analysis still matter. Adjustment can make a confirmatory statistical claim more defensible, but it does not establish causality or remove residual confounding. These distinctions are essential when exposure–response findings inform clinical interpretation or later development decisions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Six independent tests use valid p-values at \(\alpha=0.05\), and all six null hypotheses are true. What is the upper bound on FWER? When is it an equality?
- For four tests, a valid raw p-value is 0.009 and \(\alpha=0.05\). Does Bonferroni reject this hypothesis? Show the cutoff.
- In a fixed sequence, the first p-value is 0.06 and the second is 0.001 at \(\alpha=0.05\). Which hypotheses can be rejected by this procedure?
- Why does reporting only the smallest p-value after trying several exposure metrics create a problem, even when each test’s p-value is valid?
Solutions. (1) The upper bound is \(1-(1-0.05)^6=1-0.95^6=0.2649\) (to four significant figures). It is attained when each independent test has exact size 0.05 under its null; validity alone gives only the bound. (2) The Bonferroni cutoff is \(0.05/4=0.0125\), and \(0.009\leq0.0125\), so the hypothesis is rejected under this procedure. (3) The first hypothesis is not rejected because \(0.06>0.05\); the procedure stops, so the second is not rejected confirmatorily even though its p-value is 0.001. (4) Selecting the smallest p-value makes the reported result depend on the search. Its nominal p-value describes the test before selection, not the chance of obtaining a result that small somewhere in the full set of tried metrics. The search should be accounted for or the finding transparently labeled exploratory.