Why Multiplicity Matters in Competing-Risks Studies
A study may compare groups for several causes of failure, or compare several pairs of groups for one cause. Each additional test creates another opportunity to declare a difference when none exists. If the scientific conclusion is based on the collection of tests, controlling the error rate for each test separately does not generally control the error rate for the collection.
For competing-risk outcomes, a test is usually tied to a specified cause and a specified comparison. Gray’s test assesses equality of the cumulative incidence functions (CIFs) across groups for that cause. A study that tests several causes therefore has several related Gray tests. A study with three or more groups may also test several pairwise contrasts after an omnibus comparison. The multiplicity plan should specify which hypotheses belong to the same family before examining the results.
The family is a scientific and design choice, not merely a software setting. For example, researchers might define the family as tests of three prespecified causes, or as all pairwise contrasts for the primary cause. If both sets will support confirmatory claims, they may need joint consideration, or a prespecified hierarchy that allocates error across them. Defining separate families after seeing which results are promising does not provide the planned error control.
Bonferroni and Holm Adjustments
For \(m\) hypotheses and family-wise level \(\alpha\), the Bonferroni procedure tests each hypothesis at level \(\alpha/m\). Equivalently, it rejects a hypothesis with unadjusted p-value \(p_j\) when \(p_j \leq \alpha/m\), or reports the adjusted p-value \(\min(1,mp_j)\). This control holds regardless of dependence among the test statistics, provided each individual p-value is valid under its null hypothesis.
Holm’s step-down procedure also controls FWER without requiring independence. Sort the unadjusted p-values as \(p_{(1)} \leq \cdots \leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\), \(p_{(2)}\) with \(\alpha/(m-1)\), and continue. Reject in order until the first comparison that fails; that hypothesis and all subsequent ones are not rejected. Holm is never less powerful than the ordinary Bonferroni procedure for the same family and level.
Holm-adjusted p-values can be calculated by multiplying \(p_{(i)}\) by \(m-i+1\), then enforcing nondecreasing adjusted values in the sorted order and capping at 1. In symbols, the adjusted value for the \(i\)th sorted test is the smaller of 1 and the maximum of the preceding raw products \((m-j+1)p_{(j)}\) for \(j\leq i\). The maximum operation preserves the required ordering.
Worked Example: Holm Adjustment for Three Cause-Specific Tests
Suppose a hypothetical study prespecified Gray tests for three competing causes, with raw p-values \(0.0205\), \(0.008\), and \(0.041\). The family-wise level is \(\alpha=0.05\). These p-values are assumed valid for their respective null hypotheses, and all three tests are in the same family.
Step 1: Sort the p-values. The ordered values are \(p_{(1)}=0.008\), \(p_{(2)}=0.0205\), and \(p_{(3)}=0.041\).
Step 2: Apply Holm’s sequential thresholds. The first comparison is \(0.008 \leq 0.05/3=0.0167\), so reject the smallest-p-value null. Next, \(0.0205 \leq 0.05/2=0.025\), so reject the second. Finally, \(0.041 \leq 0.05\), so reject the third. All three cause-specific null hypotheses are rejected by this procedure at family-wise level 0.05.
Step 3: Compute adjusted p-values. The raw products are \(3(0.008)=0.024\), \(2(0.0205)=0.041\), and \(1(0.041)=0.041\). They are already nondecreasing, so the Holm-adjusted p-values in sorted order are \(0.024\), \(0.041\), and \(0.041\). Each is at most 0.05, consistent with the sequential decisions. These results provide evidence against the corresponding equality-of-CIF null hypotheses; they do not establish that group membership caused the differences.
Holm is often a useful default for a finite family of confirmatory tests because it retains strong FWER control while usually being less conservative than testing every hypothesis at \(\alpha/m\). Bonferroni is simpler and gives a direct per-test threshold, which is useful for planning. Both methods can lose power when many tests are included, especially when the true effects are modest.
From Adjusted Significance Level to Gray-Test Sample Size
Multiplicity affects design as well as analysis. A stricter per-test significance level requires stronger evidence to reject a null hypothesis and generally increases the sample size needed to attain a chosen power. The precise calculation for Gray’s test depends on the planned groups, allocation, event and censoring patterns, competing-event incidence, and the alternative CIFs. A simple approximation can illustrate the effect of changing the significance level, but it does not replace a design calculation matched to the study.
For a prespecified two-group comparison, an often-used planning approximation has the same standardized-effect form as a two-sided normal test. Let \(\delta\) denote the standardized effect on the Gray-test planning scale, \(\beta\) the target type II error probability, and \(N\) the approximate total sample size under equal allocation. The approximation is
where \(\alpha^\ast\) is the per-comparison significance level used for planning and \(z_q\) is the \(q\) quantile of the standard normal distribution. This expression assumes a fixed effect and a design in which the stated standardized effect adequately summarizes the test’s information. It is a planning approximation, not a universal Gray-test formula. In particular, changing the cause can change event information and therefore \(\delta\); that change must not be ignored when it is material.
Worked Example: Bonferroni’s Approximate Sample-Size Cost
Consider a hypothetical two-group study planning one primary cause-specific Gray comparison with standardized effect \(\delta=0.200\) and power \(1-\beta=0.80\). Suppose the unadjusted two-sided level is 0.05. The normal quantiles are \(z_{0.975}=1.96\) and \(z_{0.80}=0.8416\). Assume equal allocation and that the approximation above is suitable for this planning illustration.
Without multiplicity adjustment, the calculation is
Rounding up gives a target of 197 participants. Now suppose the primary analysis includes five confirmatory cause-specific comparisons and uses Bonferroni control of FWER at 0.05. The planned per-test level is \(\alpha^\ast=0.05/5=0.010\), for which \(z_{1-\alpha^\ast/2}=z_{0.995}\approx2.576\). Thus
Rounding up gives 292 participants. Under these assumptions, Bonferroni increases the approximate requirement from 197 to 292 participants, an increase of 95. This comparison holds the standardized effect fixed across tests. If the cause-specific event rate, competing-event frequency, or censoring differs by cause, a single fixed \(\delta\) may not be appropriate; planning should then use cause-specific information or a more tailored calculation.
Omnibus Tests, Contrasts, and the Testing Family
With more than two groups, an omnibus Gray test for one cause asks whether all group CIFs for that cause are equal. Its large-sample test statistic is compared with a chi-square reference distribution with \(K-1\) degrees of freedom for \(K\) groups. Rejecting that omnibus null establishes evidence of some group difference, not which groups differ. Pairwise comparisons are a separate set of hypotheses and require a plan for their multiplicity if they will support claims.
A useful design may test one omnibus hypothesis for each of several causes, then conduct specified contrasts only when an omnibus test is significant. This gatekeeping strategy can reduce unnecessary testing, but the complete decision rule and error allocation need to be prespecified. Simply performing unadjusted follow-up contrasts after any significant omnibus result does not automatically guarantee the desired FWER across all causes and contrasts.
Worked Example: Defining Families Across Causes and Group Contrasts
A hypothetical trial has three treatment groups and four competing causes. Investigators plan one omnibus Gray test per cause, giving four primary tests, and intend to compare all three pairs of groups for the primary cause if its omnibus test is significant. They choose Bonferroni for the four primary cause tests at family-wise level 0.05.
For the primary family, there are \(m=4\) tests, so each omnibus test uses \(\alpha^\ast=0.05/4=0.0125\). Each omnibus Gray test has \(K-1=3-1=2\) degrees of freedom. For example, a hypothetical cause-specific p-value of 0.010 is below 0.0125 and would reject that cause’s omnibus equality null; a p-value of 0.018 would not.
For the primary cause’s three pairwise contrasts, a separate Holm family has \(m=3\) and family-wise level 0.05. If the raw contrast p-values are 0.012, 0.030, and 0.044, the first Holm comparison is \(0.012 \leq 0.05/3=0.0167\), so it is rejected. The second comparison fails because \(0.030>0.05/2=0.025\); Holm stops, and neither the second nor third contrast is rejected. This procedure controls error within the contrast family. If investigators require joint control over both the four primary tests and the contrasts, they must prespecify a method that covers that joint set, rather than treating separate within-family guarantees as a joint guarantee.
For planning, the multiplicity burden can therefore differ by analysis stage. Four primary omnibus tests may use Bonferroni or Holm at the primary family level, while a prespecified contrast procedure follows a successful omnibus result. The power target should be tied to the claim of interest: power for an omnibus difference is not the same as power for a particular pairwise contrast, and it is not automatically the probability of identifying every true difference.
Justification, Diagnostics, and Limitations
Bonferroni’s control follows directly from the union bound. Let \(A_j\) be the event of falsely rejecting the \(j\)th null when it is true. The probability of at least one false rejection is no greater than the sum of the individual false-rejection probabilities:
This justification does not require independent tests. Holm’s procedure inherits FWER control by applying the same Bonferroni bound sequentially to the remaining hypotheses; its step-down thresholds allow larger p-values later in the sequence while preserving the overall bound.
Before applying a correction, check that each input p-value comes from the intended test and is valid under its null. For Gray tests, that means the groups, cause, follow-up interval, and censoring conditions match the planned analysis. The earlier material on censoring conditions and diagnosing group differences in follow-up is relevant: differing follow-up or inappropriate censoring assumptions can undermine the p-values, and multiplicity adjustment cannot repair invalid individual tests. Also verify that the stated number of tests matches the prespecified family, including planned contrasts and any additional causes on which confirmatory claims will be made.
The normal sample-size calculation is most useful for a transparent first approximation. It can mislead if the effect size is not stable across causes, if allocation is unequal, or if accrual, administrative censoring, loss to follow-up, or competing-event rates materially change the information available. Gray’s test does not require proportional subdistribution hazards, but its power still depends on the pattern of CIF differences over time and on the event and censoring experience. A single standardized effect cannot fully represent crossing or delayed differences. More detailed design calculations should reflect the planned CIF alternatives and follow-up scheme.
Bonferroni and Holm target FWER: they limit the chance of at least one false rejection in the specified family. Procedures controlling the false discovery rate instead target the expected proportion of false rejections among rejected hypotheses; that is a different error criterion and may suit exploratory screening, but it is not interchangeable with confirmatory FWER control. A Šidák adjustment can be slightly less conservative than Bonferroni under independence, but dependence among cause-specific tests makes that assumption important. When dependence is uncertain, Bonferroni or Holm offers a straightforward guarantee.
- Wrong: “Each Gray test is at 0.05, so the five-test study has a 5% chance of any false positive.” Correction: 0.05 is a per-test level; the family-wise error may exceed 0.05. Apply a prespecified family-wise procedure.
- Wrong: “A significant three-group omnibus test proves every pair of groups differs.” Correction: It indicates evidence that not all CIFs are equal; identify particular differences with planned contrasts and appropriate multiplicity handling.
- Wrong: “Bonferroni changes only reporting, not design.” Correction: A smaller per-test level raises the approximate sample size required for a fixed power and effect.
- Wrong: “Multiplicity adjustment fixes poor censoring assumptions.” Correction: Adjustments address the number of tests, not the validity of each test or the assumptions underlying its p-value.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Four valid cause-specific tests are in one confirmatory family. What Bonferroni level is used for each test if FWER is 0.05?
- Three sorted p-values are 0.006, 0.019, and 0.049. Apply Holm’s procedure at family-wise level 0.05.
- Why does a significant omnibus Gray test with four groups not establish that every pair differs?
- In the normal approximation, what happens to required sample size when the per-test significance level decreases while effect and power stay fixed?
Solutions. (1) The per-test level is \(0.05/4=0.0125\). (2) Compare \(0.006\) with \(0.05/3=0.0167\): reject. Compare \(0.019\) with \(0.05/2=0.025\): reject. Compare \(0.049\) with \(0.05\): reject. All three null hypotheses are rejected by Holm. (3) The omnibus null is equality of all four CIFs; rejecting it shows that at least one difference is present, but does not identify which pairs differ. (4) The critical normal quantile increases as the level decreases, so the squared sum in the numerator increases and the required approximate sample size rises.