Ties, Conservative P-Values, and Combination Calibration

Why Ties Change Combination Calibration

In a discrete-outcome test, a statistic can take only a limited set of values. The resulting stagewise p-values therefore occur at a limited set of values too: many observations share the same p-value, and the null distribution has jumps. This matters when the stagewise p-values are combined. A reference distribution derived for continuous uniform p-values can still control Type I error, but it may no longer describe the actual null distribution closely. The difference is often conservatism: the test rejects less often than its nominal level.

The central question is not whether ties are present by themselves. It is whether each p-value is valid under the null, including after any adaptation, and whether the calibration respects its actual distribution. The earlier material on discrete outcomes supplies the relevant setting; here we focus on how discreteness carries through combination testing.

Definition: A p-value \(P\) is super-uniform under \(H_0\) if \(P(P\leq u)\leq u\) for every \(u\) between 0 and 1, under every null parameter value for which validity is claimed. A continuous exact p-value is uniform under a simple null; a discrete exact-tail p-value is generally super-uniform, with a stepwise distribution and possible ties.

For example, for an upper-tail test statistic \(T\), a nonrandomized exact p-value is \(P_{H_0}(T\geq t_{\mathrm{obs}})\). The inequality includes equality, so every tied outcome at the observed value contributes to the tail probability. The p-value may consequently be larger than a continuous-tail approximation would suggest. That conservatism is what preserves validity; replacing the inclusive tail probability with a smaller number without recalibration can lose Type I error control.

How Super-Uniform Inputs Affect a Combination

Consider a combination statistic \(C(p_1,p_2)\) that increases as either stagewise p-value decreases: smaller p-values give stronger evidence against \(H_0\). Suppose the stage-one p-value is super-uniform and the stage-two p-value is super-uniform conditional on all stage-one information and on the rule used to choose the second-stage test. This conditional requirement is important in an adaptive design: marginal validity of \(p_2\) alone does not guarantee validity after selection.

If the stagewise p-values were independent continuous uniform variables, a standard reference distribution for \(C\) could give a level-\(\alpha\) test. With independent super-uniform p-values, that same reference calibration is conservative for a combination statistic whose rejection region is lower in each p-value: the actual probability of entering the rejection region is no greater than under independent uniforms. Conditional validity gives the corresponding sequential result when the second-stage procedure depends on stage-one information.

Key result: If stage one is super-uniform under \(H_0\), stage two is super-uniform conditional on the stage-one information and adaptation rule, and rejection is monotone in the direction of smaller p-values, then calibrating the combination by its independent-uniform reference distribution controls Type I error at no more than the nominal level. The calibration can be conservative when the stagewise p-values have atoms.

For the inverse-normal combination, the continuous-uniform reference uses \(Z_i=\Phi^{-1}(1-p_i)\) and \(C=w_1Z_1+w_2Z_2\), with nonnegative weights satisfying \(w_1^2+w_2^2=1\). For independent uniform p-values, \(C\sim N(0,1)\). A super-uniform p-value puts no more probability in a small-p-value tail than a uniform p-value does, so its transformed \(Z_i\) is stochastically no larger than a standard normal variable. The resulting normal critical value is therefore conservative under the stated assumptions. Discrete p-values equal to 1 require care because the inverse-normal transform is negative infinity; an implementation that caps or otherwise modifies such values must calibrate the modified procedure, not assume the original reference remains exact.

A Short Justification of the Conservative Bound

The argument can be seen by considering rejection regions in the two p-values. For a fixed \(p_1\), let \(r(p_1)\) be the largest stage-two p-value that still leads to rejection. Because the combination is stronger when either p-value is smaller, rejection occurs when \(p_2\leq r(p_1)\). Conditional super-uniformity gives

P(\text{reject}\mid \text{stage one}) \leq r(p_1),

where the right side is the conditional rejection probability if \(p_2\) were uniform. Averaging over stage one gives a bound using that same uniform-stage-two comparison. The resulting conditional rejection probability is a nonincreasing function of \(p_1\): a smaller stage-one p-value cannot make rejection less likely. Since a super-uniform \(p_1\) places no more probability on small values than a uniform variable does, averaging this function under the actual \(p_1\) distribution cannot increase the rejection probability beyond the independent-uniform benchmark. Thus a level-\(\alpha\) reference rule has actual Type I error at most \(\alpha\). Atoms can make the inequality strict.

Worked Examples: Discrete P-Values and Calibration

Worked Example: A Tied Binomial P-Value

Suppose, hypothetically, that a stage-one outcome is \(X\sim\operatorname{Binomial}(4,0.5)\) under a simple null hypothesis, and the alternative is that the success probability is larger than 0.5. Use the inclusive upper-tail p-value \(p(X)=P_{H_0}(X\geq X_{\mathrm{obs}})\). The binomial model and null probability are assumed correct for this illustration.

The possible p-values are \(p(4)=1/16=0.0625\), \(p(3)=5/16=0.3125\), \(p(2)=11/16=0.6875\), \(p(1)=15/16=0.9375\), and \(p(0)=1\). Their respective probabilities under the null are \(1/16,4/16,6/16,4/16,\) and \(1/16\). Thus, for example,

P_{H_0}(p(X)\leq 0.3125) =P_{H_0}(X\geq 3) =\frac{1+4}{16} =0.3125.

At cutoffs between the listed p-values, the probability \(P_{H_0}(p(X)\leq u)\) stays constant, so it never exceeds \(u\). The p-value is super-uniform, but not continuously uniform. At a nominal stagewise level of 0.10, rejection occurs only when \(X=4\), with actual Type I error \(1/16=0.0625\), not 0.10. The gap is a consequence of the discrete rejection boundary, not evidence that the p-value is invalid.

Worked Example: Fisher Combination With Discrete Inputs

Now suppose two hypothetical independent stages each use the binomial test in the preceding example, under the same null model. Independence here means the stagewise data are independent under \(H_0\). Use Fisher’s statistic \(S=-2\{\ln(p_1)+\ln(p_2)\}\), which is large when the p-values are small.

Each stagewise p-value equals 0.25 with probability 0.25, 0.75 with probability 0.50, and 1 with probability 0.25 if instead each stage has \(X\sim\operatorname{Binomial}(2,0.5)\). These values follow from the inclusive upper tails: \(P(X\geq2)=0.25\), \(P(X\geq1)=0.75\), and \(P(X\geq0)=1\). Enumerating the independent pairs gives the following possible values of \(S\):

\begin{aligned} S(0.25,0.25)&=-2\ln(0.0625)=5.545,\\ S(0.25,0.75)&=-2\ln(0.1875)=3.348,\\ S(0.25,1)&=-2\ln(0.25)=2.773,\\ S(0.75,0.75)&=-2\ln(0.5625)=1.151,\\ S(0.75,1)&=-2\ln(0.75)=0.5754,\\ S(1,1)&=0. \end{aligned}

The corresponding probabilities are 0.0625, 0.25, 0.125, 0.25, 0.25, and 0.0625, which sum to 1. Under independent continuous uniform p-values, Fisher’s statistic has a chi-square reference distribution with 4 degrees of freedom. Its 0.05 upper-tail critical value is about 9.488, above every value this discrete statistic can attain. The reference-calibrated test therefore has actual Type I error 0 in this example, at most 0.05 but quite conservative.

Exact enumeration also shows why a nonrandomized exact 0.05 test cannot reject at its largest attainable value: \(P_{H_0}(S\geq5.545)=0.0625\), which exceeds 0.05. A threshold strictly above 5.545 gives Type I error 0. If exact size 0.05 is required, one could reject at \(S=5.545\) with probability \(0.05/0.0625=0.8\), using an independent randomization. Such randomization is mathematically valid but is often not desirable operationally.

Worked Example: A Conservative Rule After Test Selection

Suppose, hypothetically, the stage-one p-value has null distribution 0.02 with probability 0.01, 0.20 with probability 0.19, and 1 with probability 0.80. This is super-uniform: for example, \(P(p_1\leq0.02)=0.01\leq0.02\), and \(P(p_1\leq0.20)=0.20\). Based on stage-one information, the design selects one of two stage-two tests. Assume each possible test has a p-value satisfying \(P(p_2\leq u\mid\text{stage-one information and selected test})\leq u\) under every null value.

Use the Bonferroni rule that rejects if \(p_1\leq0.025\) or \(p_2\leq0.025\). The first event has probability 0.01. Conditional validity of whichever stage-two test is selected gives \(P(p_2\leq0.025)\leq0.025\), including after selection. Therefore

P_{H_0}(\text{reject}) \leq P_{H_0}(p_1\leq0.025)+P_{H_0}(p_2\leq0.025) \leq 0.01+0.025=0.035.

This rule controls Type I error below 0.05, though it may be conservative. The crucial check is conditional validity of the selected stage-two test; knowing only that each test is valid before selection would not establish the required bound.

Practical Calibration Choices and Failure Modes

A continuous reference distribution is often convenient and, with valid stagewise p-values and the required dependence conditions, gives a conservative test. When the attainable p-values are sparse, however, its critical boundary can be far beyond the values the combination statistic can reach. Exact calibration can be less conservative: enumerate the null distribution of the combined statistic, or compute it from the joint stagewise null distribution, then choose a rejection region whose null probability is at most \(\alpha\). For an upper-tail statistic, select a cutoff \(c\) satisfying \(P_{H_0}(C\geq c)\leq\alpha\). Because the distribution has atoms, a nonrandomized cutoff may not attain \(\alpha\) exactly.

Randomized p-values offer another option for a discrete statistic \(T\). If \(V\) is independent Uniform(0,1), then

p_{\mathrm{rand}}=P_{H_0}(T>T_{\mathrm{obs}}) +V\,P_{H_0}(T=T_{\mathrm{obs}})

is uniform under a correctly specified simple null. It spreads the probability mass from tied outcomes across the corresponding interval of tail probabilities. This can make a continuous-uniform combination reference exact, subject to the required stage-independence or conditional-validity assumptions. It also makes decisions depend on an added random draw, so its practical acceptability must be considered.

A mid-p value, which allocates half the probability of a tie to the tail, is not generally super-uniform. It may improve average conservatism, but it can make \(P_{H_0}(p\leq u)>u\) at some cutoffs. Consequently, feeding mid-p values into a combination rule calibrated for valid p-values can inflate Type I error. Mid-p methods require a calibration argument for the complete procedure; they are not automatically protected by the standard continuous reference.

Check the tail convention. Inclusive and exclusive tails differ at tied outcomes. The p-value definition used in the analysis and its calibration must agree.
Check conditional validity. If stage two is selected or modified using stage-one information, verify its p-value is valid conditional on that information under the null.
Check the joint null structure. Marginal super-uniformity alone does not establish an independent-uniform reference distribution when stagewise p-values are dependent.
Check attainable values. If the continuous reference critical value cannot be reached by the statistic, the resulting test may have much lower size than intended.

The main alternatives answer different needs. Continuous-reference calibration is simple and often safely conservative. Exact enumeration targets the actual discrete null distribution but depends on correctly specifying that distribution and the adaptation rule. Randomization can attain exact size at an atom but introduces a random decision. Mid-p calibration may be less conservative but needs separate error control. None removes the need to justify the stagewise null model and the dependence or conditional-validity assumptions.

Common mistake: “The stagewise p-values are valid, so the combined statistic has exactly its continuous-uniform reference distribution.” This is incorrect for discrete p-values: validity gives super-uniformity, not necessarily uniformity, and ties create atoms. Use the continuous reference as a conservative bound only when the combination is monotone and the independence or conditional-validity conditions hold; use the actual null distribution if exact discrete calibration is needed.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In one sentence, distinguish a uniform p-value from a super-uniform p-value.
Why can a chi-square reference calibration for Fisher’s statistic be conservative when stagewise tests are discrete?
For a discrete upper-tail test, why is \(P(T\geq t_{\mathrm{obs}})\) generally safer than \(P(T>t_{\mathrm{obs}})\) as a p-value?
What additional condition is needed for a selected stage-two p-value to support conditional Type I error control?
Why does a mid-p value not automatically justify a standard combination-test reference distribution?

Solutions. (1) A uniform p-value has \(P(p\leq u)=u\); a super-uniform p-value only guarantees \(P(p\leq u)\leq u\). (2) Discrete valid p-values have atoms and are generally larger in their lower tail than uniform p-values, so a monotone combination can exceed its rejection threshold less often. (3) Including equality counts the probability of all outcomes tied with the observed statistic, preserving the super-uniform bound. (4) The stage-two p-value must be valid conditional on the stage-one information and the selected test under the null. (5) Mid-p values can fail super-uniformity, so the usual validity argument for the combination reference may fail.