What a P-Value Must Be Calibrated Against
In an adaptive trial, assignments depend on information accumulated during the trial. The resulting data therefore reflect both patient outcomes and the rule that used earlier information to assign later patients. A p-value can be calculated correctly from a formula and still be poorly calibrated if that formula’s reference distribution does not match this data-generating process.
The key question is not simply whether a test statistic is familiar, or whether its standard error can be computed. Ask instead: under the null hypothesis, and under the actual allocation process, how often would this procedure produce a p-value at least as small as the one observed? Calibration concerns that repeated-trial frequency. For a test at significance level \(\alpha\), a calibrated p-value should satisfy \(P(p\leq\alpha\mid H_0)\leq\alpha\) under the stated null and sampling conditions. Equality is typical for a continuous exact test; discrete tests may be conservative.
This does not mean that a p-value is the probability that the null hypothesis is true, or the probability that the result arose “by chance.” It is a probability, computed assuming the null and specified design, of obtaining a test statistic at least as extreme as the observed one. The exact meaning depends on what outcomes and randomization the reference distribution treats as variable.
Three Components to Check
For a reported p-value, identify three components before interpreting it. First, specify the null hypothesis and the statistic. Second, identify the variance estimate used to standardize that statistic. Third, determine the reference distribution and whether it represents the adaptive assignment process. A mismatch in any component can invalidate the nominal p-value.
For example, a sharp null says every patient would have the same outcome under either treatment. A weaker null might say the average treatment effect is zero. These are not interchangeable: a test exact for a sharp null need not be exact for a weaker average-effect null.
Determine which observations enter the statistic, whether sample sizes are fixed or random, and what sources of variability the standard error represents. A variance formula derived for fixed independent group sizes does not become valid for adaptive data merely because it can be evaluated from the observed groups.
Check whether the claimed normal, \(t\), or randomization distribution accounts for the sequential assignment probabilities and the dependence those probabilities create between observed outcomes and later assignments.
Earlier material on Type I error under adaptive allocation motivates this check: calibration is what makes the probability of a false rejection no greater than the chosen significance level under the specified null. The details matter because an allocation rule can make the number assigned to each arm random, and can link that number to earlier observed outcomes.
A Conventional Test: Calculation Is Not Calibration
Consider a two-arm comparison of continuous outcomes. A commonly reported statistic is the difference in observed arm means divided by a Welch standard error. For independent observations within each arm, the estimated standard error is \(SE=\sqrt{s_1^2/n_1+s_0^2/n_0}\). The usual Welch reference distribution also estimates degrees of freedom from the two sample variances and sample sizes. Those calculations describe a conventional model-based test; whether its p-value is calibrated in an adaptive trial is an additional question.
Worked Example: Checking a Reported Welch P-Value
Suppose a hypothetical adaptive trial reports 18 participants on treatment 1 and 42 on treatment 0. The observed means are 12.3 and 10.0 outcome units, and the sample standard deviations are 4.0 and 3.5 outcome units. The report tests \(H_0:\mu_1-\mu_0=0\) against a two-sided alternative using a Welch test. Assume for this calculation that observations are independent within arms and the reported summaries are accurate.
The estimated difference is \(12.3-10.0=2.3\) outcome units. The reported Welch standard error is
Using the unrounded standard error, the statistic is
Welch’s degrees-of-freedom estimate is
Against a \(t\) distribution with about 28.7 degrees of freedom, the two-sided p-value is approximately 0.043. This is the calculation under the Welch reference model. It is not, by itself, evidence that the p-value is calibrated for the trial. If the arm sizes arose through response-adaptive assignment, the ordinary Welch reference distribution may fail to represent how the statistic varies under the actual design. The report would need to justify that reference distribution or use a method whose calibration accounts for the allocation rule.
For this test to have its usual model-based justification, the outcomes must satisfy the assumptions underlying its sampling distribution, including suitable independence and variance estimation, and the reference distribution must adequately approximate the statistic’s null distribution. Adaptive assignment can challenge that last condition even if the arithmetic and the individual observations are otherwise correct. Random sample sizes, outcome-dependent allocation, small arm sizes, or very unequal variances are reasons to investigate further, not automatic proof that a particular p-value is wrong.
What “Accounting for the Design” Means
One design-based approach is a randomization test. Its reference distribution is generated by the trial’s assignment mechanism under a null hypothesis that makes the relevant outcomes known across assignments. Under a sharp null of no treatment effect for any participant, each participant’s outcome would be unchanged by assignment. The test can then repeatedly generate assignment sequences using the actual adaptive rule, retaining its dependence on the observed history, and recompute the chosen statistic for each sequence. A two-sided randomization p-value is the proportion of those sequences whose statistic is at least as extreme as the observed statistic; with a finite set of possible assignments, a conservative adjustment can include the observed assignment in the numerator and denominator.
The method is exact for its stated sharp null when the randomization mechanism is correctly represented and the test statistic is defined for the generated assignments. It is not automatically an exact test of a weaker null such as zero average treatment effect. Nor is it valid to shuffle treatment labels as though assignments were independent and equally likely if the real design used unequal, history-dependent probabilities.
A different route uses a model-based or estimating-equation statistic with variance estimation that allows for the design. Inverse-probability weighting, for instance, uses known assignment probabilities to account for unequal chances of assignment. Such methods still require their own conditions: probabilities must be correctly recorded, relevant assignment probabilities must not be zero, the variance estimate must be appropriate, and any asymptotic reference distribution must be adequate for the sample size and design. A design-aware standard error is useful, but a standard normal or \(t\) approximation still needs justification.
| Approach | Reference distribution relies on | Key calibration check |
|---|---|---|
| Conventional model-based test | A stated sampling model and its assumptions | Does that model represent the random sample sizes and adaptive assignment? |
| Randomization test | The assignment mechanism under a specified null | Does the test reproduce the actual sequential rule, and is the null sharp enough? |
| Design-aware asymptotic test | A suitable large-sample approximation and consistent variance estimate | Are assignment probabilities and variance calculations appropriate, with enough information for the approximation? |
Null Simulation as a Calibration Diagnostic
When a reference distribution is difficult to establish analytically, repeated simulation under a specified null can reveal whether a proposed procedure has the advertised error rate. Each replicate must reproduce the full design: the enrollment process, the adaptive assignment rule, the outcome model under the null, the analysis statistic, the variance estimate, and the rule used to turn that statistic into a p-value. Holding the final arm sizes fixed when the actual design makes them random can miss the very feature that affects calibration.
Worked Example: Interpreting a Null Calibration Check
A team simulates 1,000 hypothetical trials under its stated null model. In every replicate it runs the planned adaptive allocation algorithm and calculates the planned p-value. It finds that 72 p-values are at most 0.05.
The estimated false-positive rate is \(72/1000=0.072\), or 7.2%, rather than the nominal 5%. The approximate Monte Carlo standard error is
Thus the simulation suggests potential inflation, although Monte Carlo uncertainty remains: the difference from 0.05 is about \(0.022\), or roughly 2.7 of these approximate standard errors. The team should check implementation and consider more replicates before drawing a firm conclusion. The finding is specific to the simulated null model and design; it does not establish performance for other outcome distributions or null hypotheses. A useful diagnostic is also to examine the full empirical distribution of p-values: excess small values indicate a different pattern from a roughly uniform distribution under a continuous exact test.
Simulation is a diagnostic, not a substitute for specifying the estimand, null, and operating conditions. A procedure can appear well calibrated under one outcome model and fail under another. Simulation also cannot repair an incorrect implementation of the allocation algorithm or analysis.
Conditional and Unconditional Questions
A p-value may be assessed conditionally on some aspect of the observed design, or unconditionally over all random elements of repeated trials. For example, unconditional calibration averages over both the random assignments and outcomes that would arise if the complete adaptive trial were repeated. A conditional claim fixes specified observed information and asks for calibration within that restricted set of trials. These are different claims, and neither follows merely from observing a particular sequence of assignments.
When adaptive allocation depends on accumulated responses, conditioning on the final allocation counts can be especially consequential: those counts may themselves contain information about earlier outcomes. A test that treats them as fixed without justification may use the wrong conditional distribution. A report should say what was conditioned on, what was simulated or modeled as random, and why the resulting reference distribution is appropriate.
Common Pitfalls and Limits
- “The formula is standard, so the p-value is valid.” A familiar \(t\) statistic does not guarantee that its reference distribution applies after adaptive assignment. Check the design assumptions as well as the arithmetic.
- “A large estimated standard error guarantees calibration.” The standard error estimates variability under a model; it does not establish that the statistic follows the reference distribution used to calculate the p-value.
- “A randomization test can test any null.” Re-randomizing assignments is exact only for a null that supplies the required outcomes across assignments, such as the sharp no-effect null. A weaker null needs a separate justification.
- “The p-value is the chance the null is true.” It is calculated under the null and describes the tail probability of the statistic under the specified reference process; it is not a posterior probability of the hypothesis.
- “One simulation settles calibration.” A finite number of replicates has Monte Carlo error, and simulated calibration only applies to the null scenarios and design actually represented.
No single test is universally best for all adaptive trials. Model-based tests can be efficient when their assumptions are credible; randomization tests can directly reflect a known assignment mechanism for an appropriate sharp null; and design-aware asymptotic tests can be useful when their conditions and approximations are defensible. The limitation shared by all approaches is that their p-values answer only the question encoded by their null, variance calculation, and reference distribution.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A report gives a Welch p-value from an adaptive trial. What information is needed before deciding whether that p-value is calibrated?
- Under a sharp null, why must a randomization test reproduce the adaptive assignment algorithm rather than freely shuffle treatment labels?
- In 2,000 null simulations, 118 p-values are at most 0.05. What is the estimated false-positive rate, and what does it suggest?
- Does an exact randomization test of a sharp no-effect null automatically provide an exact test of a zero average-effect null? Explain.
Solutions. (1) Identify the null, statistic, standard error and its assumptions, reference distribution, and the assignment process, including whether the analysis conditions on any design features. (2) Because the actual assignment probabilities depend on trial history; shuffling as if assignments were equally likely changes the null distribution. (3) The estimate is \(118/2000=0.059\), or 5.9%, suggesting possible inflation relative to 5%; simulation uncertainty and implementation should be checked. (4) No. The sharp null specifies each participant’s outcome under each assignment sufficiently for the randomization distribution, whereas a zero average effect is weaker and does not determine those individual counterfactual outcomes.