Why a Composite Null Requires More Than One Calibration Point
A trial decision rule can have different false-positive probabilities at different parameter values that all satisfy the null hypothesis. This matters whenever the null describes a range of effects rather than one point. For example, a null hypothesis may state that the response probability is no greater than a clinically acceptable limit. Values well below that limit are still null values, but the chance of declaring a benefit may be much smaller there than at the limit.
The rule being assessed must be fixed before its operating characteristics are evaluated. In a Bayesian trial, that rule might declare efficacy when a posterior probability exceeds a prespecified cutoff. Calibration then asks how often the rule declares efficacy under repeated data generation at each true parameter value. The posterior probability used by the rule and the repeated-sampling probability used to assess it are different quantities.
A pointwise rate, \(a(\theta)\), answers the question “How often would this rule declare efficacy if this particular null value were true?” The supremum answers a different question: “What is the largest such probability anywhere in the null space?” Calibration of a composite null requires keeping these questions distinct.
A Binomial Example Across Null Values
Suppose, hypothetically, that a single-arm trial enrolls \(n=20\) participants and records a binary response. Assume independent outcomes with a common response probability \(p\), so \(X\sim\operatorname{Binomial}(20,p)\), where \(X\) is the number of responses. Let the null be \(H_0:p\leq0.20\). A fixed decision rule declares efficacy if \(X\geq8\). The rule might have arisen from a Bayesian analysis; for this calibration exercise, its action for every possible dataset is already specified.
Worked Example: Comparing Two Null Response Probabilities
Calculate the false-positive probability at \(p=0.20\) and \(p=0.10\), both of which belong to the composite null.
Under the stated binomial model, the probability of exactly \(x\) responses is \({20\choose x}p^x(1-p)^{20-x}\). Since the rule declares efficacy for \(X\geq8\), its false-positive probability is the upper tail:
At the boundary \(p=0.20\), substitution gives
Thus the rule declares efficacy about \(3.214\%\) of the time when the true response probability is \(0.20\). At the null value \(p=0.10\),
That is about \(0.04156\%\), much smaller than the rate at \(p=0.20\). Both parameter values satisfy \(H_0\), but they do not have the same false-positive probability. These calculations describe repeated trials under each fixed true value; they do not assign probabilities to the values of \(p\).
The difference is a feature of the operating characteristic, not a contradiction. As the true response probability falls, observing at least eight responses becomes less likely. In this example, the boundary governs the worst-case null rate, but that conclusion needs a justification rather than an assumption.
When Does the Boundary Govern?
For this binomial example, the false-positive probability is nondecreasing in \(p\). One direct justification uses a coupling argument. Generate independent \(U_i\sim\operatorname{Uniform}(0,1)\), for \(i=1,\ldots,20\), and define a response at probability \(p\) by \(Y_i(p)=I(U_i\leq p)\). For any \(p_1\leq p_2\), each \(Y_i(p_1)\leq Y_i(p_2)\), and therefore \(X(p_1)=\sum_iY_i(p_1)\leq X(p_2)\) in every coupled realization. Consequently, \(I\{X(p_1)\geq8\}\leq I\{X(p_2)\geq8\}\), which implies \(a(p_1)\leq a(p_2)\) after taking expectations.
This argument depends on the model and the rule. It does not establish that a boundary always controls Type I error in a Bayesian trial. With several parameters, a decision may respond differently to changes in a treatment effect, baseline risk, variability, missingness, or accrual. Even a rule that is monotone in one parameter need not be monotone in every nuisance parameter. A boundary claim must therefore be supported by an analytic argument, a valid stochastic ordering, or sufficiently thorough numerical investigation with the remaining uncertainty acknowledged.
Write the parameter values that satisfy the null, including nuisance parameters and any restrictions on them.
Define exactly what counts as a positive trial decision and how trial data are generated at each parameter value.
Use an analytic expression where available or estimate the rate at a prespecified collection of null scenarios.
Use a justified monotonicity result when one applies; otherwise investigate whether the search has missed a larger rate and report the limits of the assessment.
Simulation Across Null Scenarios
When the trial model is too complex for an analytic probability, Monte Carlo simulation estimates \(a(\theta)\) separately at each chosen null scenario. For \(M\) independent simulated trials at a fixed \(\theta\), let \(R_m=1\) if replicate \(m\) declares efficacy and \(R_m=0\) otherwise. The estimate is \(\hat a(\theta)=M^{-1}\sum_{m=1}^{M}R_m\). Because the replicate indicators are Bernoulli with probability \(a(\theta)\), their Monte Carlo standard error is estimated by \(\sqrt{\hat a(\theta)\{1-\hat a(\theta)\}/M}\). This uncertainty reflects simulation noise, not uncertainty about the true parameter in an actual trial.
Worked Example: Interpreting Simulated Rates Over a Null Grid
For the hypothetical binomial rule above, suppose \(M=100{,}000\) independent replicates are run at each of \(p=0.10,\ 0.15,\) and \(0.20\). Assume the respective numbers of positive decisions are 43, 596, and 3,190. These are stipulated hypothetical simulation results, not analytic probabilities.
At \(p=0.10\), the estimated rate is \(43/100{,}000=0.0004300\), or \(0.04300\%\). Its estimated Monte Carlo standard error is
At \(p=0.15\), the estimate is \(596/100{,}000=0.005960\), or \(0.5960\%\), with estimated standard error
At \(p=0.20\), the estimate is \(3{,}190/100{,}000=0.03190\), or \(3.190\%\), with estimated standard error
The estimated rates increase across these scenarios, and the boundary estimate is the largest of the three. The exact binomial calculation and the coupling argument establish that the boundary is the true maximum for this particular rule and model. In a more complicated setting, a rising pattern on a finite grid would be evidence to investigate, not proof that the maximum occurs at the boundary. Small rates also require care: a few extra or fewer simulated events can noticeably change a rate near zero.
Summarizing, Diagnosing, and Comparing Approaches
There are several distinct ways to summarize false-positive behavior, and they answer different questions.
| Summary | What it reports | Main limitation |
|---|---|---|
| Pointwise rate | \(a(\theta)\) at one specified null value | Does not describe other null values |
| Worst-case rate | \(\sup_{\theta\in\Theta_0}a(\theta)\) | Can be difficult to locate or estimate in a high-dimensional space |
| Prior-averaged rate | An average of \(a(\theta)\) under a chosen distribution on null parameters | Depends on that distribution and does not bound the worst case |
A prior-averaged rate can be useful if the distribution represents a deliberately chosen scenario-weighting question. It is not a substitute for a worst-case Type I error criterion when the goal is to control false positives uniformly over the null. Likewise, reporting only the boundary rate is justified only when that boundary is known to maximize the rate, or when the calibration objective explicitly concerns that point alone.
Diagnostics should target both the simulation and the parameter-space search. Confirm that each scenario lies in the stated null, that the simulated data match the intended model, and that the decision rule is applied identically in every replicate. Compare independent simulation batches or use Monte Carlo standard errors to distinguish apparent variation from simulation noise. Inspect rates across the parameter space, including plausible interior and nuisance-parameter combinations. A coarse grid can miss a narrow interior maximum; simulation uncertainty can obscure differences between nearby rates; and using too few replicates can make a small false-positive rate appear to be zero.
The main alternative to simulation is analytic evaluation. It is preferable when the probability of a positive decision can be calculated and monotonicity can be proved, as in the binomial example. Analytic results may be unavailable for realistic adaptive designs or complex data-generating mechanisms. Simulation can accommodate those mechanisms, but the answer is then conditional on the scenarios examined and carries Monte Carlo uncertainty. Neither approach makes a Bayesian posterior probability itself a repeated-sampling false-positive rate: that rate is defined by the probability of the rule’s action under repeated data generation at specified true parameter values.
Common Mistakes and Limitations
- Checking only one null point. A low false-positive rate at an interior value does not establish control over the whole null space. Evaluate the relevant range or justify why one point is worst.
- Assuming every boundary is worst. The binomial coupling proof applies to the stated model and upper-tail rule. It cannot be transferred automatically to a different rule or a multidimensional parameter space.
- Treating a grid maximum as the true supremum. A grid reports values only at its points. Refinement and analytic or structural justification may be needed to rule out a larger rate between them.
- Confusing a posterior probability with a false-positive rate. A posterior probability conditions on observed data under a Bayesian model. The false-positive rate is a repeated-trial probability under a specified true null parameter.
- Ignoring simulation error. A simulated rate is an estimate, not the exact operating characteristic. Report its simulation precision, particularly when rates are small or scenario differences are modest.
The key discipline is to state the null parameter space, evaluate the fixed rule across it, and distinguish what has been demonstrated from what has only been suggested by a finite set of simulations.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- For \(H_0:p\leq0.20\) and the rule \(X\geq8\) in the \(n=20\) binomial example, why is the rate at \(p=0.20\) the worst-case rate?
- Does observing an estimated false-positive rate of \(0.8\%\) at one interior null value establish that the worst-case rate is below \(1\%\)?
- With 10,000 simulated trials and 80 positive decisions at a scenario, calculate the estimated rate and its approximate Monte Carlo standard error.
- Why can a prior-averaged false-positive rate be below a target even when the worst-case rate exceeds that target?
Solutions. (1) The coupling makes \(X(p)\) nondecreasing in \(p\), and the indicator \(I(X\geq8)\) is nondecreasing in \(X\), so \(a(p)\) is nondecreasing and is maximized at \(0.20\). (2) No. Other null values may have larger rates; a single point cannot establish a supremum. (3) The estimate is \(80/10{,}000=0.0080\), or \(0.800\%\). The standard error is approximately \(\sqrt{0.0080(0.9920)/10{,}000}=0.000891\), or \(0.0891\) percentage points. (4) An average can be pulled down by parameter values with small rates even if some other null value has a rate above the target; the average does not bound the maximum.