Tutorials › Adaptive Clinical Trial Design › Type I Error With Nuisance-Parameter Re-Estimation

Home  /  Tutorials  /  Sample Size Re-Estimation

Sample Size Re-Estimation · Tutorial 319 of 799

Type I Error With Nuisance-Parameter Re-Estimation

Learn how conditional calibration, independence, and appropriate null tests determine whether a blinded nuisance-parameter update preserves type I error.

Advanced 12 min read

What You'll Learn

  • State a conditional-size condition that guarantees unconditional type I error control after sample-size adaptation.
  • Explain why blinding the re-estimation does not, by itself, prove type I error preservation.
  • Work a normal-outcome example using an independent variance pilot and a conditional t test.
  • Work a binary-outcome example using a blinded pooled-rate estimate and a conditional exact test.
  • Identify design and analysis assumptions that can break the preservation argument.
  • Distinguish conditional type I error control from an unconditional average over nuisance parameters.

Why Re-Estimating a Nuisance Parameter Raises a Testing Question

A sample-size rule may use an interim estimate of a nuisance parameter, such as the outcome variance or a control-group event rate, without using an estimated treatment effect. This can be attractive: the design responds to uncertainty about how much information the trial will provide, while the treatment comparison remains masked. But the final test is now applied after a data-dependent choice of sample size. Its type I error therefore needs justification under the adaptive design, not just under the original fixed design.

The key distinction is between what determines the sample size and what determines rejection under the null. A blinded update may avoid direct use of the observed treatment contrast, but that fact alone does not establish that the final test has its nominal null rejection probability. A clean guarantee follows when the final test has the right size conditional on the information used for adaptation.

Definition: The type I error of a test is its probability of rejecting \(H_0\) when \(H_0\) is true. An adaptive design preserves a nominal level \(\alpha\) if, for every parameter value under \(H_0\), the probability of rejection under the complete adaptation-and-testing procedure is at most \(\alpha\). Equality gives exact level \(\alpha\); a probability below \(\alpha\) is conservative control.

A Conditional Criterion for Error Control

Let \(\theta\) denote a nuisance parameter, such as a common outcome variance or event probability, under the null hypothesis. Let \(U\) be all the information used to choose the final sample size \(N\). This can include an interim nuisance-parameter estimate and any prespecified design variables. Write \(R\) for the event that the final test rejects \(H_0\). A sufficient condition for type I error control is that the test remains level \(\alpha\) after conditioning on the adaptation information:

$$ P_{\theta}(R\mid U)\leq \alpha \quad\text{for every possible }U\text{ and every }\theta\text{ under }H_0. $$

The probability here is over the remaining randomness in the trial and test, conditional on what the adaptation rule observed. The condition is stronger than an unconditional statement that averages over possible \(U\). It prevents the adaptation from selecting circumstances in which the final test is systematically too liberal.

Theorem: Suppose the adaptation rule is prespecified, and under every null parameter value \(\theta\), the final test satisfies \(P_{\theta}(R\mid U)\leq\alpha\) for every possible adaptation outcome \(U\). Then the adaptive procedure has type I error at most \(\alpha\): \(P_{\theta}(R)\leq\alpha\) for every \(\theta\) under \(H_0\).

The justification is the law of total probability. Conditioning on \(U\) partitions the possible trial outcomes into adaptation outcomes; the unconditional rejection probability is the average of the corresponding conditional rejection probabilities.

$$ \begin{aligned} P_{\theta}(R) &=E_{\theta}\!\left[P_{\theta}(R\mid U)\right]\\ &\leq E_{\theta}[\alpha]\\ &=\alpha. \end{aligned} $$

This argument does not require \(U\) to be independent of every trial outcome. It requires the conditional test to have the stated size given \(U\). Independence is one way to obtain that property, but not the only way. A test can also be valid conditionally because its null distribution is correctly calibrated after conditioning on an appropriate statistic.

Worked Example: An Independent Variance Pilot and a Normal-Outcome Test

Worked Example: An Independent Variance Pilot and a Normal-Outcome Test

Suppose a two-arm trial compares normally distributed outcomes with a common, unknown variance \(\sigma^2\). Before the main trial, an independent blinded pilot provides a pooled variance estimate \(s_P^2=100\) outcome-units squared. The design targets a difference \(\delta=5\) outcome units, two-sided \(\alpha=0.05\), and power \(0.80\). It uses the usual normal-approximation planning formula, with equal allocation:

$$ n=\frac{2(z_{1-\alpha/2}+z_{1-\beta})^2s_P^2}{\delta^2}. $$

Here \(z_{0.975}=1.960\), \(z_{0.80}=0.8416\), and \(\beta=0.20\). Substitution gives \(n=2(1.960+0.8416)^2(100)/5^2 =2(7.849)(100)/25=62.79\). Rounding up gives 63 participants per arm. The pilot data are used for planning and are not included in the final efficacy comparison.

For the main trial, assume independent normal outcomes within and between arms, equal variances in the two arms, and complete outcome measurement. Use the usual pooled-variance two-sample \(t\) test, with \(2n-2=124\) degrees of freedom, to test equality of the population means. Conditional on the pilot data, the main-trial sample size is fixed at 63 per arm, and the main-trial observations are independent of the pilot.

Under the null hypothesis of equal means, the pooled-variance \(t\) statistic has a \(t\) distribution with 124 degrees of freedom, conditional on the pilot. Thus a two-sided test using the corresponding \(t\) critical value rejects with probability \(0.05\) for every pilot outcome that selects this sample size. Averaging over pilot outcomes leaves the rejection probability at \(0.05\). The unknown variance may affect which sample size the pilot selects, but it does not disturb the conditional calibration of this final test.

This example’s guarantee depends on the design and analysis matching the assumptions. It is not a claim that any variance re-estimation followed by a \(t\) test is exact. In particular, the independence of pilot and efficacy data, and the validity of the \(t\) reference distribution for the main-trial analysis, are essential to this argument.

Worked Example: A Blinded Pooled Rate and a Conditional Binary Test

Worked Example: A Blinded Pooled Rate and a Conditional Binary Test

Suppose an independent pilot observes 18 events among 60 participants, pooled across the treatment arms. The blinded estimate of the common event probability for planning is \(\hat{p}=18/60=0.300\). For illustration, plan for a difference of \(\delta=0.15\), two-sided \(\alpha=0.05\), and power \(0.80\). The usual approximate equal-allocation formula is

$$ n\approx \frac{2\hat{p}(1-\hat{p})(z_{1-\alpha/2}+z_{1-\beta})^2}{\delta^2}. $$

Substituting gives \(n\approx 2(0.300)(0.700)(1.960+0.8416)^2/(0.15)^2 =3.297/0.0225=146.5\). Rounding up gives 147 participants per arm. This approximate formula is being used to choose a sample size; it is not the final test.

Assume the pilot is independent of the efficacy data and contributes only to the sample-size decision. For the final comparison, use a two-sided Fisher exact test of equal event probabilities, conditional on the total number of events in the main trial. Under the null of a common event probability in both randomized arms, conditional on the total number of events and the arm sizes, the allocation of those events between arms has a hypergeometric distribution. That conditional distribution does not depend on the unknown common event probability.

Consequently, a nonrandomized Fisher test calibrated to have conditional rejection probability no greater than \(0.05\) has conditional type I error at most \(0.05\), whatever sample size the pilot selected. Averaging over pilot outcomes preserves that bound. If a randomized boundary rule is used to attain exactly \(0.05\) for each conditional configuration, the unconditional level is exactly \(0.05\); the usual nonrandomized exact test may be conservative because its attainable probabilities are discrete.

The conclusion relies on the conditional null distribution being correct, the randomization and event recording matching the stated model, and the pilot being independent of the main-trial test data. If the final test instead uses an uncalibrated large-sample approximation, this exact conditional argument does not automatically apply.

What Blinding Does—and Does Not—Establish

Blinding can limit access to treatment-specific information and make a re-estimation rule less likely to respond directly to an apparent treatment difference. That is an important design protection, but it is not itself a mathematical condition for type I error control. The proof above uses conditional validity of the final test. A label-blind statistic can still be associated with the eventual test statistic, and conditioning on it can change the null distribution used by a fixed-design test.

In particular, distinguish these claims:

  • “The rule is blinded.” The person or algorithm making the update does not use treatment-labelled results in the specified way.
  • “The update depends only on a nuisance parameter.” The rule is designed to use information about a nuisance feature, not an estimated treatment effect. This description needs a precise mathematical definition in the design.
  • “The final test is conditionally calibrated.” Given the information that drove the update, its null rejection probability is at most \(\alpha\). This is the property that directly supports the conditional argument.

The first two statements do not imply the third without further assumptions. If pilot observations are reused in the final statistic, for example, the sample-size decision and the test statistic may share random variation. The result can also fail if the nuisance estimate is affected by treatment differences under the null model being considered, if the conditional null distribution is misidentified, or if unplanned efficacy choices are made along the way.

Worked Example: Checking a Two-Size Rule by Conditioning

Imagine a prespecified rule that chooses either \(N=100\) or \(N=140\), based on blinded information \(U\). Suppose the final test has been shown to satisfy \(P_{\theta}(R\mid U=u)\leq0.05\) for every possible \(u\) leading to either size, for a particular null value \(\theta\). If the rule selects \(N=100\) with probability 0.70 and \(N=140\) with probability 0.30, and the corresponding conditional rejection probabilities are 0.048 and 0.050, then

$$ P_{\theta}(R) =0.70(0.048)+0.30(0.050) =0.0486. $$

The overall type I error is \(0.0486\), below the nominal \(0.05\). If both conditional rejection probabilities were exactly \(0.05\), the overall value would be \(0.70(0.05)+0.30(0.05)=0.05\). The result is not obtained merely because the rule is blinded: it follows because the test is calibrated within each adaptation outcome.

Assumptions, Diagnostics, and Alternatives

Before invoking conditional preservation, specify the null model, the adaptation information \(U\), the set of possible sample sizes, and the final rejection rule. Verify that the conditional-size claim holds for all nuisance-parameter values in the null hypothesis, not just at a convenient planning value. Other assumptions to examine include:

  • The adaptation rule and any caps, floors, or rounding conventions are prespecified and implemented as stated.
  • Randomization, outcome distributions, independence, and missing-data handling support the conditional null distribution used by the final test.
  • The final test’s significance level accounts for all efficacy opportunities. An additional efficacy look or an unplanned analysis is not automatically covered by a one-test argument.
  • If the pilot and final efficacy data overlap, the conditional distribution is derived for that joint design rather than borrowed from an independent-data setting.
  • For a composite null, the size bound holds across the null parameter space; a calculation at one nuisance value does not prove uniform control.

Simulation can help diagnose a proposed design but does not replace the argument. Simulate under several null nuisance values, record the selected sample size, and estimate rejection rates both overall and within adaptation outcomes. Large discrepancies between conditional rates can expose a failure of the assumed calibration. Monte Carlo estimates are themselves uncertain, so report their simulation uncertainty and do not interpret a small number of simulated rejections as proof of control.

A fixed-sample design avoids the sample-size-selection issue, though it may be inefficient if planning assumptions are poor. A conditional argument can justify an adaptive rule when the final test remains valid after conditioning on the update information. Other designs may require a test specifically calibrated for the full adaptation procedure. These are not interchangeable justifications: the appropriate approach depends on data reuse, the adaptation rule, and the null distribution of the final test.

Common Mistake

Common mistake: “The sample-size update is blinded, so the type I error must remain at \(\alpha\).” Blinding alone does not show that the final test is level \(\alpha\) conditional on the update. The correction is to define the adaptation information and establish the conditional null rejection bound—or calibrate the test for the complete adaptive procedure. If that property is not established, nominal type I error preservation cannot be concluded from blinding by itself.

Check Your Understanding

Answer each question, then compare with the brief solutions.

  1. If \(P_{\theta}(R\mid U)\leq\alpha\) for every adaptation outcome under a null value \(\theta\), what can be concluded about \(P_{\theta}(R)\)?
  2. A blinded pilot chooses between two sample sizes. What additional property of the final test is needed for the conditional argument?
  3. In the normal-outcome example, why is independence between the pilot and the main trial relevant?
  4. For a Fisher exact test, why can conditioning on the total number of events remove dependence on the common null event probability?
  5. Does observing a simulated overall rejection rate near \(0.05\) prove that a design controls type I error for every null nuisance value? Explain.

Solutions. (1) By the law of total probability, \(P_{\theta}(R)=E_{\theta}[P_{\theta}(R\mid U)]\leq\alpha\). (2) The final test must have conditional null rejection probability at most \(\alpha\) given the information used to select the sample size. (3) Independence lets the pilot determine the sample size without changing the conditional null distribution of the main-trial \(t\) statistic. (4) Under the null, conditional on the total event count and arm sizes, event allocation between randomized arms has a hypergeometric distribution that does not involve the common event probability. (5) No. Simulation only checks the parameter values and scenarios simulated, and its estimated rate has Monte Carlo uncertainty; a general guarantee requires a valid argument over the null parameter space.

Previous

Next

Back to Tutorials
Clinical Biostats