Studentized Statistics in Conditional-Error Calculations

Why Studentization Changes the Calculation

When a variance is estimated, a test statistic such as a sample mean divided by its estimated standard error does not generally have a standard normal distribution in a finite sample. It has a Student t distribution under normal sampling. That distinction matters in an adaptive design: the conditional-error calculation is only as accurate as the stagewise p-values and their conditional distributions.

A useful way to handle estimated variances is to calculate each stage's p-value from its exact t distribution and then, when using an inverse-normal combination test, transform that p-value to a normal score. This gives an exact normal score at the null boundary without pretending the original t statistic is normal. The argument requires stagewise independence and a valid stagewise test at every sample size the adaptation may select.

Definition: Let stage $i$ have sample size $n_i$, sample mean $\bar{x}_i$, sample standard deviation $s_i$, and degrees of freedom $\nu_i=n_i-1$. For testing $H_0:\mu\leq\mu_0$ against $H_a:\mu>\mu_0$, define the stagewise statistic and its one-sided t-based p-value by $$ T_i=\frac{\bar{x}_i-\mu_0}{s_i/\sqrt{n_i}}, \qquad p_i=1-F_{\nu_i}(T_i), $$ where $F_{\nu_i}$ is the cumulative distribution function of a central t distribution with $\nu_i$ degrees of freedom. Its normal score is $Z_i=\Phi^{-1}(1-p_i)$, where $\Phi$ is the standard normal cumulative distribution function.

For the exact finite-sample statements below, observations within each stage are independent and normally distributed, with mean $\mu$ and a common variance within that stage. The stages use disjoint independent observations. The variance may be unknown, and it may differ between stages: each stage's t statistic is still central t at $\mu=\mu_0$, since each uses its own sample variance. An adaptation based on stage-one data may select the stage-two sample size or test, provided the selected stage-two p-value remains valid under its specified model and the stage-two data remain independent of stage one.

Exact t P-Values and the Composite Null

At the boundary $\mu=\mu_0$, $T_i$ has a central t distribution with $\nu_i$ degrees of freedom. Consequently, $p_i$ is exactly uniform on $[0,1]$, and $Z_i$ is exactly standard normal. These are finite-sample statements; they do not require replacing the estimated standard error by a known one.

The null hypothesis is composite, however. If $\mu<\mu_0$, the p-value is not generally uniform. It is superuniform: for every $u\in[0,1]$, $P(p_i\leq u)\leq u$. To see why, write the stagewise statistic under the normal model as $T_i=(G+\delta)/\sqrt{V/\nu_i}$, where $G\sim N(0,1)$, $V$ is an independent chi-squared variable with $\nu_i$ degrees of freedom, and $\delta=(\mu-\mu_0)\sqrt{n_i}/\sigma_i\leq0$. Coupling the null and alternative statistics using the same $G,V$ gives $T_i(\delta)\leq T_i(0)$. Therefore the probability that $T_i$ exceeds its central-t upper-tail cutoff is no greater than at the boundary. A lower mean produces p-values that tend to be larger, not another uniform distribution.

Because stage two uses independent observations, its p-value has this same distribution conditional on the stage-one data, for a fixed value of $\mu$ and the variance. In particular, it is conditionally uniform at $\mu=\mu_0$ and conditionally superuniform when $\mu<\mu_0$. This distinction is essential when describing conditional error: the boundary conditional rejection probability equals the calculated allowance; elsewhere in the composite null it is no greater than the allowance.

Worked Example: Calculating a Finite-Sample Stagewise P-Value

Suppose a hypothetical first stage has $n_1=20$, $\bar{x}_1=5.42$ units, $s_1=1.20$ units, and tests $H_0:\mu\leq5.00$ units. The independent normal-sampling assumption is stipulated for this example, and the variance is estimated from these 20 observations.

The standard error is $1.20/\sqrt{20}=0.2683$ units, so $$ T_1=\frac{5.42-5.00}{1.20/\sqrt{20}}=1.565, \qquad df=19. $$

Using the $t_{19}$ upper tail gives $p_1\approx0.067$. Its normal score is $Z_1=\Phi^{-1}(1-0.067)\approx1.50$. The p-value, rather than the t statistic itself, is what is transformed. At $\mu=5.00$, this p-value is exactly uniform under repeated sampling; for $\mu<5.00$, it is superuniform.

Conditional Error for an Inverse-Normal Test

Consider a one-sided inverse-normal combination test with fixed weights $w_1,w_2>0$, $w_1^2+w_2^2=1$, and rejection rule $w_1Z_1+w_2Z_2>c$, where $c=\Phi^{-1}(1-\alpha)$. Under the boundary null, independent stagewise exact p-values give independent standard normal scores. Thus the combined statistic is standard normal and the test has boundary type I error $\alpha$.

After observing stage one, $Z_1=z_1$ is fixed. At the boundary, rejection occurs precisely when $Z_2>(c-w_1z_1)/w_2$. Since $Z_2$ is conditionally standard normal, the conditional error is

A(z_1)=1-\Phi\!\left(\frac{c-w_1z_1}{w_2}\right).

Equivalently, the stage-two p-value must be at most $A(z_1)$. Conditional on any fixed stage-one result, this rule rejects with probability exactly $A(z_1)$ at the boundary. At $\mu<\mu_0$, stage two is conditionally superuniform, so its rejection probability is at most $A(z_1)$. A recalibrated stage-two test that uses an exact t-based p-value and this cutoff therefore respects the conditional allowance under the composite null.

The same logic supports conditional-error preservation after a stage-one-driven design change: conditional on stage one, the new stage-two rule must have rejection probability no greater than the original conditional allowance. At the boundary, the original conditional rejection probability equals the allowance, so the new rule's conditional bound implies its unconditional rejection probability is no greater than the original test's. Away from the boundary, averaging only bounds the new rejection probability by the expected conditional allowance; this expectation is at most $\alpha$, since $A(z_1)$ is increasing and the stage-one normal score is stochastically no greater than standard normal under the composite null. This argument does not make an invalid stage-two p-value valid; its conditional distribution still has to be justified for the selected test and sample size.

Worked Example: Recalibrating a Stage-Two t-Test

Suppose a hypothetical design uses $\alpha=0.025$, weights $w_1=0.60$ and $w_2=0.80$, and observes a stage-one exact t-based p-value $p_1=0.100$. The weights satisfy $0.60^2+0.80^2=1$. Assume the independent normal-sampling model and an exact one-sided t-test at stage two.

The first-stage normal score is $z_1=\Phi^{-1}(0.900)=1.2816$, and $c=\Phi^{-1}(0.975)=1.9600$. The conditional-error allowance is $$ A(z_1)=1-\Phi\!\left(\frac{1.9600-0.60(1.2816)}{0.80}\right) =1-\Phi(1.4888)\approx0.0682. $$

Suppose adaptation selects a stage-two sample size of 25, giving $df=24$. Its one-sided t-based p-value must be no greater than 0.0682 to reject. Equivalently, the stage-two statistic must exceed the $t_{24}$ quantile at probability $1-0.0682$, approximately 1.54. If the observed hypothetical stage-two statistic is 1.70, its $t_{24}$ upper-tail p-value is about 0.05103. Since $0.05103<0.0682$, the adapted test rejects. The comparison is valid because the cutoff is applied to the exact t-based p-value, and stage two is independent of the stage-one adaptation.

Exact Calibration Versus a Normal Approximation

A common shortcut is to calculate a stagewise p-value as $1-\Phi(T_i)$, treating a studentized statistic as standard normal. This can be a useful large-sample approximation, but it is not an exact finite-sample p-value when the variance is estimated. In particular, the resulting p-value is not generally uniform at the boundary for small $n_i$. Feeding it into a conditional-error formula derived for uniform p-values can therefore misstate the conditional rejection probability.

Worked Example: A Small-Sample Approximation Can Change the Decision

Suppose the conditional-error allowance after stage one is 0.040. For example, with $\alpha=0.025$, $w_1=0.60$, and $w_2=0.80$, this allowance corresponds approximately to $z_1=0.932$, or $p_1=1-\Phi(0.932)\approx0.176$. At stage two, suppose the hypothetical sample size is $n_2=5$, so $df=4$, and the observed studentized statistic is $T_2=2.20$.

The exact stage-two p-value is the $t_4$ upper tail, approximately 0.04633. Since $0.04633>0.040$, the stage-two result does not meet the conditional-error cutoff. The normal approximation instead gives $1-\Phi(2.20)\approx0.0139$, which is below 0.040 and would lead to rejection. The disagreement is not a paradox: the estimated variance makes the finite-sample t distribution heavier-tailed than the standard normal. The exact t calculation is appropriate under the stipulated model; the normal calculation is an approximation and is particularly questionable here with only four degrees of freedom.

Assumptions, Diagnostics, and Limitations

Before using this calculation, check the ingredients that make its conditional distribution valid. The stagewise observations must be independent across stages; the stage-two sample size or test choice must be determined without looking at stage-two outcomes; and the p-value used for the selected stage-two procedure must have the claimed null distribution. Under the exact t approach, normal sampling within each stage supports the finite-sample t reference distribution. If these assumptions are not credible, the exactness claim does not apply.

Check the reference distribution. Confirm that the p-value uses the correct degrees of freedom and tail direction. Replacing $F_{\nu_i}$ by $\Phi$ changes the finite-sample calculation.
Check stage independence. Reusing observations, sharing data-dependent variance estimates across stages, or allowing stage-two outcomes to influence the stage-two rule can invalidate the conditional distribution used here.
Check the adaptation rule. Stage-one-driven changes are compatible with the argument only when the selected stage-two test remains valid conditional on stage one. If selection changes the null distribution in an unaccounted-for way, recompute or bound the conditional rejection probability.
Check the null interpretation. At $\mu=\mu_0$, exact conditional error equals the allowance. At $\mu<\mu_0$, it is bounded above by the allowance, not generally equal to it.

The exact t calculation depends on a normal model and on the specified stagewise test. For nonnormal data, the t distribution may be only approximate, especially in small samples. The large-sample normal approach may then be reasonable, but its accuracy depends on sample size and distributional conditions; it should be described and assessed as an approximation. Also, the result here concerns independent stagewise tests combined through an inverse-normal rule. A pooled final t-test has a different statistic and conditional distribution, so this formula cannot simply be transferred to it.

Approach	Stagewise p-value	Boundary calibration	Main caution
Exact studentized approach	Upper tail from the appropriate t distribution	Exact under the stated normal model	Requires valid degrees of freedom and stagewise independence
Large-sample normal approximation	Upper tail from the standard normal distribution	Approximate when variance is estimated	May materially misstate tail probabilities with small samples
Naive use of a pooled final t-test	One pooled statistic rather than the stated stagewise p-values	Not given by the inverse-normal conditional-error formula	Requires its own conditional-distribution argument

Common Mistake

Incorrect reasoning: “The null is $H_0:\mu\leq\mu_0$, so the stage-two p-value is uniform conditional on stage one for every null mean.”

Correction: Exact uniformity holds at the boundary $\mu=\mu_0$. Under $\mu<\mu_0$, the stagewise p-value is superuniform, including conditionally on stage one under the independent-stage model. Thus the boundary conditional rejection probability equals the allowance, while elsewhere in the composite null the rejection probability is no greater than the allowance. This distinction is what supports type I error control without claiming equality throughout the null.

Key takeaway: With independent normal stages and estimated variances, use exact t-based stagewise p-values before applying an inverse-normal conditional-error calculation. The normal-score transformation is exact at the boundary; under the rest of the composite null, the relevant p-values are superuniform and conditional rejection is bounded by the boundary allowance.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

At what value of $\mu$ is a one-sided exact t-test p-value exactly uniform under $H_0:\mu\leq\mu_0$?
If $p_1=0.20$, $w_1=0.60$, $w_2=0.80$, and $\alpha=0.025$, calculate the conditional-error allowance to about three decimal places.
Why must a stage-two t-test use its selected sample size's degrees of freedom after re-estimation?
What does conditional superuniformity imply about rejection probability when $\mu<\mu_0$?

Solutions. (1) At the boundary, $\mu=\mu_0$; below the boundary the p-value is superuniform, not generally uniform. (2) $z_1=\Phi^{-1}(0.80)=0.8416$, so the cutoff is $(1.9600-0.60(0.8416))/0.80=1.819$, giving $A=1-\Phi(1.819)\approx0.03446$, or 0.034. (3) The t reference distribution depends on the stage-two degrees of freedom, which are determined by the selected sample size; using the wrong degrees of freedom gives the wrong finite-sample tail probability. (4) Conditional on stage one, rejection probability is no greater than the boundary allowance, so the boundary calculation controls the composite null rather than asserting equality throughout it.