Studentized Randomization Tests For Weak Nulls

Why Studentization Matters for a Weak Null

A sharp null specifies every unit’s treatment effect; a weak null specifies only an average. That difference matters for randomization inference. Under a sharp null, the unobserved potential outcomes can be imputed, so the assignment mechanism gives an exact reference distribution. Under a weak null, individual effects may vary even when their average is zero. The missing potential outcomes are therefore not determined by the null, and the same exact construction is unavailable.

Studentization addresses a different problem: it scales an estimated average effect by an estimate of its sampling variability. With suitable large-sample conditions, the resulting statistic can have an approximately standard normal randomization distribution under the weak null. This is an asymptotic conclusion, not a claim of exactness for every adaptive design. In particular, the variance estimate and reference distribution must reflect the sequential assignment mechanism.

Definition: For $N$ units with potential outcomes $Y_i(1)$ and $Y_i(0)$, the finite-population average treatment effect is $\tau_N=N^{-1}\sum_{i=1}^N\{Y_i(1)-Y_i(0)\}$. The weak null is $H_0:\tau_N=0$; it permits individual effects $Y_i(1)-Y_i(0)$ to differ across units. Given an estimator $\hat{\tau}$ and an estimated design-based variance $\hat V$, the studentized statistic is $T=(\hat{\tau}-0)/\sqrt{\hat V}$. The variance estimator must target the variance of $\hat{\tau}$ under the actual assignment design.

The notation $\hat{\tau}$ is deliberately general. A difference of observed arm means is not automatically design-unbiased or appropriately studentized when assignment probabilities change with past outcomes. A design-specific estimator and variance calculation are needed. The introductory material on sharp and weak null hypotheses distinguishes the null statements; here the focus is how studentization supports approximate inference for the weak one.

Conditions Behind Approximate Calibration

The asymptotic argument is compact, but its assumptions are substantive. Consider a sequence of finite-population trials whose size $N$ grows. The assignment at each step is randomized according to a known rule conditional on the observed history, which may include earlier assignments and responses. For the following approximation to be credible, conditions typically include:

Well-defined potential outcomes: each unit has a potential outcome under each treatment, treatment is consistently recorded, and one unit’s assignment does not alter another unit’s outcome (no interference).
Valid sequential randomization: the stated assignment probabilities are the probabilities actually used, conditional on the trial history available at each assignment.
Positivity and nondegenerate allocation: treatment probabilities remain bounded away from zero and one, or satisfy a suitable weaker condition, so information does not collapse into one arm. The arm sizes must grow.
Controlled influence: no single unit or assignment step dominates the estimator. Outcome moments and the accumulation of assignment-dependent contributions must be regular enough for a central limit theorem.
Consistent variance estimation: $\hat V$ estimates the relevant randomization variance, including variability induced by the adaptive rule. In particular, it must be positive and satisfy $\hat V/V\to 1$ in probability for the target variance $V$.
Appropriate calibration: the reference distribution used for the test must approximate the distribution of the studentized statistic under the weak null, under these same design and asymptotic conditions.

These are a framework, not a universal theorem covering every response-adaptive algorithm. For a particular design, the estimator’s limiting distribution and the consistency of its variance estimate must be justified. Bounded assignment probabilities alone do not establish validity: they do not ensure that the estimator targets $\tau_N$, that its variance estimate is correct, or that a proposed resampling distribution is appropriate.

Asymptotic justification: Suppose under the design and regularity conditions, $(\hat{\tau}-\tau_N)/\sqrt{V}$ converges in distribution to $N(0,1)$, and $\hat V/V$ converges in probability to 1. Under $H_0:\tau_N=0$, $$ T=\frac{\hat{\tau}}{\sqrt{\hat V}} =\frac{(\hat{\tau}-\tau_N)/\sqrt{V}}{\sqrt{\hat V/V}} \ \xrightarrow{d}\ N(0,1). $$ The conclusion follows from Slutsky’s theorem: the numerator has a standard normal limit and the denominator converges in probability to 1. It is asymptotic; it does not imply that $T$ is exactly standard normal in a finite trial.

Calculating and Interpreting the Statistic

Worked Example: A Two-Sided Test With a Design-Aware Standard Error

Suppose a hypothetical adaptive trial has a design-justified estimator $\hat{\tau}=2.3$ outcome units and estimated standard error $SE=\sqrt{\hat V}=1.188486$ outcome units. Assume the trial’s allocation probabilities, estimator, and variance estimator meet the conditions for the stated large-sample normal approximation. Test $H_0:\tau_N=0$ against $H_a:\tau_N\ne0$ at $\alpha=0.05$.

The studentized statistic is $$ T_{\mathrm{obs}}=\frac{2.3}{1.188486}=1.93524\ldots\approx1.935. $$

Using a two-sided standard normal reference and retaining the unrounded statistic for the tail calculation, $$ p=2\{1-\Phi(1.93524\ldots)\}\approx0.0530. $$

Because $0.0530>0.05$, the test does not reject the weak null at the 5% level. The data do not provide sufficient evidence, under the assumed calibration, that the finite-population average effect differs from zero. This is not proof that the average effect is zero. The point estimate is 2.3 outcome units, and its standard error is 1.188486 outcome units.

In a design-based analysis, the displayed standard error cannot simply be chosen because it makes the statistic look plausible. It must estimate the variance induced by the actual adaptive assignment and the chosen estimator. A Welch standard error, for example, belongs to a particular two-sample setting; it is not automatically correct for an outcome-adaptive sequential design.

Worked Example: Why a Zero Average Does Not Mean a Sharp Null

Consider a hypothetical finite population of 100 units: 50 have treatment effect $+6$ outcome units and 50 have treatment effect $-6$ outcome units. The finite-population average effect is $$ \tau_N=\frac{50(6)+50(-6)}{100}=0. $$

Thus the weak null is true, but the sharp null that every unit’s effect is zero is false. An exact sharp-null test can impute each unit’s missing outcome only under its sharp null; it cannot use that imputation to test this heterogeneous-effect population as though every effect were zero. A studentized test instead uses an estimator of the average and a variance estimate that accounts for the design and the uncertainty from heterogeneous effects.

This example does not supply enough information to calculate a standard error: the potential outcome levels and assignment mechanism also matter. Its conclusion is only about the distinction between the nulls. Treating the sharp-null reference distribution as an exact weak-null reference would be unjustified.

Randomization Calibration Under Adaptation

A randomization test is calibrated by comparing the observed statistic with a reference distribution generated under a specified null and assignment procedure. Under a sharp null, the assignment procedure and imputable outcomes give a finite-sample exact test (subject to the stated randomization scheme). Under a weak null, the assignment path alone does not provide the missing potential outcomes. Therefore, simply simulating new treatment labels while holding the observed outcomes fixed does not automatically produce a valid weak-null reference distribution.

For a studentized weak-null test, one defensible route is an asymptotic standard normal reference, provided the estimator and variance estimator have the limiting behavior described above. A design-specific simulation can instead approximate a null reference distribution when its construction is justified for the weak null—for example, through an appropriate null outcome model or a proven asymptotic resampling argument. Such a procedure may be useful, but its validity comes from that justification, not merely from rerunning the randomization algorithm.

Worked Example: Monte Carlo Tail Area With a Justified Reference

Suppose a hypothetical analysis produces $T_{\mathrm{obs}}=1.93524$. Assume a design-specific argument has already established that simulated null studentized statistics approximate the relevant weak-null reference distribution. In $B=9{,}999$ simulated draws, suppose 527 statistics satisfy $|T_b^*|\ge |T_{\mathrm{obs}}|$. The two-sided Monte Carlo p-value, using the plus-one correction, is $$ \hat p_{\mathrm{MC}}=\frac{1+\sum_{b=1}^{B}I(|T_b^*|\ge|T_{\mathrm{obs}}|)}{B+1} =\frac{1+527}{9{,}999+1} =\frac{528}{10{,}000}=0.0528. $$

At $\alpha=0.05$, this simulated p-value does not lead to rejection. The plus-one calculation accounts for Monte Carlo sampling when estimating a tail area; it does not repair an invalid reference distribution. In particular, the result would not be justified if the 9,999 draws merely permuted fixed outcomes without a weak-null validity argument.

Diagnostics, Alternatives, and Limits

Before interpreting a result, check whether the design and approximation are credible. Review the recorded sequential assignment probabilities against the trial algorithm, and inspect whether either arm has too few assignments or probabilities close to zero or one. Examine whether a small number of observations dominate the estimated effect or variance. Compare the reported standard error with a variance calculation derived for the design, and check that the calibration method reproduces the statistic and assignment history it claims to represent. Sensitivity to reasonable variance estimates or reference methods is a warning that the approximation may be fragile.

The main alternatives answer different questions or rely on different assumptions:

Approach	Null and calibration	Main caution
Exact sharp-null randomization test	Uses imputable outcomes under a sharp null and the actual assignment mechanism	Does not become an exact test of a weak average-effect null
Studentized weak-null test	Uses an estimated average effect and variance, with asymptotic or otherwise justified weak-null calibration	Depends on estimator validity, variance consistency, and adequate asymptotics
Model-based test	Uses a specified outcome model and its standard errors	Validity depends on model assumptions; it is not automatically design-based

A common mistake is to say, “The null average effect is zero, so I can hold all observed outcomes fixed and permute treatment labels; the resulting test is exact.” The correction is that the weak null leaves individual effects and missing potential outcomes unspecified. Holding outcomes fixed is justified by a sharp null, not by an average-zero statement. Studentization can make weak-null calibration asymptotically valid under stated conditions, but it cannot turn an unjustified permutation scheme into an exact test.

Even when the conditions are plausible, small samples, highly variable outcomes, rare treatment assignments, severe response-adaptive imbalance, or unstable variance estimates can undermine the normal approximation. A non-rejection may reflect low precision rather than evidence of no meaningful effect. Report the estimator, its units, the variance method, the calibration method, and the assumptions supporting that calibration.

Specify the weak null.
State the average-effect estimand and test $H_0:\tau_N=0$, rather than silently substituting a sharp null.

Choose a design-valid estimator and variance.
Use the adaptive assignment history and justify that $\hat V$ estimates the estimator’s randomization variance.

Studentize and calibrate.
Calculate $T=\hat{\tau}/\sqrt{\hat V}$, then use only a reference distribution justified for the weak null and the design.

Qualify the conclusion.
Report the approximate or simulated p-value and the assumptions; do not describe an asymptotic test as finite-sample exact.

Key takeaway: Studentization can support approximate randomization-based inference for a weak average-effect null when the estimator is design-valid, its variance is consistently estimated, and the adaptive design satisfies suitable asymptotic conditions. It does not supply missing potential outcomes or guarantee validity for an arbitrary permutation procedure.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What is the difference between a sharp null of no effect for every unit and a weak null of zero average effect?
If $\hat{\tau}=1.5$ outcome units and $SE=0.75$ outcome units, calculate the studentized statistic.
Name two conditions, beyond a zero average effect, needed to justify an asymptotic studentized test under adaptive assignment.
Why does replaying an adaptive assignment algorithm with observed outcomes held fixed not automatically give an exact weak-null test?

Solutions. (1) The sharp null says $Y_i(1)-Y_i(0)=0$ for every unit; the weak null says only that the average of those effects is zero, allowing heterogeneous effects. (2) $T=1.5/0.75=2.00$. (3) Examples include valid sequential assignment probabilities, nondegenerate allocation with growing arm sizes, a central limit condition, and a consistent design-specific variance estimator; any two of these are sufficient answers. (4) Under the weak null, individual effects and the missing potential outcomes are not specified, so fixing observed outcomes under reassigned labels is not justified by the null. Exact imputation is available under a sharp null, while weak-null calibration needs a separate asymptotic or model-based justification.