One-Look Testing as a Reference Point

Why Begin With One Planned Analysis?

A clinical trial may be designed to compare a treatment with a control using one analysis at a prespecified sample size. That single planned decision provides a clear reference point: the test statistic, rejection threshold, and Type I error all refer to the same analysis. If the trial team instead examines results repeatedly while the trial is underway, it is using a different decision procedure. The single-analysis calibration alone does not establish the error rate of that different procedure.

The aim here is to make the reference point precise. We will define a fixed-sample, one-look test, derive its Type I error under a standard model, and calculate decisions for hypothetical clinical-trial data. The focus is the baseline rejection decision, not how to design or calibrate repeated interim decisions.

For a composite null hypothesis, error control at level \(\alpha\) means that the probability of rejection is no greater than \(\alpha\) at every parameter value covered by \(H_0\). Equivalently, the largest such probability over the null parameter space is at most \(\alpha\). A test may have an error probability exactly equal to \(\alpha\) under a simple null, while for a composite null its maximum may be smaller or may occur at a boundary.

A Fixed-Sample Normal-Outcome Reference Test

Consider a hypothetical, parallel-group randomized trial with a continuous outcome measured once for each participant at the fixed end of follow-up. Let \(\mu_T\) and \(\mu_C\) be the population mean outcomes in the treatment and control groups, and define the treatment effect as \(\delta=\mu_T-\mu_C\). Suppose larger outcomes are favorable, but the scientific question is whether the groups differ in either direction. The hypotheses are \(H_0:\delta=0\) and \(H_a:\delta\ne 0\).

For an exact z test, assume the treatment and control observations are independent, normally distributed within their groups, and have known common standard deviation \(\sigma\). The allocation and sample sizes are fixed in advance, and the analysis is performed once according to the prespecified plan. Under these assumptions, the difference in sample means has mean \(\delta\) and variance \(\sigma^2(1/n_T+1/n_C)\). The standardized statistic is

Under \(H_0\), \(Z\sim N(0,1)\). For a two-sided test with Type I error level \(\alpha\), reject \(H_0\) when \(|Z|>z_{1-\alpha/2}\), where \(z_q\) is the \(q\)-quantile of the standard normal distribution. A result in either tail counts as evidence against equality: the observed treatment mean could be sufficiently higher or sufficiently lower than the control mean.

Why the Rejection Rule Has Its Nominal Error

The null distribution gives a direct justification for the one-look rule. Under the null, the probability of falling above the positive cutoff is \(\alpha/2\), and by symmetry the probability of falling below the negative cutoff is also \(\alpha/2\). These disjoint tail events together are exactly the rejection event.

This is a statement about the probability of the test’s decision over repetitions of the same trial design under the null—not the probability that \(H_0\) is true after seeing a result. The observed statistic determines whether this particular trial’s data cross the cutoff; the Type I error rate describes how often the rule would reject across hypothetical repetitions when the null holds.

Reading the p-Value and the Error Rate Correctly

For a two-sided z test, the p-value is the probability, calculated under \(H_0\), of observing a statistic at least as far from zero as the observed statistic. It is not the probability that the null is true, nor is it the chance that a future study will reproduce the observed effect. The rejection rule at level \(\alpha\) and the p-value give equivalent decisions: reject when the p-value is no greater than \(\alpha\), provided both use the same test, null distribution, and tail convention.

The nominal level \(\alpha\) is a design choice for the test’s false-positive probability. For example, setting \(\alpha=0.05\) means that, under the null and the stated model, the one planned test rejects with probability 0.05 for the exact z procedure above. It does not mean that 5% of observed rejections are necessarily false; that latter quantity would also depend on how often null and alternative states arise and on the test’s performance under alternatives.

When the Standard Deviation Is Estimated

In many trials, \(\sigma\) is unknown. A common fixed-sample alternative for independent normal outcomes is the pooled two-sample t test, provided equal population variances are reasonable. Its pooled variance estimate is formed from the two sample variances, and the statistic is compared with a t distribution rather than a standard normal distribution. The t reference distribution accounts for estimating the common variance; using a z cutoff as though the variance were known would generally not give the exact stated test.

Assumptions, Diagnostics, and Limits

A rejection rule has its stated operating characteristics only when its model and implementation are appropriate. For the exact known-variance z example, the assumptions were independent normal outcomes and a genuinely known common standard deviation. For the pooled t example, normality, independence, and equal variances support the reference distribution. In either case, the sample sizes, outcome, test direction, significance level, and analysis timing must match the plan used to define the test.

Practical checks include verifying group counts and outcome definitions, inspecting distributions and influential observations, and checking whether variability is reasonably comparable when using a pooled t test. A marked departure from the model, substantial missing outcomes, dependence between participants, or an unplanned change in the outcome or analysis can make the nominal calibration unreliable. Diagnostics do not prove assumptions, but they can reveal conflicts that warrant a justified alternative analysis.

A one-look design is also limited in scope: it makes one inferential decision at the planned analysis time. An interim examination may be useful for trial conduct, but if it can trigger a decision about efficacy, the decision procedure now includes that opportunity to reject. The one-look Type I error calculation above is not, by itself, a justification for that expanded procedure. Repeated monitoring requires its own prespecified error-control reasoning.

Common Mistakes and Comparisons

• Confusing \(\alpha\) with the p-value. The significance level is set as part of the test plan; the p-value is calculated from the observed statistic. Compare them only after the test and tail have been specified.
• Changing tails after seeing the data. A one-sided test is justified by a prespecified directional question. If either direction matters, use a two-sided test rather than selecting a tail post hoc.
• Treating non-rejection as proof of no effect. Failure to cross the rejection boundary means the test did not find sufficient evidence at its chosen level. It does not establish equality or rule out clinically important effects.
• Assuming the final one-look p-value repairs earlier unplanned decisions. The p-value belongs to the stated test procedure. Additional data-dependent decision opportunities require separate error-control justification.

A fixed one-look test is simpler than a monitoring procedure because its rejection event is tied to one prespecified analysis. Its simplicity is useful as a reference, not as a reason to ignore the actual conduct of a trial. Likewise, alternatives such as a t test or a model-based analysis may better match unknown variances or other design features, but each requires its own assumptions and calibration. The appropriate baseline is the test that was specified for the trial—not whichever analysis produces the preferred conclusion.