Estimating Effects With an Estimated Standard Error

Why an Estimated Standard Error Changes the Stopped Estimate

In a group-sequential trial, a decision to stop can depend on an effect estimate divided by its estimated standard error. When the variance is unknown, that denominator is random. Thus a fixed boundary on the standardized statistic does not correspond to a fixed boundary on the effect scale: for the same estimated effect, a smaller estimated standard error makes crossing an efficacy boundary more likely. Stopping consequently changes not only which sample size is observed but also the distribution of the estimate and its estimated standard error among trials that stop.

The normal model gives a precise way to see these effects. The fixed-sample result is familiar: estimating the variance changes a standard normal statistic into a t statistic. But the fixed-sample t distribution alone does not describe an estimate selected by a stopping rule. The selection event depends on the studentized statistic, so the distribution at a stopping look is selected as well.

Model and notation: Suppose the observations are independent and identically distributed as $X_i\sim N(\theta,\sigma^2)$, with unknown mean $\theta$ and unknown variance $\sigma^2$. At look $k$, $n_k$ observations have been accrued, and $\bar{X}_k$ and $S_k$ are their sample mean and sample standard deviation. For testing a reference value $\theta_0$, define the studentized statistic $T_k=(\bar{X}_k-\theta_0)/(S_k/\sqrt{n_k})$. A design stops at the first look where its prespecified stopping criterion is met; denote that look by $N$. The reported ordinary effect estimate is $\hat{\theta}=\bar{X}_N$, and its estimated standard error is $S_N/\sqrt{n_N}$.

This illustration assumes normal observations, a common variance across observations, and a variance estimate calculated from the accrued observations. It also assumes that the looks and stopping boundaries are specified in advance. These assumptions give the exact fixed-look t result below. They do not imply that studentized statistics from different looks are independent, or that a boundary calibrated as if the variance were known remains calibrated when the variance is estimated.

From a Fixed-Look t Statistic to a Stopped Statistic

At a fixed sample size $n$, the normal model gives two standard facts: $\bar{X}$ is normally distributed with mean $\theta$ and variance $\sigma^2/n$, and the sample variance is independent of $\bar{X}$. In particular, $(n-1)S^2/\sigma^2$ has a chi-squared distribution with $n-1$ degrees of freedom. Combining these facts shows that

\frac{\bar{X}-\theta}{S/\sqrt{n}}\sim t_{n-1}.

This is the distribution of the standardized estimation error at a fixed, prespecified sample size. For a statistic centered at $\theta_0$, its fixed-look distribution is a noncentral t distribution with $n-1$ degrees of freedom and noncentrality parameter $\sqrt{n}(\theta-\theta_0)/\sigma$. Under $H_0:\theta=\theta_0$, that noncentrality is zero and the distribution is central t.

Stopping changes the question. At look $k$, the event $N=k$ includes the condition that earlier stopping criteria were not met and that the criterion at look $k$ was met. Therefore the distribution of the estimate at that look, given that the trial stops there, is its distribution conditional on this selection event. In general it is not the fixed-look normal distribution for $\bar{X}_k$, nor the unselected fixed-look t distribution for $T_k$.

Key result: The stopped estimate has a mixture distribution over stopping looks. For any set $A$, $$ P(\hat{\theta}\in A)=\sum_k P(N=k)\,P(\bar{X}_k\in A\mid N=k). $$ The conditional distributions in this mixture depend on the stopping rule. Estimating $\sigma$ makes the rule depend on the random $S_k$ as well as on the estimated effect. Thus the stopped distribution is generally not obtained by simply replacing a known $\sigma$ with $S_N$ in a fixed-sample formula.

A useful way to see the variance-estimation effect is to rewrite a one-sided efficacy criterion $T_k\ge c_k$, for $c_k>0$, on the effect scale:

\bar{X}_k-\theta_0\ge c_k\frac{S_k}{\sqrt{n_k}}.

The threshold on the right is random. A low observed $S_k$ lowers the effect required to cross the boundary; a high $S_k$ raises it. Even though $\bar{X}_k$ and $S_k$ are independent at a fixed, nonselected sample size under the normal model, the event that a studentized statistic crosses a boundary connects them in the selected sample. Independence before selection is not a reason to assume independence after selection.

Worked Examples

Worked Example: A Studentized Effect at an Efficacy Look

Consider a hypothetical normal-model trial with a continuous outcome measured in points. At a planned look, $n=16$, the observed mean is 52.4 points, the reference mean is $\theta_0=50.0$ points, and the sample standard deviation is $S=4.8$ points. Suppose the illustrative efficacy boundary is $c=1.40$. Calculate the estimated effect, its estimated standard error, and the studentized statistic.

The estimated effect relative to the reference is $52.4-50.0=2.4$ points. The estimated standard error is $$ SE=\frac{S}{\sqrt{n}}=\frac{4.8}{\sqrt{16}}=\frac{4.8}{4}=1.2\text{ points}. $$ Therefore $$ T=\frac{2.4\text{ points}}{1.2\text{ points}}=2.00. $$ Since $2.00\ge 1.40$, this hypothetical trial crosses the stated boundary at this look. In effect units, the boundary requires an estimated effect of at least $$ c\frac{S}{\sqrt{n}}=1.40(1.2)=1.68\text{ points}, $$ and the observed 2.4-point effect exceeds that threshold.

The boundary here is illustrative, not a claim that this cutoff controls a specified trial-wide error rate. A fixed-look t reference distribution can describe a statistic at a prespecified look under its model assumptions; it does not, by itself, provide a stopping-adjusted p-value for this sequential decision.

Worked Example: The Same Effect Can Face Different Boundaries

Suppose two hypothetical samples at the same planned look each have $n=25$ and an estimated effect of 2.0 points above $\theta_0$. The first has $S=4.0$ points and the second has $S=8.0$ points. Use an illustrative one-sided boundary $c=2.00$ to compare their decisions.

For the first sample, $$ SE_1=\frac{4.0}{\sqrt{25}}=0.8\text{ points},\qquad T_1=\frac{2.0}{0.8}=2.50. $$ Its effect-scale boundary is $2.00(0.8)=1.6$ points, so the 2.0-point estimate crosses.

For the second sample, $$ SE_2=\frac{8.0}{\sqrt{25}}=1.6\text{ points},\qquad T_2=\frac{2.0}{1.6}=1.25. $$ Its effect-scale boundary is $2.00(1.6)=3.2$ points, so the 2.0-point estimate does not cross. The effect estimate is identical, but the estimated variability changes the stopping decision. In repeated trials, this rule preferentially selects combinations of effect estimates and standard errors that meet the studentized criterion; the distribution of estimates among stopping trials therefore cannot be understood from the effect estimates alone.

Worked Example: Selection at a Single Look Under the Null

For a mathematical illustration, suppose a hypothetical normal sample has $n=4$, the true mean equals $\theta_0$, and the trial stops for efficacy at its only look if $T>1$. This cutoff is chosen to make the selection effect visible; it is not proposed as a conventional significance boundary. Under $H_0$, $T\sim t_3$, so the probability of stopping is $$ P(T>1)=1-F_{t_3}(1)\approx 0.1955, $$ where $F_{t_3}$ is the cumulative distribution function of a t distribution with 3 degrees of freedom.

The conditional density after stopping is the right-truncated t density: $$ f_{T\mid T>1}(t)=\frac{f_{t_3}(t)}{P(T>1)},\qquad t>1. $$ Its conditional mean can be calculated by integrating the t density. For $\nu>1$, the t density is $f_{t_\nu}(t)=C_\nu(1+t^2/\nu)^{-(\nu+1)/2}$, where $C_\nu=\Gamma((\nu+1)/2)/[\sqrt{\nu\pi}\Gamma(\nu/2)]$. Integrating $t f_{t_\nu}(t)$ from $c$ to infinity gives $$ \int_c^\infty t f_{t_\nu}(t)\,dt =C_\nu\frac{\nu}{\nu-1}\left(1+\frac{c^2}{\nu}\right)^{-(\nu-1)/2}. $$ For $\nu=3$ and $c=1$, this numerator is approximately $0.4135$, so $$ E(T\mid T>1)=\frac{0.4135}{0.1955}\approx 2.115. $$ Thus the studentized statistic among trials that stop is, on average, well above the cutoff even though the true effect relative to $\theta_0$ is zero. This calculation concerns the selected ratio $T$, not the raw effect in points: because $T$ includes the random $S$, its conditional mean cannot be converted to the raw effect mean by multiplying by an unselected standard error.

What the Derivation Does and Does Not Establish

The one-look calculation illustrates the central selection mechanism. At a fixed look under $H_0$, the studentized statistic has a central t density. Given that the trial stops because $T>c$, the density is proportional to the original density only in the region $t>c$. This follows directly from the definition of conditional density:

f_{T\mid T>c}(t) =\frac{f_T(t)\,I(t>c)}{P(T>c)}.

The conditional distribution is therefore shifted toward values that satisfy the stopping criterion. With several looks, the event $N=k$ also includes non-crossing conditions at previous looks. The stopped distribution then combines stage-specific selected distributions, and the statistics at different looks share accrued observations. It is not generally correct to treat each look as an independent t test or to infer the stopped distribution from a single t table.

The ordinary estimate $\bar{X}_N$ remains a natural estimate of the mean at the observed sample size. That does not mean it is unbiased under the experiment's stopping plan: its repeated-sampling bias is $$ E_\theta(\bar{X}_N-\theta)=\sum_k P_\theta(N=k)\,E_\theta(\bar{X}_k-\theta\mid N=k), $$ which need not be zero. The selection can favor positive or negative estimates depending on the boundaries and which stopping outcomes are included. Estimating the variance adds a random scale to this selection and can make its magnitude sensitive to the behavior of $S_k$.

Diagnostics, Alternatives, and Limits

Several checks help reveal when a studentized normal-model analysis may be unreliable:

Check the variance estimate by look. Very small sample sizes can make $S_k$ unstable. Review the observed $S_k$ values and their degrees of freedom, rather than treating the estimated standard error as fixed.
Check the outcome model. Strong skewness, outliers, dependence, or changing variance can undermine the exact fixed-look t reference. The normal-model result is not automatically exact for other outcome distributions.
Check the actual stopping criterion. Record whether the boundary used $S_k/\sqrt{n_k}$, a variance estimate pooled across groups, or another estimated standard error. The distribution depends on that choice.
Check boundary calibration under the implemented rule. A design calibrated using known variance or normal rather than studentized statistics may not have its intended operating characteristics when variance is estimated. Calibration must match the statistic, variance estimator, looks, and stopping rules actually used.

The main alternative is a design with known variance, where the standardized statistic uses $\sigma/\sqrt{n_k}$ and the effect-scale boundary is fixed at each look. That removes the random-denominator component, but stopping still selects estimates and changes their distributions. A fixed-sample t analysis is another useful reference when the sample size is fixed in advance; it does not account for selection caused by interim stopping. Earlier material on conditional likelihood and bias-adjusted estimators offers different ways to estimate after stopping, but the present result concerns how studentization itself enters the sampling and selection mechanism.

These calculations do not supply confidence intervals or a general stopping-adjusted p-value. Nor does a more accurate variance estimate remove selection bias: the key issue is that the estimate of variability is part of the stopping rule. In practice, conclusions about stopped effects should be based on a method whose sampling calculation reproduces the design's variance estimation and stopping decisions.

Common Mistake

Wrong reasoning: “At each look the statistic has a t distribution, so after stopping I can report the fixed-sample t distribution and regard the standard error as an ordinary, nonrandom scale.” The fixed-look t result applies before conditioning on the stopping event. Once the decision depends on the statistic, the observations reported at a stopping look are selected. The correction is to distinguish the unselected fixed-look distribution from the conditional distribution given the stopping rule, and to remember that the estimated standard error makes the effect-scale threshold random.

Key takeaway: With an estimated standard error, a studentized boundary selects both effect estimates and variance estimates. Fixed-look t theory is an essential building block, but it is not the sampling distribution of an estimate conditional on stopping.

Check Your Understanding

Try each question, then compare your reasoning with the concise solutions below.

At a fixed look under the normal model, what distribution does $(\bar{X}-\theta)/(S/\sqrt{n})$ have, and what are its degrees of freedom?
A one-sided boundary is $T_k\ge c_k$. Write the equivalent condition on $\bar{X}_k-\theta_0$.
Why does the fixed-look t distribution not automatically describe $T_k$ conditional on stopping at look $k$?
At $n=36$, an illustrative boundary is $c=1.8$ and the estimated standard deviation is 6 points. What is the boundary in effect units?
Does estimating the variance remove the selection effect caused by stopping? Explain briefly.

Solutions. (1) It has a t distribution with $n-1$ degrees of freedom. (2) The condition is $\bar{X}_k-\theta_0\ge c_k S_k/\sqrt{n_k}$. (3) Stopping conditions on the statistic, so the relevant distribution is conditional on the selection event, not the unselected fixed-look distribution. (4) The estimated standard error is $6/\sqrt{36}=1$ point, so the effect-scale boundary is $1.8(1)=1.8$ points. (5) No. Estimating the variance makes the boundary random in effect units, and stopping still selects estimates and standard errors.