Diagnostic: Regression to the Mean After a Favorable Look

Why a Favorable Look Can Mislead

An interim estimate is a mixture of signal and sampling noise. If the true treatment effect is modest but the interim estimate happens to be unusually favorable, a rule that responds to that estimate may treat temporary noise as if it were durable evidence. When more data are collected, the additional estimate is not selected for being favorable; on average, it is closer to the true effect. This tendency is regression to the mean.

In an adaptive design, regression to the mean matters because a favorable interim result can affect whether the trial continues and how many participants are added. It does not mean that every favorable estimate will fall, nor does it prove that a particular sample-size increase was wasteful. It identifies a risk: the estimate that prompted the decision may overstate the effect that future data will support.

Model and Assumptions

Let \(\delta\) be the true treatment effect, with positive values denoting benefit, and let \(\hat{\delta}_I\) be its estimate at an interim look with information \(I\). For the calculations here, assume a normally distributed endpoint, equal allocation, known common standard deviation \(\sigma\), independent observations, and no missing-data or other departures from the planned analysis. With \(n_I\) participants per arm at the interim, the estimator has standard error \(s_I=\sigma\sqrt{2/n_I}\), so the model is \(\hat{\delta}_I\sim N(\delta,s_I^2)\). Equivalently, information is \(I=1/s_I^2\).

These assumptions make the sampling noise explicit. The results below describe the behavior of the estimate under this model; they do not guarantee that a real endpoint, variance estimate, or adaptive rule satisfies it. In particular, the independent-increments calculations require the new observations to be independent of the interim observations.

Definition: Regression to the mean after a favorable look is the tendency for an estimate selected because it is high to be less extreme on average at a later measurement. Under the normal model, this follows because the interim estimate contains noise, whereas the mean of a fresh, independent estimate is the true effect.

Conditioning on a Favorable Interim Estimate

A useful diagnostic is to ask what interim estimate is expected among trials whose estimate exceeds a favorable threshold \(c\). Write \(Z=(\hat{\delta}_I-\delta)/s_I\), so \(Z\sim N(0,1)\), and let \(a=(c-\delta)/s_I\). Conditional on crossing the threshold, the expected estimate is

E(\hat{\delta}_I\mid \hat{\delta}_I\ge c) =\delta+s_I\frac{\phi(a)}{1-\Phi(a)},

where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. The ratio is positive, so the average selected estimate exceeds \(\delta\). The amount of excess depends on how extreme the threshold is relative to the true effect and the interim standard error.

The result follows from the conditional mean of a truncated standard normal variable. Since \(Z\ge a\) is the selection event,

E(Z\mid Z\ge a) =\frac{\int_a^\infty z\phi(z)\,dz}{1-\Phi(a)} =\frac{\phi(a)}{1-\Phi(a)}.

For the second equality, use \(\phi'(z)=-z\phi(z)\). Then \(\int_a^\infty z\phi(z)\,dz=-\int_a^\infty\phi'(z)\,dz=\phi(a)\), because \(\phi(z)\) tends to zero as \(z\) tends to infinity. Substituting \(\hat{\delta}_I=\delta+s_I Z\) gives the stated conditional expectation.

Worked Example: Average Effect Among Trials Passing a Favorable Threshold

Suppose a hypothetical trial has true effect \(\delta=0.20\) units and interim standard error \(s_I=0.15\) units. A favorable interim estimate is defined as at least \(c=0.40\) units. Under the model, the threshold on the standardized noise is \(a=(0.40-0.20)/0.15=1.333\).

Using \(\phi(1.333)\approx0.1641\) and \(1-\Phi(1.333)\approx0.09127\), the mean estimate among trials passing the threshold is

E(\hat{\delta}_I\mid\hat{\delta}_I\ge0.40) =0.20+0.15\left(\frac{0.1641}{0.09127}\right) \approx0.4697.

Thus, among hypothetical trials selected for an estimate of at least 0.40 units, the average interim estimate is about 0.4697 units, even though the true effect is 0.20 units. This conditional average is not a correction to apply automatically to one trial; it quantifies the optimism induced by threshold-based selection under the stated model.

What Happens When More Data Are Added?

Suppose the interim estimate uses information \(I_1\), and a new, independent estimate \(\hat{\delta}_2\) uses information \(I_2\). Under the same normal model, the combined estimate is the information-weighted average

\hat{\delta}_{\mathrm{combined}} =\frac{I_1\hat{\delta}_I+I_2\hat{\delta}_2}{I_1+I_2}.

Before observing the new data, \(E(\hat{\delta}_2\mid\hat{\delta}_I)=\delta\), by independence and unbiasedness. Therefore, assuming \(I_2\) is fixed across trials selected by the threshold, after selecting on \(\hat{\delta}_I\ge c\), the expected combined estimate is. If \(I_2\) depends on the interim estimate, the excess term instead is \(E[\{I_1/(I_1+I_2)\}(\hat{\delta}_I-\delta)\mid\hat{\delta}_I\ge c]\).

E(\hat{\delta}_{\mathrm{combined}}\mid\hat{\delta}_I\ge c) =\delta+\frac{I_1}{I_1+I_2}s_I \frac{\phi(a)}{1-\Phi(a)}.

The initial favorable excess is multiplied by \(I_1/(I_1+I_2)\), which is less than one when new information is added. With equal interim and added information, half of that expected excess remains in the combined estimate. The expected estimate moves toward \(\delta\); any particular follow-up estimate can still move either upward or downward.

Worked Example: Combining an Interim and a Follow-Up Estimate

Continue with the hypothetical true effect of 0.20 units and interim standard error of 0.15 units. Suppose a selected trial has an interim estimate of 0.44 units, and an equal amount of new information is collected. The new estimate is independent and has expectation 0.20 units.

With equal information, the expected combined estimate, conditional on the observed interim estimate, is \((0.44+0.20)/2=0.32\) units. The excess over the true effect has halved, from \(0.44-0.20=0.24\) to \(0.32-0.20=0.12\) units. If, in one hypothetical realization, the new estimate is 0.16 units, the observed combined estimate is \((0.44+0.16)/2=0.30\) units. It has moved toward the true effect, but it has not equaled it.

The conditional expectation and the realized combined estimate answer different questions. The value 0.32 is an average over possible independent follow-up estimates; the value 0.30 follows from the particular new estimate of 0.16 units.

How Optimism Can Affect an Increase

The direction of the decision effect depends on the rule. A rule that recalculates sample size using a larger estimated effect may actually recommend fewer participants, since the required sample size typically decreases as the assumed effect increases. By contrast, a promising-zone rule may increase the sample size for interim results that indicate possible benefit but do not meet a criterion for stopping. If such a rule responds to a favorable estimate or statistic, upward interim noise can help trigger an increase even when the true effect is smaller.

For context, under a fixed two-arm normal design with equal allocation, known \(\sigma\), a two-sided test at level \(\alpha\), and target power \(1-\beta\) for a specified effect \(\delta_{\mathrm{plan}}\), the usual approximate sample size per arm is

n=\frac{2\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\delta_{\mathrm{plan}}^2}.

This relationship makes the risk plain: a sample-size calculation based on an effect larger than the true effect may substantially understate the sample size needed for the target power. Conversely, a design that increases sample size after a favorable look can spend resources based on a result whose favorable component partly reflects noise. The equation is a fixed-design reference here, not a complete formula for an adaptive trial’s operating characteristics.

Worked Example: A Favorable-Estimate Trigger and Its Risk

Consider a hypothetical rule that adds 50% to the planned sample size per arm if the interim estimate is at least 0.35 units. Suppose the true effect is 0.10 units and the interim standard error is 0.15 units. The chance of triggering an increase is

P(\hat{\delta}_I\ge0.35) =1-\Phi\left(\frac{0.35-0.10}{0.15}\right) =1-\Phi(1.667) \approx0.0478.

The increase is therefore triggered in about 4.78% of trials under these assumptions. Among those selected trials, the expected estimate is

E(\hat{\delta}_I\mid\hat{\delta}_I\ge0.35) =0.10+0.15\left(\frac{\phi(1.667)}{1-\Phi(1.667)}\right) \approx0.10+0.15\left(\frac{0.0995}{0.0478}\right) \approx0.412\text{ units}.

The selected estimate averages about 0.412 units, although the true effect is 0.10 units. This does not show that every triggered increase is useless or that the rule is wrong: additional participants can still improve precision. It does show why the increase should not be justified as though 0.412 units were the underlying effect. Its value must be assessed under the true-effect scenarios relevant to the design, with the full rule and its costs taken into account.

Diagnostics, Alternatives, and Limitations

A useful diagnostic is to compare the effect estimate that prompted an increase with estimates from later data, while remembering that estimates vary by chance. At the design stage, calculate how often the increase rule is triggered under plausible true effects and compare the expected added sample with the resulting power or precision. If the rule depends on crossing a threshold, report how selection changes the expected estimate under the model. These checks help distinguish a decision driven by a genuinely large signal from one that is frequently activated by favorable noise.

Regression to the mean is not itself a defect in a design. It is a consequence of noisy estimation and selection. Nor does it by itself calculate the final power, error rate, or net value of a sample-size increase. Those quantities depend on the complete adaptation and analysis procedures. The normal known-variance derivations here are less reliable if the endpoint distribution, variance, independence, or prespecified decision rule differs materially from the assumptions.

Approach	How it relates to a favorable look	Main caution
Fixed sample size	Does not change sample size in response to the interim estimate.	Does not adapt resources to interim evidence.
Effect-estimate threshold rule	Uses an observed effect threshold; selection can make estimates beyond the threshold optimistic on average.	Do not treat the selected estimate as the true effect.
Conditional-power rule using a planned effect	Uses an assumed effect in its calculation rather than equating a favorable interim estimate with the true effect.	Its conclusion still depends on the assumed effect and the specified rule.

The practical choice is not simply between “adaptive” and “nonadaptive.” The key is to make the assumed effect and the decision rule explicit, and to assess the consequences across plausible true effects. A favorable look is evidence about the treatment effect, but it is also a selection event when it is used to trigger a decision.

Common Pitfall

Wrong reasoning: “The interim estimate is favorable, so the same effect should be expected in the added participants.”
Correction: Under the model, the independent new estimate has expectation \(\delta\), not \(\hat{\delta}_I\). The combined estimate retains only the information-weighted share of the interim estimate’s selected excess.
Another wrong conclusion: “Regression to the mean proves that an increase triggered by a favorable look is wasteful.”
Correction: It establishes a source of optimism, not the value of the increase. Determine whether the added sample is worthwhile by evaluating the full rule under relevant true effects and resource constraints.

Key takeaway: After selection for an unusually favorable interim result, the estimate is expected to exceed the true effect. Independent follow-up data dilute that excess, so a sample-size increase should be assessed against plausible true effects rather than justified by the selected estimate alone.

Check Your Understanding

Answer each question using the normal-model assumptions stated above. Solutions follow the questions.

If \(\hat{\delta}_I\sim N(\delta,s_I^2)\), what is \(a\) for the selection event \(\hat{\delta}_I\ge c\)?
Why does the expected combined estimate move toward \(\delta\) when independent follow-up information is added?
Does a favorable estimate necessarily cause an effect-based sample-size recalculation to increase \(n\)?
What does the conditional mean among threshold-crossing trials tell you, and what does it not tell you about one particular trial?

Solutions. (1) \(a=(c-\delta)/s_I\), the threshold expressed in standard-error units above the true effect. (2) The new independent estimate has conditional expectation \(\delta\), so the interim estimate’s excess is multiplied by \(I_1/(I_1+I_2)<1\). (3) No. If the assumed effect is larger, the fixed-design formula gives a smaller required sample size; an increase depends on the adaptive rule, not on favorability alone. (4) It quantifies the average optimism induced by selection under the model. A specific trial’s interim estimate may be above or below that conditional average, and its follow-up estimate may move either direction.