Selection Events And Their Sampling Distributions

Why Selection Changes a Sampling Distribution

An interim rule may retain an arm only when its estimated effect exceeds a threshold, or may continue only the arm with the largest interim statistic. Once the rule has been applied, the data under consideration are no longer an unselected sample from the original sampling distribution: they are observations known to satisfy a particular constraint. The distribution conditional on that constraint is the selection-conditional sampling distribution.

This distinction matters even when the original estimator and its standard error were correctly specified. An estimator can be unbiased before selection and have a conditional mean above its target among selected arms. The selection event, rather than a change in the estimator’s formula, creates the shift. This tutorial derives that change for normal models and shows how to represent more general selection rules.

Writing Selection as a Mathematical Event

A rule should first be written as a set of values that lead to a particular decision. For one arm retained when its interim estimate \(\hat{\theta}\) exceeds a threshold \(c\), the event is \(A=\{\hat{\theta}>c\}\). For two arms where arm 1 is selected if its statistic exceeds arm 2’s, \(A=\{Z_1>Z_2\}\). With several constraints, the event is an intersection, such as \(A=\{Z_1>c,\ Z_1>Z_2\}\).

Suppose the interim vector \(Y\) has density \(f_Y(y)\), and the rule selects on event \(A\), with \(P(A)>0\). The conditional density is the original density restricted to the selection region and renormalized:

Here \(\mathbf{1}\{y\in A\}\) equals 1 inside the selection region and 0 outside it. The denominator is essential: it makes the restricted density integrate to 1. This formula applies whether the region is a simple threshold or a more complicated ranking rule. In a multivariate normal model, the original density may be normal, but conditioning on a general region does not usually leave a normal distribution.

For a later statistic \(T\), the corresponding expression is \(f_{T\mid A}(t)=\int f_{T,Y}(t,y)\mathbf{1}\{y\in A\}\,dy/P(A)\). It shows that the effect on \(T\) depends on the joint distribution of \(T\) and the interim data. If they are independent, conditioning on \(A\), which is determined by \(Y\), does not change \(T\)’s distribution. If they are dependent, it generally does.

A Normal Estimate Truncated by a Threshold

Assume a single interim estimate has the model \(\hat{\theta}\sim N(\theta,SE^2)\), where \(\theta\) and \(SE\) are fixed and \(SE\) is known for this calculation. The normal model may be exact under a normal sampling model or an approximation justified by the design. Suppose the rule retains the arm when \(\hat{\theta}>c\). Standardize by setting \(Z=(\hat{\theta}-\theta)/SE\), so \(Z\sim N(0,1)\), and define \(a=(c-\theta)/SE\). Retention is then the event \(Z>a\).

Write \(\phi\) and \(\Phi\) for the standard normal density and cumulative distribution function. The chance of retention is \(1-\Phi(a)\). Integrating the truncated density gives the conditional mean and variance:

The quantity \(\lambda(a)\) is the inverse Mills ratio for this upper-tail truncation. Returning to the estimate, \(E(\hat{\theta}\mid\hat{\theta}>c)=\theta+SE\lambda(a)\); its conditional variance is \(SE^2[1+a\lambda(a)-\lambda(a)^2]\). Thus the retained estimates have a conditional mean above \(\theta\), even though the unselected estimate has mean \(\theta\).

Ranking Arms: The Distribution of a Winner

A ranking rule creates a selection region bounded by comparisons between statistics. Consider two independent interim estimates \(X_1,X_2\), each distributed as \(N(0,\sigma^2)\), and select arm 1 when \(X_1>X_2\). Independence and equal distributions imply \(P(X_1>X_2)=1/2\). Given selection, the density of \(X_1\) is \(f_{X_1\mid X_1>X_2}(x)=2f_{X_1}(x)P(X_2<x)\). For standard normal variables this becomes \(2\phi(x)\Phi(x)\), not \(\phi(x)\). The winner’s distribution is therefore not the original normal distribution.

Its conditional mean can be obtained directly. Define \(D=X_1-X_2\) and \(S=X_1+X_2\). Under the stated model, \(S\) and \(D\) are independent, \(E(S)=0\), and \(D\sim N(0,2\sigma^2)\). Since \(X_1=(S+D)/2\), selection \(X_1>X_2\) is \(D>0\), and \(E(X_1\mid D>0)=\tfrac12 E(D\mid D>0)=\sigma/\sqrt{\pi}\). Thus even if both arms have true effect zero, the selected estimate has a positive conditional mean.

When the Final Statistic Is Correlated With Interim Selection

A final test statistic need not equal the interim statistic. Its conditional distribution still changes if the two are correlated. For a jointly normal pair \((Y,T)\), with means \(\mu_Y,\mu_T\), variances \(\sigma_Y^2,\sigma_T^2\), and correlation \(\rho\), the conditional distribution given the exact value \(Y=y\) is normal: \(T\mid Y=y\sim N(\mu_T+\rho(\sigma_T/\sigma_Y)(y-\mu_Y),\ \sigma_T^2(1-\rho^2))\). When the selection event is \(Y>c\), the conditional distribution \(T\mid Y>c\) averages these conditional distributions over the truncated distribution of \(Y\). It is generally a mixture, not a normal distribution.

Diagnostics, Alternatives, and Limitations

The first diagnostic is to state exactly which data determine selection and which statistic will later be reported or tested. Then check whether the latter shares data with, or is otherwise correlated with, the selection statistic. If it does, an unselected reference distribution needs justification rather than assumption. For a normal model, inspect whether the event is a one-sided threshold, an interval, or a ranking region: each imposes different constraints. With many arms, shared controls, unequal standard errors, or multiple constraints, the joint covariance structure affects the selection probability and conditional distribution.

The derivations above rely on a correctly specified normal sampling model and a prespecified rule. Normal approximations can be inaccurate for small samples or highly discrete outcomes; a threshold that depends on estimated nuisance parameters can also make a simple fixed-threshold calculation inadequate. In those settings, the conditional distribution can be evaluated under an appropriate sampling model, often by numerical integration or simulation that repeats both data generation and selection. Simulation must apply the selection rule in every replicate; simulating only the final statistic after ignoring the rule does not reproduce its conditional distribution.

There are useful contrasts. If selection is independent of the statistic being evaluated, conditioning on selection leaves its distribution unchanged. If the same data determine both selection and the reported statistic, truncation or ranking generally changes the distribution. A joint model that explicitly represents interim and final statistics can describe this dependence, while a model that treats them as independent when they are not will miss the conditional shift. These are distributional distinctions, not by themselves a method for adjusting a confirmatory p-value or confidence interval.

A common mistake is to reason, “The estimate was normal and unbiased before selection, so it is still centered at the true effect among selected arms.” That conclusion ignores the condition. Under a threshold rule, retained observations are precisely those in the upper tail, and under winner selection they are those that outrank competitors. The correction is to define the event, condition on it, and use the resulting distribution for the statistic of interest. Selection-conditioned distributions describe the selected data; they do not alone determine a complete inferential procedure.