Conditioning on a Selection Event

Why Selection Changes the Reference Distribution

In an adaptive trial, interim data can determine which population, treatment, endpoint, or hypothesis proceeds to the final analysis. Once a statistic has helped determine what will be analyzed, its distribution among the analyses that proceed need not be its original null distribution. The relevant question is not only, “What values can this statistic take under the null?” It is also, “What values can it take under the null given that the selection rule chose it?”

This distinction matters for interpretation. A p-value calculated from an unselected reference distribution can appear small because the statistic was large enough to pass a screen, even when such a value is common among selected statistics under the null. The distribution after selection depends on the rule that selected the statistic and on how that statistic is related to the data used by the rule.

The conditional distribution is defined only when \(P(S\mid H_0)>0\). In a continuous model, a useful general expression for the conditional density is \(f_{T\mid S,H_0}(t)=f_{T\mid H_0}(t)P(S\mid T=t,H_0)/P(S\mid H_0)\). The factor \(P(S\mid T=t,H_0)\) describes how selection weights different values of the statistic. If selection is determined by \(T\) itself, this factor can sharply favor large values. If selection depends only on independent data, it does not change the distribution of \(T\).

A Threshold Screen on the Same Statistic

Suppose a one-sided test statistic \(Z\) has the standard normal distribution under the null, \(Z\sim N(0,1)\), and a candidate proceeds only when \(Z>c\). Assume \(c\) is fixed in advance, the null model is correct, and the statistic is continuous. The selection event is \(S=\{Z>c\}\). For a selected observed value \(z\ge c\), the conditional upper-tail probability is

The numerator is the probability of a value at least as large as the observation; because \(z\ge c\), every such value also passes the screen. The denominator is the probability of passing the screen at all. Dividing by it expresses the tail probability on the selected subset. By contrast, the unselected upper-tail p-value is \(1-\Phi(z)\). The two answer different questions.

Deriving the Conditional Distribution

The threshold result follows directly from the definition of conditional probability. For any \(z\ge c\), the event \(\{Z\ge z\}\) is contained in \(\{Z>c\}\), apart from a boundary of probability zero. Therefore their intersection has probability \(P(Z\ge z\mid H_0)\). Dividing by the probability of selection gives the displayed conditional tail formula.

The same argument gives the conditional cumulative distribution function. For \(z\le c\), a selected statistic cannot be at or below \(z\), so the conditional CDF is zero. For \(z>c\), subtract the probability of exceeding \(z\) from the probability of passing the screen:

This is a truncated normal distribution: values at or below the threshold have been removed and the remaining probabilities rescaled to sum to one. It is not a standard normal distribution. In particular, treating the selected \(Z\) as though it were an unselected standard normal statistic uses the wrong reference distribution for a conditional calculation.

Selection of the Largest Candidate

Selection can also favor a statistic because it is the largest among several candidates. Consider \(K\) independent statistics \(Z_1,\ldots,Z_K\), each standard normal under a joint null, and let \(M=\max_j Z_j\). Independence is important for the following formula. For any \(m\),

Consequently, \(P(M\ge m\mid H_0)=1-\Phi(m)^K\). If selection means that the maximum exceeds a threshold \(a\), then, for \(m\ge a\), \(P(M\ge m\mid M>a,H_0)=[1-\Phi(m)^K]/[1-\Phi(a)^K]\). The selected maximum is not distributed like one prespecified \(Z_j\). The selection rule, including the number of candidates, is part of the null distribution.

When Selection Does Not Change the Test Statistic

Selection does not automatically distort every later statistic. If the screen uses data independent of the data used to calculate the final statistic, the final statistic’s distribution can remain unchanged under the null. Formally, if \(T\) and the selection event \(S\) are independent under \(H_0\), then \(P(T\ge t\mid S,H_0)=P(T\ge t\mid H_0)\). This is a conditional probability identity, not a claim that independence is always plausible in a trial.

Diagnostics, Alternatives, and Limits

Before interpreting a p-value after an adaptive choice, specify the selection event precisely. Ask which hypotheses, populations, endpoints, or statistics could have been selected; what data determined the choice; whether the final statistic reuses those data; and whether multiple candidates were compared. Then identify the null distribution jointly with that rule. A derivation that conditions on a threshold but omits the fact that the largest of several candidates was chosen is not conditioning on the actual selection process.

A useful alternative is to base inference on independent confirmatory data, when that independence and the analysis plan are defensible. Another approach is to calculate probabilities using the joint null distribution and the selection rule, as in the maximum example. A prespecified analysis that accounts for the full data-dependent procedure can also be used. These approaches are not interchangeable by default: each relies on its own model and design assumptions.

Conditioning changes the question, not necessarily the trial’s unconditional error probability. For example, the conditional probability of rejection among selected null cases can differ greatly from the probability of rejection across all null cases. They are related by \(P(\text{reject},S\mid H_0)=P(\text{reject}\mid S,H_0)P(S\mid H_0)\). A conditional probability alone therefore does not determine the unconditional probability. Conversely, a marginal error calculation does not automatically establish a desired conditional guarantee.

The calculations above also have limits. The standard-normal formulas rely on the specified null model, continuity, and (for the maximum formula) independence. With correlated statistics, the joint dependence must be included. With discrete statistics, conditional tail probabilities must be computed from the actual probability mass function; ties can affect the attainable levels. In practice, model misspecification, unrecorded selection steps, or reuse of data can invalidate a claimed conditional distribution.

A common mistake is: “The observed statistic is 2.000, and its ordinary standard normal p-value is 0.0228, so it is equally unusual among the cases selected by a rule requiring \(Z>1.645\).” The correction is that 0.0228 is an unselected tail probability. Under the stated screen, the conditional tail probability is about 0.455. The ordinary value is not intrinsically wrong; it answers a different question and is inappropriate if the intended reference distribution is conditional on passing that screen.