Conditional Inference Given The Selected Arm

Why condition on the selected arm?

When several treatment estimates are examined and the largest is selected, its value is not distributed like an estimate chosen in advance. Even if every estimate is unbiased before selection, the estimate of the winner tends to be high because being selected is evidence that its observed value was comparatively large. Conditional inference asks a more specific question: given the selection event that actually occurred, what reference distribution should be used for a test or estimate?

The answer depends on both the probability model and the precise selection rule. Conditioning is not a generic correction that can be applied after the analysis. The rule, including any thresholds and tie handling, must be represented in the model. The method below uses a multivariate normal model with known covariance and a selection event expressed by linear inequalities. Its conditional distributions are exact under those assumptions.

Definition: Conditional inference given a selected arm calibrates a test or likelihood after conditioning on the observed selection event and, where needed, on an observed component of the data. The resulting reference distribution is the distribution of the statistic under the model subject to those conditions; it is generally not the same as its unconditional distribution.

Deriving the conditional reference distribution

Let \(Y\) be a vector of treatment-effect estimates with \(Y\sim N(\mu,\Sigma)\), where \(\Sigma\) is known and positive definite. Suppose the arm-selection rule is the event \(AY\leq b\), a collection of linear inequalities. For example, selecting arm 1 as the largest of three estimates requires \(Y_2-Y_1\leq0\) and \(Y_3-Y_1\leq0\).

To test a scalar parameter \(\eta^\mathsf{T}\mu\), define \(T=\eta^\mathsf{T}Y\), its variance \(v=\eta^\mathsf{T}\Sigma\eta\), and the residual vector \(V=Y-qT\), where \(q=\Sigma\eta/v\). The residual retains the part of the data orthogonal to the tested statistic in the covariance geometry. In particular, \(V\) is uncorrelated with \(T\); because the joint distribution is normal, they are independent.

\operatorname{Cov}(V,T)=\Sigma\eta-qv=0,\qquad T\sim N(\eta^\mathsf{T}\mu,v).

Condition on the observed \(V\). Substituting \(Y=V+qT\) into the selection inequalities gives \(A_iV+(A_iq)T\leq b_i\) for each row \(i\). If \(A_iq>0\), that row places an upper bound on \(T\); if \(A_iq<0\), it places a lower bound. Rows with \(A_iq=0\) either impose no bound on \(T\) or make the observed residual incompatible with the selection event. Taking the tightest bounds gives an interval \([L(V),U(V)]\), possibly with an infinite endpoint.

Before selection, \(T\) is normal and independent of \(V\). After conditioning on \(V\) and selection, it has that normal distribution restricted to the interval. The restriction changes its normalizing constant, which is why an ordinary normal tail probability is generally wrong.

Conditional distribution: Given \(V\) and \(AY\leq b\), \(T\) has a \(N(\eta^\mathsf{T}\mu,v)\) distribution truncated to \([L(V),U(V)]\). This result assumes the stated multivariate normal model, known \(\Sigma\), a correctly specified selection event, and an interval with positive probability under the parameter value being tested.

For a one-sided test of \(H_0:\eta^\mathsf{T}\mu=\theta_0\) against larger values, the conditional p-value for an observed \(t\) is the probability of obtaining \(T\geq t\) under the null, conditional on the observed residual and selection. For \(L\leq t\leq U\), it is

p_{\mathrm{cond}} = \frac{ \Phi\!\left(\frac{U-\theta_0}{\sqrt v}\right) - \Phi\!\left(\frac{t-\theta_0}{\sqrt v}\right) }{ \Phi\!\left(\frac{U-\theta_0}{\sqrt v}\right) - \Phi\!\left(\frac{L-\theta_0}{\sqrt v}\right) }.

Here \(\Phi\) is the standard normal cumulative distribution function, with \(\Phi(-\infty)=0\) and \(\Phi(\infty)=1\). The denominator is the null probability of the allowed interval; the numerator is the null probability of the part of that interval at least as large as the observation. A two-sided conditional test must also specify its tail convention; one common choice is twice the smaller conditional tail probability, capped at 1.

Worked examples

Worked Example: A Threshold-Selected Estimate

Suppose a hypothetical interim estimate \(X\) has distribution \(N(\theta,1)\), and the arm is advanced only if \(X>1\). The observed estimate is \(x=2\). Test \(H_0:\theta=0\) against \(H_a:\theta>0\), conditioning on advancement.

Under the null, \(X\) is standard normal. Given advancement, its reference distribution is standard normal truncated below at 1. Thus the conditional p-value is

P(X\geq2\mid X>1,H_0) = \frac{1-\Phi(2)}{1-\Phi(1)} = \frac{0.02275}{0.15866} \approx 0.1434.

The unconditioned upper-tail probability would be \(1-\Phi(2)=0.02275\). It is not the relevant p-value for this question because it ignores that values at or below 1 could not have led to the observed advancement. The conditional result is not evidence that the null is true; it describes how unusual this estimate is under the null among trials meeting the selection criterion.

Worked Example: Selecting the Larger of Two Correlated Arms

Consider hypothetical estimates \(Y_1,Y_2\) with mean vector \((\theta_1,\theta_2)^\mathsf{T}\) and covariance matrix

\Sigma= \begin{pmatrix} 1&0.5\\ 0.5&1 \end{pmatrix}.

Arm 1 is selected when \(Y_1\geq Y_2\). The observed values are \(y_1=1.8\) and \(y_2=1.0\). Test \(H_0:\theta_1=0\) against \(H_a:\theta_1>0\), using \(\eta=(1,0)^\mathsf{T}\). Then \(T=Y_1\), \(v=1\), and \(q=(1,0.5)^\mathsf{T}\), so the observed residual is \(V=(0,1.0-0.5(1.8))=(0,0.1)^\mathsf{T}\).

Given this residual, write \(Y_2=0.1+0.5T\). The selection condition becomes \(T\geq0.1+0.5T\), or \(T\geq0.2\). Under the null, \(T\sim N(0,1)\); conditioning on the residual and selection therefore gives a standard normal truncated below at 0.2. The conditional p-value is

P(T\geq1.8\mid T\geq0.2,H_0,V) = \frac{1-\Phi(1.8)}{1-\Phi(0.2)} = \frac{0.03593}{0.42074} \approx 0.0854.

The correlation matters: the lower truncation point follows from the covariance-based residual, not simply from treating \(Y_2\) as fixed independently of \(Y_1\). In this model, conditioning on \(V\) makes the mean of \(T\) under the null equal to \(\theta_1=0\), so the nuisance mean \(\theta_2\) does not enter this conditional test.

Worked Example: A Conditional Likelihood Estimate

Return to \(X\sim N(\theta,1)\), selected when \(X>1\), and suppose a hypothetical observed value is \(x=1.5\). The conditional density at \(x\), as a function of \(\theta\), is the ordinary normal density divided by the probability of selection:

L_{\mathrm{cond}}(\theta) = \frac{\phi(x-\theta)}{1-\Phi(1-\theta)},\qquad x>1,

where \(\phi\) is the standard normal density. Differentiating the log likelihood and setting the score to zero gives

\frac{\partial\log L_{\mathrm{cond}}(\theta)}{\partial\theta} = x-\theta-\frac{\phi(1-\theta)}{1-\Phi(1-\theta)} =0.

Substituting \(x=1.5\), the equation to solve is \(1.5-\theta=\phi(1-\theta)/[1-\Phi(1-\theta)]\). Numerical solution gives a conditional maximum-likelihood estimate of approximately \(-0.13\). For example, at \(\theta=-0.13\), the left side is 1.63 and the ratio on the right is about 1.63. The estimate can fall below the selection threshold because the likelihood accounts for the fact that only estimates above 1 were observed for advancement. It is a conditional estimate under this model, not a universal correction for selection.

What conditioning does—and what it does not do

The derivation gives a practical workflow. First write the sampling model and the complete selection rule. Next identify the parameter and statistic to test, and condition on the residual component needed to remove nuisance variation. Then translate the selection inequalities into bounds, and calculate probabilities or likelihoods using the resulting truncated distribution.

For a conditional estimate, the likelihood must include the selection probability in its denominator. Maximizing only the untruncated normal density recovers the ordinary estimate and ignores the condition that generated the observed sample. Conversely, a truncated likelihood is appropriate only when the conditioning event and the distribution are correctly specified. A conditional estimate may differ substantially from the ordinary estimate, and may be unstable when the selection interval has very small probability.

Approach	Reference distribution	What it conditions on	Main caution
Unadjusted inference	Unconditional sampling distribution	No selection event	Can miscalibrate inference for an estimate chosen because it was extreme
Conditional inference	Distribution restricted to the observed selection region	Selection and a specified residual component	Requires the exact rule and model; conditioning can reduce information
Independent confirmation data	Sampling distribution of the separate confirmation estimate	Selection data are not reused for the test	Requires separate data and a design that preserves the intended independence

The independent-confirmation approach can avoid the particular truncation of the reused estimate, but it is not the same estimand or data analysis as conditioning on the selected estimate. Conditional inference directly answers a conditional question for the observed selection path. It does not, by itself, establish that every selected arm has a desired unconditional error rate or interval property.

Assumptions, diagnostics, and common errors

Exactness here depends on a normal sampling model with known covariance, an accurately recorded selection event, and valid conditioning calculations. In a trial, the model must also match the estimand and the way the estimates were obtained. If the selection rule involved interim thresholds, rankings, or tie-breaking, omitting any part changes the truncation region. If \(\Sigma\) is estimated rather than known, the stated normal conditional reference is no longer exact without further justification.

Check the event. Confirm that the observed data satisfy every stated selection inequality and that the derived interval contains the observed \(t\).
Check the denominator. The conditional probability of the selection interval under the null must be positive. A tiny denominator can make the conditional calculation numerically sensitive.
Check nuisance parameters. The projection construction gives a conditional law for \(T\) with mean \(\eta^\mathsf{T}\mu\); it does not automatically remove nuisance parameters from every possible statistic or every non-normal model.
Check the model and covariance. Strong departures from normality, covariance misspecification, or unmodeled adaptive decisions can invalidate the calculated reference distribution.

A common mistake is to say, “The selected estimate is 1.8 standard errors above zero, so use the ordinary normal tail.” That reasoning treats the arm as if it had been specified before seeing the data. The correction is to condition on the actual selection event, derive the allowed region for the statistic, and use its conditional tail probability. Another mistake is to assume conditioning always improves inference: conditioning can discard information, and an incorrectly specified event can make the result less reliable rather than more reliable.

Key takeaway: Selection changes the reference distribution. A conditional test or likelihood is valid only relative to a stated model, a fully specified selection event, and the conditioning variables used in its derivation.

Check Your Understanding

Answer each question, then compare your reasoning with the solution immediately below it.

An estimate \(X\sim N(\theta,1)\) is selected when \(X>0\). Under \(H_0:\theta=0\), the observed value is \(x=1\). What is its one-sided conditional p-value?

Solution: The conditional distribution is standard normal truncated below at 0. Thus \(p=[1-\Phi(1)]/[1-\Phi(0)]=0.15866/0.5\approx0.3173\).
In the projection construction, why are \(T\) and \(V\) independent under a multivariate normal model?

Solution: Their covariance is \(\operatorname{Cov}(V,T)=\Sigma\eta-qv=0\), where \(q=\Sigma\eta/v\). Jointly normal random variables with zero cross-covariance are independent.
Suppose a selection inequality becomes \(T\geq V+0.5T\), and the observed residual is \(V=0.4\). What lower bound on \(T\) does selection impose?

Solution: Subtract \(0.5T\) to obtain \(0.5T\geq0.4\), so \(T\geq0.8\). The conditional reference distribution must be truncated below at 0.8.
Why does the conditional likelihood for a threshold-selected normal estimate include a selection-probability denominator?

Solution: A density conditional on \(X>c\) equals the original density divided by \(P_\theta(X>c)\). Omitting that denominator ignores the fact that only observations meeting the threshold were selected.