Sensitivity to Outcome-Model Assumptions

Why Outcome-Model Assumptions Matter After Selection

A treatment arm is often carried forward because its interim result looks promising. The estimate reported for that arm is then conditional on an event: it passed the selection rule. Even if the selection rule is correctly specified, an adjustment for selection depends on the probability model used for the outcomes and for the estimates across stages. A model that gives the wrong tail probabilities or dependence structure can give a misleading adjustment.

This tutorial focuses on sensitivity to those outcome-model assumptions, not on changing the selection rule. The aim is to ask whether the post-selection estimate is materially different under reasonable alternative models. Sensitivity analysis does not identify which model is true; it makes explicit how much the conclusion relies on a particular model.

For clarity, write \(\theta\) for the treatment effect of interest and \(\hat{\theta}\) for an estimator. In a multi-stage design, \(\hat{\theta}_k\) denotes the estimate at stage \(k\). The selection event \(S\) is determined by the design’s specified rule. A selection-aware estimate must account for the fact that the observed estimate is available after \(S\), rather than treating it as an unconditional draw. Earlier material on selective confidence intervals and selection-aware estimation develops procedures for this conditioning; here the emphasis is on how their inputs affect the result.

Assumptions to Vary and Hold Fixed

A useful sensitivity analysis distinguishes three sources of model uncertainty. The first is the marginal outcome or estimator distribution: for example, whether a normal approximation adequately describes the estimator’s tails. The second is treatment-effect heterogeneity: whether individual treatment effects vary around a population mean, and how that variation is represented. The third is cross-stage dependence: how much stage-specific estimates co-vary, perhaps because of shared controls, overlapping participants, or a common source of outcome variation.

The assumptions required depend on the calculation. For the normal-theory derivation below, suppose that before selection \(\hat{\theta}\sim N(\theta,s^2)\), where \(s\) is treated as known, and selection occurs when the standardized estimate \(\hat{\theta}/s\) exceeds a fixed threshold \(c\). Thus the model concerns the estimate itself; its standardized estimation error is \(U=(\hat{\theta}-\theta)/s\). These are distinct quantities. Under this model \(U\sim N(0,1)\), and selection is equivalent to \(U>c-\theta/s\).

In a real analysis, \(s\) may itself be estimated, outcome distributions may be nonnormal, and the selection event may involve multiple arms or stages. Those complications require a model appropriate to the design. The simple calculation that follows isolates the basic mechanism and should not be treated as a universal correction.

Normal-Model Derivation and Example

Let \(\delta=\theta/s\) and \(a=c-\delta\). Selection is \(U>a\). For a standard normal variable, the conditional mean above a threshold is \(E(U\mid U>a)=\phi(a)/\{1-\Phi(a)\}\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. This ratio is often called the inverse Mills ratio.

The derivation follows directly from the truncated normal density. Since \(\hat{\theta}=\theta+sU\), it is enough to calculate the conditional expectation of \(U\). The standard normal density satisfies \(\phi'(u)=-u\phi(u)\), so \(\int_a^\infty u\phi(u)\,du=\phi(a)\). The probability of selection is \(P(U>a)=1-\Phi(a)\). Dividing the integral of \(u\) over the selected region by that probability gives the stated conditional mean. Therefore the conditional selection bias is \(s\phi(a)/\{1-\Phi(a)\}\), which is positive for an upper-tail selection rule.

Changing the Estimator-Error Distribution

The normal model supplies a convenient formula, but the relevant feature for selection is often the probability and magnitude of values in the selected tail. Two distributions with the same mean and variance can have different tail behavior and therefore different conditional means after selection. A normal approximation that fits the center of the estimator distribution may still be inaccurate in the region that determines selection.

The next example compares two stylized models for an estimator error \(U\): a standard normal distribution and a mean-zero Laplace distribution with variance one. This is a sensitivity calculation, not a claim that a particular individual-level outcome must follow either distribution. Both models have \(E(U)=0\) and \(\operatorname{Var}(U)=1\). For the Laplace model, the density is \(f(u)=\exp(-|u|/b)/(2b)\), with \(b=1/\sqrt{2}\), which gives variance \(2b^2=1\).

For more complicated outcome models, the conditional expectation may not have a closed form. The principle remains the same: calculate or approximate the distribution of the estimator under each specified model, apply the same selection event, and compare the resulting selected estimates. Any approximation must represent the selection event as well as the unselected estimator distribution.

Heterogeneity and Cross-Stage Dependence

Treatment-effect heterogeneity concerns variation in individual effects, not merely a change in the average effect. If the target is the average treatment effect, a model that assumes a common effect for everyone may understate uncertainty when individual effects vary. Depending on the design and estimand, heterogeneity can also affect the distribution’s shape. The sensitivity analysis should state whether the target is a population mean effect or an effect for a particular subgroup; these are not interchangeable.

Across stages, correlation matters because selection on one stage changes the distribution of estimates correlated with it. To see this, suppose two stage-specific estimates have the same standard error \(s\), the same mean \(\theta\), and jointly normal standardized errors \(U_1,U_2\) with correlation \(\rho\). Select using stage 1, \(\hat{\theta}_1/s>c\), and consider the equal-weight estimate \(\bar{\theta}=(\hat{\theta}_1+\hat{\theta}_2)/2\). These assumptions are deliberately simple: the stage estimates have equal variances, their means are the same, and the joint distribution is bivariate normal.

Under these assumptions, \(E(U_1\mid U_1>a)=\lambda(a)\), where \(\lambda(a)=\phi(a)/\{1-\Phi(a)\}\). For a bivariate normal pair, \(E(U_2\mid U_1)=\rho U_1\), so conditioning on selection gives \(E(U_2\mid U_1>a)=\rho\lambda(a)\). Therefore

This result shows why dependence cannot be ignored when combining stage information after selection. If the stages are independent, the second-stage error has conditional mean zero; positive correlation makes its conditional mean positive as well. A dependence assumption that is too small can therefore understate conditional bias in this setting.

Conducting and Interpreting a Sensitivity Analysis

Useful diagnostics include checking whether the assumed distribution has plausible tail behavior, whether observed stage estimates and their covariance are compatible with the dependence model, and whether heterogeneity assumptions are consistent with available outcome information. These checks may be limited when few arms or stages are observed. Instability across reasonable model choices is itself important: it signals that a single model-based corrected estimate should not be presented as assumption-free.

These approaches address different aspects of uncertainty and are not substitutes in every setting. In particular, changing only a standard-error formula does not account for selection if the estimate’s conditional distribution is still treated as unconditional.

Common Pitfall and Limitations

Common mistake: “The model fits the unselected data reasonably well, so the selection-adjusted estimate must be reliable.” This reasoning can fail because selection depends disproportionately on the part of the distribution near and beyond the selection boundary. A model may fit the center while misrepresenting that tail, or may fit each stage marginally while getting their correlation wrong.

Correction: Evaluate assumptions in relation to the selection event. Compare plausible tail shapes, heterogeneity levels, and cross-stage correlations while holding the selection rule and estimation procedure fixed. State the range of resulting estimates and identify which assumptions drive it. Sensitivity analysis is limited by the alternatives considered; it cannot protect against an unexamined model failure, and sparse data may not support precise estimation of complex dependence or heterogeneity.