Confidence Intervals For Selected Marker Effects

Why selection changes interval coverage

A confidence interval for a marker effect is often calculated after looking across many genes and reporting the gene with the largest observed difference between clusters. That order matters. Even when every estimate is unbiased before selection, random errors make some estimates unusually large; choosing the largest tends to choose one with a positive error. An ordinary interval may therefore look more precise, or more decisive, than its coverage warrants.

The issue is not that confidence intervals are invalid in general. It is that a pointwise interval is calibrated for a specified effect, not automatically for an effect selected because its estimate was extreme. A selection-adjusted interval must match the selection rule and state its coverage guarantee. Two useful approaches are simultaneous intervals, which protect against selection within a prespecified family, and conditional intervals, which condition on a specified selection event.

For concreteness, an effect might be a prespecified difference in mean expression score between two clusters, expressed in score units. The sampling model below concerns that estimated contrast, not the individual cells as independent experimental replicates. Its standard error must come from a design and analysis that appropriately represent the biological sampling and dependence. A normal approximation is being assumed for the estimator; if that approximation or the standard error is unreliable, interval calibration will also be unreliable.

Pointwise and simultaneous intervals

Suppose a family of \(m\) candidate genes is specified, with estimates satisfying \(\hat{\theta}_j\) approximately normal with mean \(\theta_j\) and known or suitably estimated standard error \(SE_j\). For a gene fixed in advance, an ordinary two-sided interval is

where \(z_q\) is the \(q\) quantile of the standard normal distribution. This interval has approximate pointwise coverage \(1-\alpha\) under the normal model. It does not generally retain that coverage for the gene selected as the largest observed difference.

A direct protection against this problem is to construct intervals that cover all \(m\) candidate effects simultaneously. The Bonferroni interval for each gene is

These intervals are wider because their critical value accounts for the number of candidates. Their key guarantee does not require independent gene estimates. It does require that the family being protected is specified and that each gene-wise interval has its stated marginal error bound under the model.

This guarantee is an unconditional, familywise guarantee: over repeated samples, the probability that any interval in the family misses its corresponding effect is at most \(\alpha\). It implies at least \(1-\alpha\) unconditional coverage for the reported interval, even if the gene is selected after examining all estimates. It does not promise \(1-\alpha\) coverage conditional on the particular event that a particular gene was selected.

Conditional intervals after threshold selection

A different adjustment is possible when the selection event is precisely described. Consider a single estimate \(Y\sim N(\theta,s^2)\), where \(s\) is known, and suppose it is reported only when \(Y>c\), for a fixed threshold \(c\). Conditional on passing the threshold, \(Y\) has a truncated normal distribution. Its conditional cumulative distribution function at an observed value \(y>c\) is

where \(\Phi\) is the standard normal cumulative distribution function. Under the assumed model and selection rule, \(F_{\theta,c}(Y)\), conditional on \(Y>c\), is uniformly distributed between 0 and 1. Inverting this conditional pivot gives a \(1-\alpha\) confidence set: retain values of \(\theta\) satisfying \(\alpha/2\le F_{\theta,c}(y)\le1-\alpha/2\). The endpoints are found by solving the two boundary equations. This construction aims for coverage conditional on passing the threshold, rather than only unconditional coverage.

Matching the interval to the selection rule

The threshold calculation is not a general correction for selecting the largest gene. If a gene is chosen because its estimate exceeds a threshold and is larger than every other candidate estimate, the event includes all those comparisons. Conditioning only on \(Y>c\) omits part of the selection rule and does not generally give conditional coverage for the actual winner. For a maximum, the joint selection event and the joint distribution of the estimates matter. The earlier material on selective inference and truncated-normal tests develops those event-based ideas; here the practical point is to avoid treating a simpler event as if it were the real one.

When the candidate family is fixed and the goal is a straightforward guarantee that remains valid after choosing a gene, simultaneous intervals are often easier to audit. The Bonferroni construction can be conservative, especially when many genes are tested or their estimates are strongly dependent. More specialized simultaneous procedures may be narrower if their assumptions are justified. A conditional selective interval can be more tailored to the exact selection event, but it requires a correctly specified selection rule and model, and may require accounting for nuisance parameters or dependence.

The scope of the family is crucial. If genes were filtered, clusters were selected, or contrasts were chosen using the same data, a family consisting only of the final reported genes may omit earlier selection steps. A simultaneous guarantee applies to the family and estimates actually covered by its assumptions; it does not automatically repair unmodeled data-driven filtering or clustering. Independent validation data or a valid sample-splitting design can separate selection from estimation, but the held-out estimates still need a defensible sampling model and standard errors.

Diagnostics and common mistakes

Before interpreting a selected marker interval, make the selection process explicit. Identify the candidate genes, the contrast, whether selection used the same estimates being reported, and whether the rule was a threshold, a maximum, or a sequence of filtering and clustering decisions. Then check that the interval’s model and coverage statement correspond to that process.

• Using a pointwise interval for a winner. The fact that a 95% interval is valid for each fixed gene does not make it valid at 95% after selecting the largest observed difference.
• Calling Bonferroni conditional coverage. The simultaneous procedure protects the selected interval through an unconditional familywise event. It does not generally provide 95% coverage conditional on a specific gene winning.
• Conditioning on the wrong event. A threshold-only truncated-normal interval does not account for selection as the maximum among many genes unless the selection event truly is just that threshold.
• Ignoring an incomplete family. If the candidate genes were screened using the same data, adjusting only for the final shortlist may fail to cover the full selection process.
• Treating a narrow interval as proof of reliable measurement. Selection adjustment addresses selection-induced coverage, not a misspecified model, underestimated standard error, unmodeled dependence, or an unstable contrast.

A useful alternative is to estimate a marker effect using data not used to select it. Such a design can support ordinary intervals when the held-out estimator and its standard error meet their assumptions. It may sacrifice precision, and it does not eliminate the need to define the target effect and sampling unit correctly. Selection-adjusted intervals and independent estimation solve related but distinct problems.