Limits of the Jackknife for Selected-Arm Estimates

Why the Ordinary Jackknife Can Mislead After Selection

In a multi-arm, multi-stage trial, the reported treatment estimate can depend on a sequence of decisions: an arm may be retained or dropped at an interim analysis, and one or more arms may be selected for reporting. The ordinary leave-one-out jackknife estimates bias by repeatedly removing one observation and recalculating the statistic. That procedure is most informative when removing one observation makes only a small, smooth change to the statistic. A selection rule can violate that condition: a small data change may switch the selected arm or change whether an arm is reported at all.

The issue is not that a jackknife calculation is always impossible after selection. It is that its usual bias interpretation can be unreliable when leave-one-out samples cross selection boundaries. Even when every replicate produces a number, some replicates may estimate different arms’ effects. Their average then need not describe bias for one fixed treatment effect.

For a MAMS application, the relevant statistic must be specified before computing these quantities. If it is the treatment-effect estimate for the arm selected by a prespecified rule, each deletion replicate should rerun that rule. Holding the observed selection fixed instead answers a different, conditional question: how the estimate changes when an observation is removed, given the selected arm. Neither calculation repairs an unstable selection boundary; they simply describe different procedures.

Why Smoothness Matters

A short derivation shows why the max of two estimates is problematic near a tie. Suppose two arms have independent interim estimates \(A\) and \(B\), each distributed as \(N(\mu,\sigma^2/m)\), where \(m\) is the number of observations contributing to each estimate. The reported estimate for the arm with the larger interim estimate is \(\max(A,B)\). The identity

gives the expected selected estimate directly. Since \(A-B\sim N(0,2\sigma^2/m)\), and a mean-zero normal variable with variance \(v\) has expected absolute value \(\sqrt{2v/\pi}\),

Thus, when the true arm effects are equal, selecting the larger noisy estimate creates a positive bias of order \(m^{-1/2}\). For many smooth statistics, the ordinary jackknife is motivated by small, gradual changes in the statistic under one-observation perturbations; it is not a general correction for a boundary effect of this kind. At a tie, the function \(\max(A,B)\) has a kink: an arbitrarily small change can change which arm determines the reported estimate. A drop rule has a related boundary, where a small change can change the set of arms still under study.

This calculation describes a simple selection rule, not every MAMS design. Multi-stage designs can have several boundaries, and their estimates may use information from different stages. The earlier material on bootstrap failure at selection boundaries considers the same broad difficulty for a different resampling method; here the point is specifically that leave-one-out perturbations can also cross a boundary and make an ordinary jackknife bias estimate unstable.

When a Jackknife Replicate Is Not Defined

A separate problem occurs when a deletion changes a retain/drop decision and the selected-arm estimator is defined only for arms that remain in the trial. In that case, a replicate may not be a noisy version of the original estimate: it may have no estimate at all under the stated reporting rule. Assigning an arbitrary number to that missing replicate does not solve the problem; it changes the statistic being jackknifed.

When Deletions Are Informative—and When They Are Not

The jackknife is not automatically useless after a selection decision. If arm selection is made using interim data and a separate, independent validation sample is used to estimate the chosen arm’s effect, then deleting observations only from that validation sample leaves the selection fixed. The resulting jackknife describes variation in the validation estimator conditional on the selected arm. It does not estimate the full procedure’s selection bias, because the selection data and decision were not perturbed.

Diagnostics, Alternatives, and Limitations

Before interpreting an ordinary jackknife correction, check what the deletion replicates actually do. Record the selected arm or retained set for each replicate, not only the resulting estimate. If many deletions switch arms, or if an estimate changes sharply when the selected arm changes, the average may mix distinct effects. Also record each replicate’s distance from relevant selection boundaries. A full-sample result just beyond a boundary, combined with leave-one-out results on both sides, is direct evidence that the result is locally sensitive to individual observations.

Jackknife pseudo-values can expose extreme contributions:

Large or negative pseudo-values deserve investigation, especially when they coincide with changes in the selected arm. They do not establish that the jackknife is invalid in every setting, but they help reveal how strongly the correction depends on particular deletions. If deletion can make the estimator undefined, that is a more basic failure: the stated statistic does not supply all the replicates required by the ordinary formula.

The main alternatives answer different questions. A separate validation sample can estimate an effect conditional on the arm chosen by independent interim data. Sample splitting likewise separates selection information from estimation information, at the cost of using less data for each task. Selection-aware estimation instead models or conditions on the selection rule. These approaches require their own design and modeling assumptions; they are not interchangeable corrections to a jackknife number. The ordinary jackknife may still be useful as a local sensitivity check when selection is stable and all replicates target the same estimand, but a small correction does not prove that selection bias is absent.