Data Carving for Selection-Adjusted Estimation

Why Carve Information?

When an interim comparison selects an arm for further attention, its observed estimate is typically unusually favorable: the selection rule has favored arms with high estimates. Estimating the selected effect as if that estimate had been chosen in advance can therefore exaggerate it. The earlier material on sample splitting and validation estimation motivates two familiar responses: set aside data for estimation before selection, or collect an independent validation sample. Both protect estimation from selection, but they do not use all the information available.

Data carving aims to use more of the data. It distinguishes the information that determines selection from information that remains available for estimation, then reuses the selection information with an adjustment for the event that caused it to be observed in the selected set. The adjustment is not a generic shrinkage factor: it follows from conditioning the likelihood on the actual selection rule.

Definition: Data carving is a post-selection estimation approach that retains information used in selection and additional information not used to make that selection. A selection-adjusted, or conditional, likelihood for the carved data is the joint data likelihood divided by the probability of the observed selection event under the parameter value.

A simple model makes the distinction precise. For one treatment effect \(\theta\), let \(Y\) be the statistic used for selection and \(Z\) be additional information retained for estimation. Suppose \(Y\sim N(\theta,\sigma_Y^2)\) and \(Z\sim N(\theta,\sigma_Z^2)\), independently, with known standard errors \(\sigma_Y\) and \(\sigma_Z\). Selection is the event \(A=\{Y>c\}\), for a prespecified threshold \(c\). After observing \(Y=y\) and \(Z=z\) with \(y>c\), the likelihood conditional on selection is

L_A(\theta;y,z) = \frac{f_\theta(y,z)}{P_\theta(A)} = \frac{ \frac{1}{\sigma_Y}\phi\!\left(\frac{y-\theta}{\sigma_Y}\right) \frac{1}{\sigma_Z}\phi\!\left(\frac{z-\theta}{\sigma_Z}\right) }{ 1-\Phi\!\left(\frac{c-\theta}{\sigma_Y}\right) }.

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The numerator uses both observed components. The denominator accounts for the fact that the analysis is restricted to cases in which \(Y\) crossed the threshold. If \(Z\) is independent of \(Y\), it contributes an ordinary normal likelihood factor; the selection correction applies to the selection statistic.

What the Conditional Likelihood Conditions On

Conditioning on the event \(A\) is not the same as conditioning on the exact observed value \(Y=y\). If we conditioned on \(Y=y\), its density would no longer contribute to estimation of \(\theta\); in this simple model, only \(Z\) would remain informative. Carving instead conditions on the rule that selected the observation, such as \(Y>c\), while using the observed value \(y\) within that selected region. This is why the numerator retains selection-stage information and the denominator corrects its distribution.

The likelihood formula follows directly from the conditional density identity. For any observed data \(D=(Y,Z)\) that imply event \(A\),

f_\theta(D\mid A) = \frac{P_\theta(D\in dD,\ A)}{P_\theta(A)\,dD} = \frac{f_\theta(D)}{P_\theta(A)}.

The second equality holds because the observed data lie in the selection region, so observing \(D\) already implies \(A\). The denominator is parameter-dependent and must generally remain in the likelihood. A term that is constant in \(\theta\) could be dropped when maximizing, but \(P_\theta(A)\) is not such a term.

Worked Example: A Threshold-Selected Estimate and Retained Information

Suppose hypothetical estimates \(Y\) and \(Z\) measure an arm’s treatment effect in outcome units. Let \(Y\sim N(\theta,0.6^2)\) be used for selection, let the independent retained estimate satisfy \(Z\sim N(\theta,0.5^2)\), and select the arm if \(Y>0.5\). The observed values are \(y=1.4\) and \(z=0.8\). Evaluate the conditional-likelihood adjustment at the hypothetical parameter value \(\theta=0.9\).

The selection probability at this parameter value is \(P_{0.9}(Y>0.5)=1-\Phi((0.5-0.9)/0.6)=\Phi(0.6667)\approx0.7475\). The two observed values are inside the selection event because \(1.4>0.5\). Their joint density at \(\theta=0.9\) is approximately

\left\{\frac{\phi((1.4-0.9)/0.6)}{0.6}\right\} \left\{\frac{\phi((0.8-0.9)/0.5)}{0.5}\right\} \approx (0.470)(0.782)=0.3675.

Thus the conditional density is approximately \(0.3675/0.7475=0.4916\), in inverse squared outcome units because it is a joint density for two continuous observations. This calculation is a likelihood evaluation, not an estimate by itself. It shows exactly where carving uses both \(y\) and \(z\), and where the selection probability enters. An independent retained estimate would contribute its density but would not make the selection denominator disappear.

Conditional Maximum Likelihood Under a Threshold Rule

A conditional maximum-likelihood estimate is a value of \(\theta\) that maximizes \(L_A(\theta;y,z)\), or equivalently its log. In the independent, equal-variance case \(Y,Z\sim N(\theta,1)\), with selection \(Y>0\), the conditional log likelihood, up to a constant not depending on \(\theta\), is

\ell_A(\theta) = -\frac{1}{2}(y-\theta)^2 -\frac{1}{2}(z-\theta)^2 -\log\Phi(\theta).

The final term is the log selection probability because \(P_\theta(Y>0)=\Phi(\theta)\). Differentiating gives the score equation

\ell_A'(\theta) = (y-\theta)+(z-\theta)-\frac{\phi(\theta)}{\Phi(\theta)} =0.

The ratio \(\phi(\theta)/\Phi(\theta)\) is the inverse Mills ratio for this lower-truncated selection probability. Its appearance is not an added correction chosen by convention; it is the derivative of the log denominator. In other selection rules, the derivative of the corresponding selection probability replaces it.

Worked Example: Solving a Carved Conditional Score Equation

In a hypothetical two-component analysis, \(Y\sim N(\theta,1)\) is used to select an arm when \(Y>0\), and an independent estimate \(Z\sim N(\theta,1)\) is retained. The observed values are \(y=1.0\) and \(z=0.5\). Find the conditional maximum-likelihood estimate.

The score equation is \(1.5-2\theta-\phi(\theta)/\Phi(\theta)=0\). At \(\theta=0.494\), the inverse Mills ratio is approximately \(\phi(0.494)/\Phi(0.494)\approx0.5122483\). The score is therefore \(1.5-2(0.494)-0.5122483=0.5120-0.5122483\approx-0.0002483\). Numerically solving the score equation gives \(\hat{\theta}\approx0.4938\), or \(0.494\) outcome units to three decimal places.

For comparison, ignoring selection and maximizing the ordinary joint likelihood gives \((1.0+0.5)/2=0.75\) outcome units. That estimate treats \(Y\) as if it had not been selected for being positive. The conditional estimate is lower because the observed \(Y\) is interpreted in light of the threshold rule. Neither value is automatically correct under every design: the conditional estimate relies on the stated model and the actual selection event.

Carving When the Selected Arm Is the Largest

In a multi-arm setting, selection may choose the arm with the largest interim estimate rather than apply a fixed threshold. Suppose there are \(K\) independent selection estimates \(Y_i\sim N(\theta_i,\sigma_i^2)\), and arm \(j\) is selected when its estimate is the largest. With continuous distributions, ties have probability zero. The selection event is \(A_j=\{Y_j>Y_k\text{ for every }k\ne j\}\), with probability

P_{\boldsymbol{\theta}}(A_j) = \int_{-\infty}^{\infty} f_j(t)\prod_{k\ne j}F_k(t)\,dt,

where \(f_i\) and \(F_i\) are the normal density and cumulative distribution function for \(Y_i\). This expression integrates over the possible value \(t\) of the winning estimate and requires all competitors to fall below it. If \(Z_i\) are independent retained estimates, the likelihood for all observed selection and retained data, conditional on selecting \(j\), is

L_{A_j}(\boldsymbol{\theta};\mathbf{y},\mathbf{z}) = \frac{\prod_{i=1}^{K} f_i(y_i)\,g_i(z_i)} {P_{\boldsymbol{\theta}}(A_j)}.

Here \(g_i\) is the density of retained estimate \(Z_i\). This form assumes the retained components are independent of the selection estimates and that the full set of \(Y_i\) and \(Z_i\) is observed. If estimates share a control group, or if the components are otherwise correlated, the product densities are not valid: use the appropriate joint density in the numerator and compute the selection probability from the corresponding joint distribution.

Worked Example: Accounting for a Two-Arm Winner

Consider two hypothetical arms with independent selection estimates \(Y_1\sim N(\theta_1,1)\) and \(Y_2\sim N(\theta_2,1)\). The arm with the larger estimate is selected. The observed selection estimates are \(y_1=1.2\) and \(y_2=0.7\), so arm 1 is selected. Independent retained estimates are \(z_1=0.8\) and \(z_2=0.5\). Evaluate the selection probability at the illustrative parameter values \(\theta_1=0.4\) and \(\theta_2=0.1\).

The difference \(Y_1-Y_2\) has a normal distribution with mean \(0.4-0.1=0.3\) and variance \(1+1=2\). Therefore \(P_{\boldsymbol{\theta}}(A_1)=P(Y_1-Y_2>0)=\Phi(0.3/\sqrt{2})=\Phi(0.2121)\approx0.5840\). The conditional likelihood at these parameter values is the joint density of the four observed estimates divided by \(0.5840\):

L_{A_1} = \frac{f_1(1.2)f_2(0.7)g_1(0.8)g_2(0.5)} {0.5840}.

This is a likelihood evaluation, not an optimized estimate of either treatment effect. The denominator reflects competition between the arms: an arm 1 estimate is observed in the selected analysis partly because it exceeded arm 2’s estimate. Retaining both arms’ observed selection statistics and the independent estimates can provide more information than discarding the selection-stage data, while the denominator prevents the winning event from being ignored.

Assumptions, Diagnostics, and Alternatives

The conditional likelihood is only as credible as the model and event used to construct it. Before applying it, check that the analysis represents the design and data actually observed.

Correct selection event. Specify the rule that was applied, including thresholds, ranking, eligibility restrictions, and any tie-breaking rule. If the event in the denominator differs from the actual rule, the adjustment does not describe the data collection process.
Correct joint distribution. Normal estimates with known variances provide the formulas above. In a MAMS trial, shared controls or other dependencies can induce covariance; assuming independence in that case understates or misrepresents the selection probability.
Correct separation of information. The simple factorizations require independence of \(Y\) and \(Z\). If carving creates correlated components, use their joint model rather than multiplying independent densities. Independence should follow from the data construction and model, not be assumed merely because the data are labelled “selection” and “retained.”
Known selection mechanism and available data. The likelihood must include the information needed to identify which event occurred. Unrecorded changes to the rule or missing competing-arm estimates can make the proposed conditional likelihood inappropriate.

Useful diagnostics include checking that the observed values satisfy the stated selection event, comparing the modeled selection probability with the design’s behavior under plausible parameter values, and verifying covariance assumptions from the analysis plan or data construction. Very small or very large selection probabilities can make the conditional likelihood sensitive to model details. Numerical calculations should also be checked over a range of parameter values: the computed conditional log likelihood should include the negative log selection probability, not its positive.

The main alternatives make different information-use choices. Sample splitting discards the selection portion for estimation and uses only the reserved part; this can simplify post-selection reasoning but may lose precision. An independent validation stage supplies new data unaffected by the earlier selection, but requires additional observations. Data carving reuses selection data through the conditional likelihood and can be more efficient when its joint model is credible. A naive, unadjusted estimate is simpler but fails to account for selection and may be optimistically biased. Carving is not a universal guarantee of lower uncertainty: dependence, model misspecification, a poorly captured selection rule, or weak information in the retained component can undermine its benefits.

Key takeaway: Carving does not make selected data behave as though no selection occurred. It retains the observed information and adjusts its likelihood by the probability of the selection event under each candidate parameter value.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For \(Y\sim N(\theta,1)\), an arm is selected when \(Y>0\), and an independent \(Z\sim N(\theta,1)\) is retained. What is the conditional likelihood for observed \(y>0,z\)?
Why is conditioning on the event \(Y>0\) different from conditioning on the exact value \(Y=y\)?
Two independent arm estimates have unit variances and means \(\theta_1,\theta_2\). Express the probability that arm 2 has the larger estimate.
Name one assumption that fails if the selection estimates share a control group, and state the needed correction.

Solutions. (1) The conditional likelihood is \(\{\phi(y-\theta)\phi(z-\theta)\}/\Phi(\theta)\), because \(P_\theta(Y>0)=\Phi(\theta)\). (2) Conditioning on the event retains the density contribution from the observed \(y\) within the selected region and divides by the event probability. Conditioning on the exact \(Y=y\) removes that density contribution; in the independent model, estimation then uses \(Z\) alone. (3) The difference \(Y_2-Y_1\) is normal with mean \(\theta_2-\theta_1\) and variance \(2\), so the probability is \(\Phi((\theta_2-\theta_1)/\sqrt{2})\). (4) Independence of the arm estimates fails. Use their joint density in the likelihood and calculate the selection probability from their joint distribution, including the covariance induced by the shared control.