Tutorials › Causal Inference › Honest Sample Splitting in Causal Trees

Causal Learning and Policy Decisions · Tutorial 957 of 972

Honest Sample Splitting in Causal Trees

See how to select a causal-tree partition on one sample and estimate its leaf-specific treatment effects on a separate sample.

Advanced 10 min read

What You'll Learn

  • Distinguish the structure-selection sample from the leaf-estimation sample
  • Derive a within-leaf difference-in-means estimator under randomized treatment
  • Explain how independence from tree selection limits adaptive bias
  • Calculate leaf estimates from a hypothetical split-sample dataset
  • Identify why pooled treated-control differences need not average leaf effects
  • Recognize why honesty alone does not remove confounding or guarantee precision

Why Separate Tree Selection From Effect Estimation?

A causal tree divides people into leaves so that treatment effects can vary across groups defined by covariates. The difficulty is that a tree is usually selected by searching over many possible splits. If the same outcomes both guide that search and supply the reported leaf effects, a split can look especially promising partly because of random noise in those outcomes. The resulting leaf estimates may be adaptively biased: they are entangled with the process that selected the leaves.

Honest sample splitting addresses this problem by assigning distinct roles to two samples. The structure-selection sample chooses the tree; the estimation sample calculates treatment effects within the resulting leaves. The estimation outcomes therefore do not determine which partition is reported. This separation reduces bias from adaptive tree selection, but it does not by itself establish causal identification, ensure adequate sample sizes, or make estimates precise.

Definition: Let \(S\) denote a structure-selection sample and \(E\) an independent estimation sample. A causal tree is honest when its partition \(\hat{\Pi}\) is selected using \(S\), while treatment effects for its leaves are estimated using \(E\), without using the estimation outcomes to select or revise the partition. A leaf \(L\) is one of the groups defined by the selected partition.

The Leaf Effect and Its Estimator

Let \(A_i\in\{0,1\}\) indicate whether unit \(i\) receives treatment. Write \(Y_i(1)\) and \(Y_i(0)\) for that unit’s potential outcomes under treatment and control. The leaf-specific average treatment effect in a target population is

$$ \tau(L)=E[Y(1)-Y(0)\mid X\in L], $$

where \(X\) denotes covariates used to assign units to leaves. A direct estimator in the estimation sample is the difference between the treatment and control outcome means within that leaf:

$$ \hat{\tau}(L)=\bar{Y}_{1L}-\bar{Y}_{0L}. $$

Here \(\bar{Y}_{1L}\) is the mean observed outcome among estimation-sample units in \(L\) with \(A=1\), and \(\bar{Y}_{0L}\) is the corresponding mean among units with \(A=0\). Both groups must be represented for this estimator to be defined.

This difference has a causal interpretation under appropriate identification conditions. For the derivation below, assume consistency (the observed outcome is \(Y_i(A_i)\)), no interference between units, and randomized treatment within the leaf. In particular, treatment assignment must be independent of the potential outcomes, with a positive probability of receiving either arm. Also require that, conditional on the arm counts within each leaf, every subset of the relevant size is equally likely to be assigned to that arm. A randomized trial with treatment assigned independently at a common probability is a simple setting satisfying these conditions. In observational data, a raw difference in means generally needs additional adjustment and assumptions, such as conditional exchangeability and positivity; splitting the sample does not supply them.

Why the Estimation Step Is Unbiased Under Randomization

First imagine the estimation sample’s units, covariates, and potential outcomes are fixed. Once the structure sample has selected the partition, each leaf is fixed too. Under randomized assignment, and conditional on the number assigned to each arm within a leaf, the treated units form a random subset of that leaf and the controls form the complementary subset. Each arm’s mean outcome is therefore an unbiased estimate of the corresponding mean potential outcome in the leaf.

To see the implication, write \(N_L\) for the number of estimation-sample units in leaf \(L\), and let \(\bar{Y}_L(1)\) and \(\bar{Y}_L(0)\) be the averages of their potential outcomes. Randomization gives

$$ E[\bar{Y}_{1L}\mid S,\{X_i,Y_i(1),Y_i(0)\}_{i\in E},n_{1L}] =\bar{Y}_L(1) $$

and likewise \(E[\bar{Y}_{0L}\mid S,\{X_i,Y_i(1),Y_i(0)\}_{i\in E},n_{0L}]=\bar{Y}_L(0)\). Subtracting the two expectations yields

$$ E[\hat{\tau}(L)\mid S,\{X_i,Y_i(1),Y_i(0)\}_{i\in E},n_{1L},n_{0L}] =\bar{Y}_L(1)-\bar{Y}_L(0) =\frac{1}{N_L}\sum_{i\in E:X_i\in L}\{Y_i(1)-Y_i(0)\}. $$

Thus the estimator is design-unbiased for the finite-sample average treatment effect among the estimation-sample units in that leaf, under the stated randomization conditions and provided both arm counts are positive. The key role of honesty is that conditioning on the selected tree does not also condition on the estimation outcomes having helped produce that tree. If the estimation sample is drawn independently from a target population, the leaf estimate can also be used to learn about that population’s leaf effect, subject to sampling variability and the relevant sampling assumptions.

Key distinction: Sample splitting protects the estimation step from outcome-driven partition selection. It does not turn an observational comparison into a causal effect, and it does not guarantee that a selected leaf contains enough treated and control units for a stable estimate.

Worked Examples

Worked Example: Select a Split, Then Estimate Its Leaf Effects

Consider a hypothetical randomized study with a covariate \(X\). The structure sample is used to choose a threshold at \(X=5\). Suppose the tree-selection procedure selects this split from its candidate splits. The structure-sample data are:

Structure-sample leafTreated outcomesControl outcomesDifference
\(X<5\)9, 86, 5\((9+8)/2-(6+5)/2=3\)
\(X\geq 5\)8, 76, 5\((8+7)/2-(6+5)/2=2\)

The estimated difference between these two structure-sample contrasts is \(3-2=1\) outcome unit. These data illustrate how outcomes can inform a split; they are not used to report the final leaf estimates. The separate estimation-sample observations are:

Estimation-sample leafTreated outcomesControl outcomesLeaf estimate
\(X<5\)10, 8, 96, 7, 5\((10+8+9)/3-(6+7+5)/3=9-6=3\)
\(X\geq 5\)9, 8, 107, 8, 6\((9+8+10)/3-(7+8+6)/3=9-7=2\)

The honest estimates are therefore 3 outcome units for \(X<5\) and 2 outcome units for \(X\geq5\). This calculation assumes the randomized treatment assignment described above and a positive count in each arm of each leaf. The separation means the estimation outcomes did not choose the threshold. The small sample sizes still make these estimates uncertain; honesty is not a guarantee of precision.

Worked Example: Why the Pooled Difference Need Not Average Leaf Effects

Suppose there are two leaves, each containing six units. In leaf 1, every unit has outcome 0 under either treatment. In leaf 2, every unit has outcome 10 under either treatment. Thus each leaf’s treatment effect is \(0-0=0\) or \(10-10=0\), respectively. Now suppose the treated counts are 4 in leaf 1 and 2 in leaf 2, while the control counts are 2 in leaf 1 and 4 in leaf 2.

The pooled treated mean is

$$ \bar{Y}_1=\frac{2(0)+1(10)}{3}=\frac{10}{3}. $$

The pooled control mean is

$$ \bar{Y}_0=\frac{1(0)+2(10)}{3}=\frac{20}{3}. $$

Consequently, the pooled difference is \(10/3-20/3=-10/3\), even though both within-leaf treatment effects are zero. The treated and control groups have different proportions from the two leaves, so the pooled comparison also reflects their different compositions.

In general, if \(n_{1L}\) and \(n_{0L}\) are the treated and control counts in leaf \(L\), and \(n_1\) and \(n_0\) are the corresponding totals, then

$$ \bar{Y}_1-\bar{Y}_0 =\sum_L\frac{n_{1L}}{n_1}\bar{Y}_{1L} -\sum_L\frac{n_{0L}}{n_0}\bar{Y}_{0L}. $$

This is not generally a weighted average of the leaf contrasts \(\bar{Y}_{1L}-\bar{Y}_{0L}\). It is such an average with common leaf weights when the treated and control proportions in each leaf match, so \(n_{1L}/n_1=n_{0L}/n_0\) for every leaf. In that special case, the pooled difference is \(\sum_L(n_{1L}/n_1)(\bar{Y}_{1L}-\bar{Y}_{0L})\). Without matching proportions, the pooled contrast can reflect both within-leaf outcome differences and composition differences.

Worked Example: Sample Splitting Does Not Remove Confounding

Suppose one leaf contains two severity groups, mild and severe. Treatment effects are \(+2\) outcome units in both groups, but treatment assignment in this hypothetical observational sample depends on severity. Among four treated people, one is mild with \(Y(1)=12\), and three are severe with \(Y(1)=4\). Among four controls, three are mild with \(Y(0)=10\), and one is severe with \(Y(0)=2\). These values correspond to \(Y(1)=Y(0)+2\) in each severity group.

The unadjusted difference within the leaf is

$$ \bar{Y}_{1L}-\bar{Y}_{0L} =\frac{12+4+4+4}{4}-\frac{10+10+10+2}{4} =6-8=-2. $$

Yet the treatment effect is \(+2\) in each severity group. The raw leaf contrast is misleading because the treated group contains more severe people, whose outcomes would be lower even without treatment. Using an independent estimation sample would prevent its outcomes from selecting the tree, but would not remove this confounding. Causal interpretation requires suitable identification assumptions and, where needed, adjustment for covariates.

Practical Design Choices and Limitations

Honesty requires a clear separation of data roles, not a particular fraction of observations for each role. A larger structure sample can help the tree identify useful splits; a larger estimation sample can improve the precision of leaf estimates. These aims compete when the total sample is fixed. Very small leaves are especially problematic: they may have few observations in one treatment arm, unstable means, or no observations in an arm at all. A tree-building procedure should therefore respect minimum leaf-size and treatment-support requirements, while recognizing that such safeguards do not replace causal assumptions.

The separation must also be maintained after the partition is chosen. If investigators inspect estimation outcomes and then change the threshold, merge leaves, or select a different tree because the effects look more appealing, those outcomes have influenced the final structure. The estimates are no longer protected from that adaptive selection. Any new outcome-driven choice requires an independent estimation sample for the revised structure if the same honesty rationale is to apply.

Common mistake: “The tree was trained on one sample, so its leaf effects are automatically causal and unbiased.” The correction is that honest splitting reduces the selection bias created by using the same outcomes to choose and estimate a partition. Causal interpretation still requires identification conditions; unbiasedness of the stated difference-in-means estimator also depends on the treatment-assignment design. Precision depends on the amount and distribution of estimation data within leaves.

How to Apply the Honest Procedure

1
Separate the samples.
Assign independent observations to the structure-selection sample and the estimation sample before using outcomes to select the tree.
2
Select the partition.
Use the structure sample and the chosen tree-building criterion to define \(\hat{\Pi}\). Do not use estimation outcomes to revise it.
3
Estimate within leaves.
In the estimation sample, calculate the treatment and control means separately in each leaf and subtract the control mean from the treatment mean.
4
Check what the contrast means.
Confirm that the treatment design and causal assumptions support the intended interpretation, and check that both arms have observations in each reported leaf.

The central benefit is a separation between discovering where effects may differ and estimating effects in the groups that discovery produced. That separation makes a selected tree’s leaf estimates less vulnerable to the same noise that guided its construction, while keeping causal identification and adequate data as separate requirements.

Check Your Understanding

Answer each question using the assumptions and notation introduced above.

  1. What role does the structure-selection sample play, and what role does the estimation sample play?
  2. Under randomized treatment, why does the difference between the two within-leaf means estimate the leaf’s average treatment effect?
  3. In the pooled-contrast example, why was the pooled difference \(-10/3\) even though both leaf effects were zero?
  4. Does honest splitting by itself make a difference in means causal in observational data? Explain.
  5. Why can a leaf estimate remain unstable even when its tree was selected honestly?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats