Honest Causal Forests and Sample Splitting

Why Separate Split Selection From Effect Estimation?

A causal tree searches for covariate partitions that reveal differences in treatment effects. But a search over many candidate splits can select a partition partly because of random outcome variation. If the same observations then estimate the effects in the selected leaves, that variation can influence both the choice of the leaves and the reported effects. The resulting estimates can be too extreme even when the underlying effects are not.

An honest causal tree separates these tasks. One sample determines the tree structure; a different sample estimates treatment effects within the resulting leaves. This does not guarantee that the selected tree is scientifically correct or that every estimate is precise. It does reduce the direct reuse of outcome noise that can make adaptively selected leaf estimates biased.

For a covariate vector \(X\), let \(Y(1)\) and \(Y(0)\) denote the potential outcomes under treatment and control. The conditional average treatment effect is \(\tau(x)=E[Y(1)-Y(0)\mid X=x]\). A tree groups covariate values into leaves, so the leaf-level target is generally an average of \(\tau(X)\) over the covariate distribution in that leaf—not necessarily one common effect for every individual in it.

How Honesty Limits Adaptive Bias

Suppose a researcher compares several candidate partitions. Even if each candidate’s estimated effect contrast has mean-zero noise, the selected partition may be the one whose noise makes it look most promising. Using those same observations to report the selected contrast carries some of that favorable noise into the estimate. This is a form of adaptive bias: the estimate is affected by the data-dependent choice made before it.

A simple calculation shows the mechanism. Suppose two candidate split scores have the same true value \(\Delta\), but their observed scores are \(S_j=\Delta+\varepsilon_j\), where \(\varepsilon_1\) and \(\varepsilon_2\) are independent \(N(0,\sigma^2)\) errors. The selected score is the larger of the two. Since \(\max(\varepsilon_1,\varepsilon_2)=(\varepsilon_1+\varepsilon_2+|\varepsilon_1-\varepsilon_2|)/2\), its expectation is positive: \(\varepsilon_1-\varepsilon_2\sim N(0,2\sigma^2)\), and \(E|\varepsilon_1-\varepsilon_2|=2\sigma/\sqrt{\pi}\). Thus the selected score has expected noise \(\sigma/\sqrt{\pi}\), not zero.

This calculation is a simplified illustration, not a formula for the bias of every tree algorithm. In an honest tree, the structure sample may still select a split that looks unusually promising in that sample. But if the estimation observations are independent of the structure sample, the estimation noise in the fixed selected leaves has conditional mean zero under the relevant sampling and causal assumptions. The split-selection noise is then not automatically carried into the separate leaf estimate.

Estimation Logic and Assumptions

The basic logic can be stated without committing to one particular split criterion. Randomly allocate independent observations to a structure sample and an estimation sample. Use only the structure sample to choose split variables, cut points, and the resulting leaves. Then, holding that partition fixed, use estimation-sample observations in each leaf to estimate the causal contrast. In a forest, the allocation and tree construction are repeated across trees, and the resulting tree estimates are aggregated.

For a randomized experiment, a simple leaf estimate is the difference between the estimation-sample treatment and control means in that leaf. Its causal interpretation relies on valid treatment randomization, consistency of observed outcomes with the assigned treatment’s potential outcome, and positive probability of both treatment arms in the leaf. The treatment assignment must not be confounded with potential outcomes under the randomization design. If the design uses strata or clusters, the estimator and its uncertainty calculation must respect that design. In small leaves, chance imbalance and few observations can still make estimates unstable.

In an observational study, a causal interpretation requires conditional exchangeability, \( (Y(1),Y(0))\perp W\mid X\), where \(W\) is treatment status, along with consistency and overlap, \(0<P(W=1\mid X=x)<1\) in the target region. One option is to average an orthogonal score within each leaf. Write \(e(x)=P(W=1\mid X=x)\), \(\mu_w(x)=E[Y\mid W=w,X=x]\), and define

Under the stated causal assumptions and correct nuisance functions, \(E[\psi_i\mid X_i=x]=\tau(x)\). In applications, nuisance functions are typically estimated, and cross-fitting can help control overfitting in those estimates. The honest partition still has its own sample-separation requirement: the outcomes used to choose the tree structure should not also be used to estimate the leaf effects. Nuisance estimation and splitting must be organized so that the intended separation is not inadvertently undone.

Worked Tree With a Randomized Estimation Sample

Consider a hypothetical randomized trial with a binary covariate \(X\), measured in years of prior program experience. The structure sample’s splitting procedure selects the threshold \(X\leq 2.5\) years. For this example, take that selection as given: the numerical split criterion is not needed to estimate effects once the partition is fixed. The estimation sample is independent of the structure sample, and treatment was randomized with both arms represented in each leaf.

Worked Orthogonal-Score Calculation

The same separation applies when leaf estimates use scores rather than raw treatment-control means. The following small calculation illustrates how an honest leaf estimate is formed after a partition has already been selected.

What Honesty Does Not Guarantee

Honesty addresses a particular source of bias; it is not a substitute for causal identification, adequate data, or careful inference. It does not make treatment assignment ignorable in an observational study, repair a lack of overlap, or ensure that a leaf contains enough treated and control observations. Nor does it guarantee that the selected covariates or thresholds capture meaningful effect variation.

Because only part of the data estimates each tree’s effects, honest trees can have higher variance than trees that reuse all observations. Forests can mitigate some variance by combining many trees, but estimates may remain unstable when leaves are small, treatment is rare, or candidate partitions are numerous. Honest inference also depends on the sampling design and on the way trees and nuisance functions are built; separation alone does not automatically validate a standard error or confidence interval.

The leaf target deserves attention as well. If effects vary within a leaf, its estimate averages over the covariate distribution represented by the estimation observations in that leaf. Changing the estimation population can change that average even if the partition is unchanged. Therefore, report the target population and the leaf definition alongside an estimated effect.

Common Mistakes

• Using the split sample’s outcomes again for leaf effects. Choosing a partition with one set of observations and then calculating its reported effects from those same observations does not satisfy the basic honesty definition.
• Treating honesty as randomization. Honest splitting does not make an observational treatment assignment unconfounded. The identification assumptions still need to hold.
• Confusing honesty with nuisance cross-fitting. Cross-fitting addresses overfitting in nuisance predictions; an honest tree separately restricts which observations choose the partition and which estimate leaf effects.
• Claiming no uncertainty remains. Independent estimation removes direct reuse of split-selection outcomes, but finite-sample noise, small-leaf instability, and uncertainty from adaptive learning still require attention.
• Interpreting a leaf estimate as an individual effect. The estimate is a group average for the leaf’s target population, not a measured treatment effect for each person in it.