Tutorials › Causal Inference › From CATE Estimates to a Job-Training Policy

Causal Learning and Policy Decisions · Tutorial 972 of 972

From CATE Estimates to a Job-Training Policy

Learn to estimate conditional effects of a job-training offer, turn them into a cost-aware allocation rule, and assess that rule using independent randomized data.

Advanced 11 min read

What You'll Learn

  • Define the conditional treatment effect for a randomized job-training offer.
  • Fit a simple subgroup effect learner and distinguish its estimates from individual effects.
  • Check prediction calibration and assignment overlap using separate data.
  • Derive an offer rule that accounts for the cost of each offer.
  • Estimate a fixed policy’s net value on held-out randomized data and interpret its uncertainty.

From Effect Estimates to a Decision

A conditional treatment-effect estimate becomes useful for policy only when it is connected to a specific decision. A job-training program might increase earnings for some applicants more than for others, but training places and staff time are costly. The practical question is not simply whether an offer has a positive average effect. It is whether the expected gain for an applicant is large enough to justify the cost of offering the program.

This example follows that reasoning from the target effect to a policy and then to an evaluation. It uses a hypothetical randomized study. Throughout, \(W=1\) means that an applicant was offered training, and \(W=0\) means that the applicant was not offered training. The outcome \(Y\) is the applicant’s change in earnings over the following six months, measured in dollars. The target is the effect of the offer, not necessarily the effect of attending training. Applicants may decline an offer; no perfect-compliance assumption is made.

Definition: Let \(Y(1)\) and \(Y(0)\) be an applicant’s potential six-month earnings changes under, respectively, an offer and no offer. For pretreatment characteristics \(X=x\), the conditional average treatment effect of the offer is \(\tau(x)=E[Y(1)-Y(0)\mid X=x]\). A policy \(\pi(x)\) specifies whether to offer training to an applicant with characteristics \(x\), with \(\pi(x)=1\) for an offer and \(\pi(x)=0\) for no offer.

To identify offer effects from the randomized data, assume consistency, no interference between applicants, randomized assignment conditional on the recorded pretreatment characteristics, and a positive probability of each assignment in the applicant groups of interest. Consistency here means \(Y=WY(1)+(1-W)Y(0)\): observed earnings change corresponds to the potential outcome under the offer status actually assigned. Because \(W\) is offer assignment, this statement does not equate outcomes to training received. If applicants sometimes do not attend, the identified effect remains the effect of the offer under the stated assumptions.

A policy also needs an explicit objective. Suppose the program’s cost is \(c(x)\) dollars per offer, whether or not the applicant attends. If earnings and cost are valued in the same dollar units, a simple net-value objective is expected earnings change minus offer cost. This objective leaves out other benefits or burdens unless they are explicitly added.

Fitting a Conditional-Effect Learner

For a transparent example, use a shallow, two-leaf learner based on one pretreatment characteristic: prior annual earnings. Applicants with prior earnings below $25,000 are in group L; those with prior earnings of at least $25,000 are in group H. The cutoff is fixed before fitting. Within each group, estimate the offer effect by subtracting the randomized control mean from the randomized offer mean. This estimates a group-average effect and gives the learner a constant prediction within each leaf; it does not recover each person’s unobserved individual effect.

For this procedure, the assumptions above need to hold within each group. In particular, offer assignment must be randomized within groups, both assignment arms must be represented, and outcomes must be observed as defined. In the hypothetical study, assignment probabilities are one-half in each group, all applicants have observed outcomes, and no applicant’s offer changes another applicant’s earnings. Under those conditions, the difference in group means estimates the group-average effect of the offer.

Worked Example: Estimating the Two Leaf Effects

In the learner’s estimation sample, group L has 100 applicants offered training and 100 not offered. Their mean six-month earnings changes are $1,800 and $900, respectively. Group H also has 100 applicants in each arm, with mean changes of $1,050 for the offer arm and $800 for the no-offer arm. These are hypothetical figures.

For group L, the estimated offer effect is

$$ \hat{\tau}_{L}=\bar{Y}_{W=1,L}-\bar{Y}_{W=0,L} =\$1{,}800-\$900=\$900. $$

For group H, it is

$$ \hat{\tau}_{H}=\bar{Y}_{W=1,H}-\bar{Y}_{W=0,H} =\$1{,}050-\$800=\$250. $$

The fitted learner therefore predicts a $900 average earnings-change benefit from an offer for group L and a $250 benefit for group H. These are conditional group averages for the offer. They are not guarantees about any particular applicant, and the difference between the estimates could partly reflect sampling uncertainty.

A more flexible learner could select splits or combine many baseline characteristics. If outcomes help choose the splits, fitting and effect estimation must be separated or otherwise appropriately corrected; otherwise, the same random noise can make selected groups appear unusually different. The fixed cutoff here avoids outcome-driven split selection. More generally, honest sample splitting separates adaptive structure selection from estimation of effects within the selected structure.

Checking the Learner Before Making a Rule

A plausible-looking set of predictions is not enough. First check whether the predictions agree, within sampling uncertainty, with treatment-effect contrasts in data not used to fit the learner. Calibration assesses this agreement at a group or prediction-range level; it does not verify every person’s effect. Also check overlap: each relevant kind of applicant must have a meaningful chance of being assigned to either arm. In a randomized study the assignment probabilities are known, making this check especially direct.

Worked Example: Calibration and Assignment Overlap

Consider an independent calibration sample, not used to fit the learner or to evaluate the final policy. In that sample, the offer-effect contrast is $820 for group L, with standard error $300, and $310 for group H, with standard error $280. The learner predicted $900 and $250, respectively.

The calibration residuals, defined as observed group contrast minus predicted effect, are \(820-900=-\$80\) for L and \(310-250=\$60\) for H. For a rough check, 95% normal-approximation intervals for the validation contrasts are

$$ \text{L: }\$820\mathbin{\pm}1.96(\$300)=[\$232,\ \$1{,}408], \qquad \text{H: }\$310\mathbin{\pm}1.96(\$280)\approx[-\$239,\ \$859]. $$

Both predictions fall within their corresponding intervals. That is compatible with calibration at this coarse group level, but the intervals are wide: this check does not establish that the predictions are accurate, nor does it establish precise effects in either group. In both groups, the known offer probability is \(P(W=1\mid X)=0.5\), so overlap holds for these data. If some applicant group had no chance of receiving an offer or no-offer, its two potential-outcome means could not be compared from this randomized sample.

Calibration should be interpreted as a diagnostic rather than a pass-or-fail certificate. An overall average can look well calibrated even when predictions are too high in one part of the population and too low in another. Conversely, noisy subgroup contrasts can disagree with predictions by chance. A useful check reports the groups or ranges examined, the estimates’ uncertainty, and whether assignment probabilities support comparisons there.

Converting Effects and Costs Into an Offer Rule

Suppose the program costs $400 per offer for every applicant. For a person with characteristics \(x\), offering training has estimated incremental net value \(\hat{\tau}(x)-c(x)\) compared with making no offer. Thus a simple cost-aware rule offers training when the estimated earnings gain exceeds the cost. It does not require the estimated effect to be merely positive.

Decision rule: With no additional utility terms or capacity constraint, the estimated net-value rule is \(\hat{\pi}(x)=1\) when \(\hat{\tau}(x)>c(x)\), and \(\hat{\pi}(x)=0\) otherwise. If estimated gain equals cost, either action has estimated net value zero relative to no offer; a tie convention should be specified.

The rule follows directly from comparing the two actions. No offer has incremental net value zero. An offer has expected incremental net value \(\tau(x)-c(x)\). The action with the larger value is therefore the offer exactly when that difference is positive. With estimated effects, the same comparison gives a plug-in decision rule; it does not remove uncertainty in the estimates.

$$ \text{Offer if }\hat{\tau}(x)-c(x)>0 \quad\Longleftrightarrow\quad \hat{\tau}(x)>c(x). $$

Worked Example: Applying the Cost-Aware Rule

For group L, the estimated offer gain is $900 and the offer cost is $400. Its estimated net value per offer is

$$ \$900-\$400=\$500. $$

The rule offers training to group L. For group H, the corresponding calculation is

$$ \$250-\$400=-\$150. $$

The rule does not offer training to group H. If 40% of the eligible population is in L and 60% is in H, the learner’s estimated net value averaged across that population is \(0.40(\$500)+0.60(\$0)=\$200\) per eligible applicant. This is a value predicted from the fitted estimates, not an independent estimate of what the policy will achieve. A noisy estimate near the cost threshold may also make an offer decision unstable; sensitivity to costs and utilities should be considered before deployment.

Evaluating the Fixed Policy on Held-Out Data

A policy should be evaluated on data that played no role in fitting the effect learner, selecting its rule, or revising the policy after diagnostics. In randomized evaluation data, the policy’s offer and no-offer outcomes can be compared within the groups it treats. The policy must be fixed before looking at these evaluation outcomes; otherwise, reported value can benefit from the same noise used to adapt the rule.

For this example, compare the policy with the benchmark of making no offers. Let \(q_L\) be the share of applicants in group L in the target population. Since the rule offers only to L, the policy’s value relative to no offers is \(q_L\) times the group L offer effect minus the cost per offer. More generally, with groups \(g\), shares \(q_g\), and a fixed rule \(\pi_g\), the relative net value is the share-weighted sum of the net effects in groups the policy treats.

$$ \Delta V(\pi)=\sum_g q_g\pi_g\{\tau_g-c_g\}. $$

This expression follows by comparing actions group by group. In group \(g\), no offer is the benchmark and has incremental value zero. An offer changes expected earnings by \(\tau_g\) and incurs cost \(c_g\), so its net increment is \(\tau_g-c_g\). The policy incurs that increment for the share \(q_g\pi_g\) of applicants offered treatment; summing across groups gives the formula.

Worked Example: Held-Out Value of the Learned Rule

Suppose the final held-out randomized sample has the same target group mix: 40% L and 60% H. It contains 80 L applicants, split evenly between offer and no-offer, and 120 H applicants, also split evenly. For the L group, mean earnings changes are $1,500 in the offer arm and $900 in the no-offer arm. The policy was fixed in advance to offer only to L, and the cost remains $400 per offer.

The held-out L-group contrast is \(\$1{,}500-\$900=\$600\). The estimated net increment per L applicant is \(\$600-\$400=\$200\). Group H receives no offers under this policy, so its increment relative to no offers is zero. Consequently, the held-out policy-value estimate relative to making no offers is

$$ \widehat{\Delta V}(\pi) =0.40(\$600-\$400)+0.60(\$0) =\$80 $$

per eligible applicant. This is a point estimate; it does not show that the policy’s true net value is positive. For illustration, suppose the standard deviation of earnings changes is $1,200 in each L arm. Treating the target group shares as fixed, the standard error of the estimated value is

$$ SE(\widehat{\Delta V}) =0.40\sqrt{\frac{\$1{,}200^2}{40}+\frac{\$1{,}200^2}{40}} \approx \$107.3. $$

A rough 95% normal-approximation interval is \(\$80\mathbin{\pm}1.96(\$107.3)\), or approximately \([-\$130,\ \$290]\) per eligible applicant. Because this interval includes zero, these hypothetical held-out data do not establish positive net value at this level of uncertainty. The interval concerns the value of this fixed offer policy relative to no offers, under the randomized assignment and target-population assumptions; it is not an interval for any applicant’s individual effect.

Common Mistakes and Practical Interpretation

  • Calling an offer effect an attendance effect. Here \(W\) denotes assignment to an offer. If some applicants decline, the offer-effect estimate is not automatically the causal effect of receiving training. The potential outcomes, consistency statement, policy, and cost must all refer to the same intervention.
  • Offering whenever the estimated effect is positive. A positive earnings effect may still be smaller than the cost. Compare estimated benefit with cost in matching units.
  • Treating a CATE prediction as an individual fact. The learner estimates conditional averages. It does not reveal both potential outcomes for an applicant or promise that the applicant will benefit.
  • Evaluating an adaptive policy on the data used to create it. That can make estimated value optimistic. Keep an untouched randomized evaluation sample for the final assessment.
  • Reading a positive point estimate as proof of success. In the worked evaluation, the estimated net value is positive, but its approximate interval includes zero. The point estimate and its uncertainty should be reported together.
Key takeaway: A defensible job-training policy aligns the causal target with the actual intervention, checks that conditional-effect predictions are credible enough for decision use, compares estimated benefits with costs, and evaluates the fixed rule on independent data. Each step answers a different question; none alone establishes that a policy will improve outcomes in every setting.

Check Your Understanding

Answer each question using the definitions and example above.

  1. In this example, what intervention does \(W=1\) denote, and why does the estimated effect not automatically represent the effect of attending training?
  2. If a subgroup’s estimated offer effect is $350 and the cost per offer is $400, what action does the stated rule choose, and what is its estimated net value relative to no offer?
  3. Why does calibration in broad subgroups not establish that the learner predicts each applicant’s individual effect accurately?
  4. What role does overlap play in estimating offer effects from randomized data?
  5. The held-out estimate is $80 per eligible applicant, with an approximate interval from \(-\$130\) to $290. What conclusion about positive net value is warranted, and what conclusion is not?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats