Overfitting In Estimated Treatment Rules

Why Flexible Q-Models Can Select Noise

In Q-learning, treatment decisions are obtained by comparing fitted Q-values across available treatments. Flexibility can help represent genuine treatment-effect modification, but every additional interaction or subgroup also gives the model more opportunities to fit random variation. The concern is not merely that some fitted coefficients are inaccurate: maximizing fitted Q-values can turn those inaccuracies into treatment choices. A fitted rule may therefore look effective in the data used to construct it and perform less well on new individuals.

For one decision with binary treatment $A\in\{0,1\}$ and covariates $X$, write the conditional mean outcome under treatment $a$ as $Q(a,x)=E[Y^a\mid X=x]$. A rule $d$ assigns a treatment $d(x)$ to each covariate value. In a working Q-model, treatment interactions determine how the estimated contrast $\hat Q(1,x)-\hat Q(0,x)$ changes with $x$; the plug-in rule assigns treatment 1 when that contrast is positive, treatment 0 when it is negative, and either treatment when it is zero.

Definition: Overfitting in an estimated treatment rule occurs when the fitted Q-model or the search over candidate rules captures sample-specific noise sufficiently to make the estimated rule perform worse, on new data, than its apparent performance in the data used to fit or select it. A highly flexible model is not necessarily overfit; the concern is excessive adaptation relative to the information available.

Overfitting is especially plausible when treatment-effect interactions are estimated in small subgroups. A subgroup may contain few treated individuals, few controls, or both. Its fitted contrast can then be driven by a handful of outcomes. A large collection of candidate splits or interactions compounds the issue: even if each candidate is noisy, the candidate with the most favorable fitted value is likely to have benefited from favorable noise.

A Small-Sample Interaction Example

Worked Example: A Sparse Subgroup Reverses a Treatment Decision

Suppose a saturated Q-model estimates separate mean outcomes for two subgroups $S=0$ and $S=1$. The training data have these cell summaries:

Subgroup	Control: sample size	Control mean	Treatment: sample size	Treatment mean	Fitted contrast
$S=0$	20	5	20	7	$7-5=2$
$S=1$	20	5	2	2	$2-5=-3$

In subgroup $S=0$, the fitted treatment contrast is $2$, so the plug-in rule chooses treatment. In subgroup $S=1$, it is $-3$, so the rule chooses control. The treatment mean of $2$ in subgroup $S=1$ comes from just two outcomes, $-8$ and $12$, since $(-8+12)/2=2$. One extreme observation has substantial influence on this cell mean.

To make the overfitting issue explicit, suppose the underlying treatment contrast in subgroup $S=1$ is actually $1$. The observed contrast of $-3$ then reflects sample variation, not a true reason to withhold treatment in that subgroup. The fitted rule has followed the noisy estimate. This example does not show that every small subgroup produces a wrong decision; it shows why a decision based on a sparse interaction can be fragile. More flexible partitions can create still smaller cells.

A related issue arises even when no single cell is exceptionally small. If analysts try many plausible interactions, thresholds, or subgroup definitions and retain the rule with the best training-sample value, the search itself selects favorable noise. This is a form of selection optimism, distinct from the question of whether the Q-model is correctly specified.

Selection Optimism: A Simple Result

Theorem: Let $K\geq 2$ candidate rules have the same true value $V$, and let their estimated values be $\hat V_k=V+\varepsilon_k$, where each error has finite expectation zero. Then $E[\max_{1\leq k\leq K}\hat V_k]\geq V$. For two candidates, the inequality is strict if $E[|\varepsilon_1-\varepsilon_2|]>0$.

This result does not require independent errors. It describes the effect of choosing the largest of several noisy estimates when the candidate rules have equal true value.

Compare the selected estimate with each candidate.
For every $k$, $\max_j \hat V_j\geq \hat V_k$. Taking expectations gives $E[\max_j \hat V_j]\geq E[\hat V_k]=V$.

Check when the two-candidate inequality is strict.
For $K=2$, $\max(\varepsilon_1,\varepsilon_2)=(\varepsilon_1+\varepsilon_2+|\varepsilon_1-\varepsilon_2|)/2$. Its expectation is $E[|\varepsilon_1-\varepsilon_2|]/2$, which is positive under the stated condition. Adding $V$ proves the claim. $\square$

Thus the estimated value of the rule selected by maximizing training estimates is not, by itself, a reliable estimate of that rule’s performance. The amount of optimism depends on the candidate set and the dependence and variability of the estimates. This result does not say that a flexible rule must be inferior: it says that selection and evaluation on the same data can make its apparent value optimistic.

Independent Validation of a Fixed Rule

A direct safeguard is to fit and select the rule using training data, then evaluate that locked rule on independent validation data. The validation observations should not be used to choose interactions, thresholds, or other aspects of the rule being evaluated. For clarity, consider a single decision, consistency $Y=Y^A$, conditional exchangeability $Y^a\perp A\mid X$, and a known treatment probability $g(a\mid x)=P(A=a\mid X=x)$. Assume positivity: $g(d(x)\mid x)>0$ for covariate values relevant to the target population. Also assume the outcomes are integrable. These conditions hold, for example, in a randomized study with known assignment probabilities, if assignment is independent of potential outcomes given $X$.

Definition: For a fixed rule $d$, the inverse-probability weighted validation estimator based on $n_{\mathrm{val}}$ independent validation observations is $$ \hat V_{\mathrm{val}}(d)=\frac{1}{n_{\mathrm{val}}}\sum_{i=1}^{n_{\mathrm{val}}} \frac{I\{A_i=d(X_i)\}Y_i}{g(A_i\mid X_i)}. $$ Its target is the value $V(d)=E[Y^{d(X)}]$ in the population represented by the validation sample.

Theorem: If the rule $d$ is fixed independently of the validation observations and the assumptions above hold, then $E[\hat V_{\mathrm{val}}(d)\mid d]=V(d)$. If the validation observations are independent and identically distributed and the estimator has finite variance, then $\hat V_{\mathrm{val}}(d)\xrightarrow{p}V(d)$ as $n_{\mathrm{val}}\to\infty$.

To prove the expectation statement, condition first on $X=x$. Since $d$ is fixed independently of validation data, it is a fixed treatment choice at $x$. The indicator in the summand is nonzero only when $A=d(x)$. Conditional exchangeability and consistency then give

\begin{aligned} E\left[\left.\frac{I\{A=d(x)\}Y}{g(A\mid x)}\right|X=x,d\right] &=\frac{P(A=d(x)\mid X=x)}{g(d(x)\mid x)} E[Y\mid A=d(x),X=x]\\ &=E[Y^{d(x)}\mid X=x]. \end{aligned}

The denominator is positive by positivity. Taking expectations over $X$ yields $E[Y^{d(X)}]=V(d)$. Averaging independent validation contributions preserves this expectation. If their variance is finite, the weak law of large numbers applied to those independent, identically distributed contributions gives convergence in probability to their expectation. $\square$

The theorem evaluates a rule that is already fixed; it does not justify using the validation sample repeatedly to modify the rule and then reporting the same estimate as an unbiased evaluation. Once validation outcomes influence selection, the evaluated rule is no longer independent of those outcomes.

Calculating Validation Contributions

Worked Example: Validating the Rule That Always Assigns Treatment Zero

Suppose treatment is randomized with $g(0\mid X)=g(1\mid X)=0.5$. The eight validation observations have the following observed treatments and outcomes:

Person	$A$	$Y$	Contribution for rule $d(X)=0$
1	1	2	0
2	0	2	$2(2)=4$
3	1	4	0
4	0	4	$2(4)=8$
5	0	5	$2(5)=10$
6	1	3	0
7	0	3	$2(3)=6$
8	1	1	0

For this rule, the contribution is $I(A=0)Y/0.5$. In person order, the contributions are $0,4,0,8,10,0,6,0$. Their sum is $28$, so the estimated value is $28/8=3.5$. For instance, person 4 received treatment 0 and had $Y=4$, giving contribution $4/0.5=8$; person 1 received treatment 1, so their contribution for the always-zero rule is zero. This estimate is noisy with only eight observations, but the calculation correctly targets the value of the fixed rule under the stated assumptions.

The zero contributions do not mean those individuals have no outcome under treatment zero. They mean their observed treatment does not reveal that outcome, so their contribution to this particular inverse-probability weighted estimate is zero. In larger samples, the estimator uses weighted outcomes from individuals whose observed treatment matches the rule.

Sample Splitting in Practice

Worked Example: Comparing a Locked Subgroup Rule on Held-Out Data

Suppose a rule was selected using training data: assign treatment 0 when $X=0$, and treatment 1 when $X=1$. It is now evaluated on six independent validation observations under randomized assignment with probability $0.5$ for each treatment.

Person	$X$	$A$	$Y$	Contribution for the locked rule	Contribution for always 0
1	0	0	4	8	8
2	0	1	1	0	0
3	1	1	3	6	0
4	1	0	2	0	4
5	0	1	2	0	0
6	1	1	4	8	0

For the subgroup rule, people 1, 3, and 6 received the treatment the rule assigns. Their contributions are $4/0.5=8$, $3/0.5=6$, and $4/0.5=8$; the other contributions are zero. The estimate is $(8+6+8)/6=22/6\approx3.67$. For the always-zero rule, people 1 and 4 contribute $8$ and $4$, and the estimate is $12/6=2$. In this validation sample, the locked subgroup rule has the larger estimated value. With only six observations, this difference is not strong evidence that it has greater population value; the example illustrates evaluation, not a guarantee of superiority.

A single random split is simple, but it uses fewer observations to fit the rule and fewer to evaluate it. Repeating the process with different splits can reveal whether selected interactions and decisions are sensitive to which observations are used for training. Out-of-fold evaluation follows a related idea: each observation is evaluated using a rule trained without that observation. When folds are used, the estimates share training data, so treating all fold-specific results as independent for uncertainty calculations is generally inappropriate.

Diagnostics, Alternatives, and Limitations

Potential warning signs include treatment-by-covariate cells with very few observations, unusually large or unstable interaction estimates, and decisions that change after small perturbations of the training sample. Compare fitted treatment contrasts with cell counts and examine how the selected rule changes across reasonable resamples or training splits. A discrepancy between training and independent validation performance is another warning. None of these checks alone proves overfitting: genuine treatment-effect heterogeneity can also produce complex rules, and a validation estimate may fluctuate substantially in a small sample.

Regularization and simpler prespecified interaction structures can reduce sensitivity to noisy cells, while sample splitting provides an evaluation that is separated from rule fitting. These approaches answer different needs: regularization changes how the rule is estimated; a holdout sample assesses the performance of the resulting fixed rule. A simpler model may miss real effect modification, and a split sample sacrifices precision in both training and evaluation. If the validation sample is used repeatedly to select among many candidate rules, it becomes part of the selection process and no longer provides an untouched evaluation.

Validation also does not repair violations of exchangeability or positivity, incorrect outcome measurement, or a mismatch between the validation population and the population of interest. When treatment probabilities are estimated rather than known, the estimator and its justification must account for that estimation; the theorem above states the known-probability case. Very small treatment probabilities can create highly variable weighted contributions even when positivity holds mathematically.

Common pitfall: Selecting the rule with the largest estimated training value and interpreting that same maximum as an unbiased estimate of its value. The mistake is to ignore that the rule was chosen partly because its estimation error was favorable. Lock the rule before evaluating it on independent data, or use an evaluation design that separates each observation from the fitting step that determines its prediction.

Check Your Understanding

Answer each question, then check the solutions below.

Why can selecting the largest training-sample Q-value across many candidate rules lead to optimism even if all rules have the same true value?
In a randomized study with treatment probability $0.25$, what is the inverse-probability contribution for a rule-matching observation with outcome $Y=8$? What is the contribution if treatment does not match?
What independence condition makes the validation unbiasedness result apply to a learned rule?
Name two diagnostics for possible overfitting in treatment interactions and one limitation of a holdout sample.

Solutions. (1) The maximum selects not only a rule but also favorable estimation noise; the expected maximum of equal-value noisy estimates is at least their common true value. (2) For a match, the contribution is $8/0.25=32$; for a mismatch, the indicator is zero and the contribution is $0$. (3) The rule must be fixed independently of the validation observations, and the validation sample must satisfy the stated treatment-assignment assumptions. (4) Examples include sparse treatment-by-covariate cells, unstable interaction estimates, and decisions that change across training splits. A holdout reduces the data available for fitting and can itself give an imprecise value estimate.

Person	\(A\)	\(Y\)	Contribution for rule \(d(X)=0\)
1	1	2	0
2	0	2	\(2(2)=4\)
3	1	4	0
4	0	4	\(2(4)=8\)
5	0	5	\(2(5)=10\)
6	1	3	0
7	0	3	\(2(3)=6\)
8	1	1	0