Q-Learning Under Model Misspecification

Why Misspecification Matters in Q-Learning

Q-learning estimates conditional mean outcomes and then chooses the action with the largest fitted value. If the working Q-model is wrong, its fitted coefficients generally do not estimate the true conditional means. They instead describe a best approximation under the model and the distribution of histories and treatments in the data. That approximation can alter both the estimated treatment contrast and the action selected for a given history.

The distinction is important: a misspecified model may predict outcomes imperfectly yet still make the right treatment decisions, or it may fit average outcomes reasonably well while placing the decision boundary incorrectly. The consequences depend on where the contrast is wrong, how often those histories occur, and how large the true treatment effects are there.

Definition: At a treatment stage, let $H$ denote the observed history, $A\in\{0,1\}$ the treatment, and $Y$ the outcome being regressed in the Q-learning step. Write $Q(h,a)=E[Y\mid H=h,A=a]$, and define the true treatment contrast (or blip) as $\delta(h)=Q(h,1)-Q(h,0)$. A working model $Q_\theta(h,a)$ is misspecified if no value of $\theta$ makes it equal to $Q(h,a)$ almost surely. Under squared-error fitting, its population target is a minimizer of $E[(Y-Q_\theta(H,A))^2]$, called the pseudo-true parameter.

For causal interpretation, the conditional means must correspond to the treatment choices of interest. A sufficient single-stage setup is consistency, conditional exchangeability of treatment given $H$, and positivity: each treatment has positive probability at histories in the target population. For the calculations below, also assume $E[Y^2]<\infty$. Let $p(h)=P(A=1\mid H=h)$. Positivity means $0<p(H)<1$ almost surely; a uniform bound away from zero and one is a useful stronger condition for stable estimation.

What a Misspecified Contrast Estimates

A particularly revealing case separates the baseline mean from the treatment contrast. Suppose the working model is $g(H)+A\tau_\beta(H)$, where the baseline function $g$ is unrestricted among square-integrable functions and the contrast $\tau_\beta$ belongs to a specified model class. Although an unrestricted baseline is an idealization, it isolates how the contrast model is chosen.

Result: For any fixed candidate contrast $\tau(H)$, minimizing $E[(Y-g(H)-A\tau(H))^2]$ over unrestricted $g$ leaves, apart from a term that does not depend on $\tau$, the criterion $E[p(H)\{1-p(H)\}\{\delta(H)-\tau(H)\}^2]$. Thus the best working contrast is a weighted squared-error projection of the true blip, with weight $p(H)\{1-p(H)\}$.

To derive this result, set $q_0(H)=E[Y\mid H,A=0]$. Then $E[Y\mid H,A]=q_0(H)+A\delta(H)$ and $E[Y\mid H]=q_0(H)+p(H)\delta(H)$. The essential decomposition is

Y-E[Y\mid H] = \{Y-E[Y\mid H,A]\}+\{A-p(H)\}\delta(H).

The first term has conditional mean zero given $(H,A)$; the second generally does not. For a fixed $g$ and $\tau$, write the residual as

Y-g(H)-A\tau(H) = \{Y-E[Y\mid H,A]\} +\{q_0(H)-g(H)+A(\delta(H)-\tau(H))\}.

The cross-product has expectation zero: the second brace is a function of $(H,A)$, and the first brace has conditional mean zero given $(H,A)$. The squared-error risk therefore equals $E[\operatorname{Var}(Y\mid H,A)]$ plus the expectation of the squared second brace. Conditional on $H=h$, put $c=q_0(h)-g(h)$ and $d=\delta(h)-\tau(h)$. The latter squared term has conditional expectation

(1-p)c^2+p(c+d)^2 = \{c+pd\}^2+p(1-p)d^2.

It is minimized over $g(h)$, equivalently over $c$, at $c=-pd$. The remaining term is $p(1-p)d^2$. Taking expectation over $H$ proves the result. In particular, if $\tau$ is restricted to a linear span, its population target is the weighted projection of $\delta$ onto that span, not necessarily the true blip. $\square$

This projection statement requires an unrestricted baseline. If the baseline is also restricted, the fitted treatment coefficient can be affected by baseline misspecification and treatment allocation. It need not equal the weighted projection just derived.

Worked Example: A Constant Contrast With Unequal Projection Weights

Suppose $P(H=0)=0.75$, $P(H=1)=0.25$, $p(0)=0.5$, and $p(1)=0.9$. Let the true baseline mean be zero at both histories, with $\delta(0)=2$ and $\delta(1)=-1$. Fit a constant contrast $\tau$, while leaving $g(H)$ unrestricted.

The two history-specific projection weights, including their probabilities in the population, are

P(H=0)p(0)\{1-p(0)\}=0.75(0.5)(0.5)=0.1875,

P(H=1)p(1)\{1-p(1)\}=0.25(0.9)(0.1)=0.0225.

The weighted least-squares minimizer is the weighted mean of the two blips:

\tau^* = \frac{0.1875(2)+0.0225(-1)}{0.1875+0.0225} = \frac{0.375-0.0225}{0.21} = \frac{47}{28} \approx 1.679.

The fitted contrast is positive, so the fitted rule treats at both histories. The true rule treats at $H=0$ and does not treat at $H=1$. With $q_0(h)=0$, the value of the fitted rule is $0.75(2)+0.25(-1)=1.25$, while the optimal rule has value $0.75(2)+0.25(0)=1.5$. The value loss is $0.25$. The constant model’s projection emphasizes histories with more treatment overlap; it does not average the two blips using only the distribution of $H$.

Contrast Errors, Decision Boundaries, and Value

For a binary treatment, the optimal action at history $h$ is to treat when $\delta(h)>0$, not when the fitted outcome under treatment is large in isolation. A misspecified contrast $\tau^*(h)$ yields the rule $\hat d(h)=I\{\tau^*(h)>0\}$, with a specified convention for ties. The rule is decision-consistent when its action agrees with the optimal action almost surely, though its estimated contrast need not equal the true blip.

Result: For any deterministic single-stage rule $d(H)\in\{0,1\}$, define its value as $V(d)=E[Q(H,d(H))]$. An optimal rule is $d^*(h)=I\{\delta(h)>0\}$, with either action allowed when $\delta(h)=0$. Its value difference from $d$ is $$ V(d^*)-V(d)=E[|\delta(H)|I\{d(H)\ne d^*(H),\ \delta(H)\ne 0\}]. $$

To verify the identity, note that $Q(H,d(H))=q_0(H)+d(H)\delta(H)$. If $d=d^*$, the two rules have equal conditional value. If they disagree and $\delta(H)>0$, the optimal rule treats and the loss from $d$ is $\delta(H)$. If they disagree and $\delta(H)<0$, the optimal rule does not treat and the loss is $-\delta(H)$. In both cases the loss is $|\delta(H)|$. When $\delta(H)=0$, either action has the same value. Taking expectations gives the stated formula. $\square$

Worked Example: A Nonlinear Blip and a Distorted Boundary

Let $H$ be uniform on $[-1,1]$, let $p(H)=0.5$, and let $q_0(H)=0$. Suppose the true blip is $\delta(h)=h^2-0.25$, but the working contrast is affine, $\tau(h)=\alpha+\beta h$, with an unrestricted baseline.

The projection weights are constant because $p(1-p)=0.25$. Under the uniform distribution, symmetry gives $E[H]=E[H^3]=0$ and $E[H^2]=1/3$. Thus the affine projection has

\alpha=E[\delta(H)]=\frac{1}{3}-\frac{1}{4}=\frac{1}{12}, \qquad \beta=\frac{\operatorname{Cov}(H,\delta(H))}{\operatorname{Var}(H)}=0.

The fitted contrast is positive everywhere, so the fitted rule treats everybody. The true contrast is positive only for $|h|>0.5$. The fitted rule’s value is $E[\delta(H)]=1/12$. The optimal rule’s value is

E[\max\{\delta(H),0\}] = \int_{0.5}^{1}(h^2-0.25)\,dh = \left[\frac{h^3}{3}-\frac{h}{4}\right]_{0.5}^{1} = \frac{1}{6}.

The value loss is $1/6-1/12=1/12$. The error is not simply a shifted threshold: the affine contrast has erased the curved boundary at both ends of the history range.

When the Baseline Model Is Also Wrong

The weighted projection result should not be applied automatically to an ordinary regression with a restricted baseline. For instance, omitting an important prognostic variable can make treatment and baseline composition differ across observed treatment groups. A fitted treatment coefficient may then combine baseline differences with treatment effects, even when treatment is randomized conditional on history.

Worked Example: An Omitted Baseline Reverses the Fitted Contrast

Let $P(H=0)=P(H=1)=0.5$, with $q_0(0)=0$, $q_0(1)=10$, and $\delta(0)=\delta(1)=1$. Treatment is randomized conditional on $H$, with $p(0)=0.9$ and $p(1)=0.1$. Fit only an intercept and $A$, omitting $H$.

Among treated individuals, the probability of $H=0$ is $0.9$, because the joint probabilities for treated $H=0$ and $H=1$ are $0.45$ and $0.05$. Their mean outcome is $0.9(1)+0.1(11)=2$. Among untreated individuals, the corresponding joint probabilities are $0.05$ and $0.45$, so the probability of $H=1$ is $0.9$; their mean outcome is $0.1(0)+0.9(10)=9$. With an intercept and binary $A$, the least-squares treatment coefficient is the treated mean minus the untreated mean, $2-9=-7$.

The fitted model therefore recommends no treatment, despite the true conditional effect being $+1$ at both histories. The failure is not a contradiction of the weighted projection result: the baseline was restricted rather than unrestricted. Adjusting for $H$ would be essential to represent the conditional outcome means in this example.

Misspecification Across Stages and Practical Checks

In a multistage regime, a Q-model at an earlier stage is fitted to a pseudo-outcome constructed using later-stage fitted values. If the later-stage model is misspecified, the pseudo-outcome can inherit its errors; the earlier-stage fit can then add its own approximation error. Consequently, each stage’s pseudo-true target depends on the downstream models and the distribution generated by the observed data. Correct estimation of one stage does not, by itself, repair misspecification at another.

In practice, useful checks include comparing fitted treatment contrasts across plausible functional forms, examining whether clinically important interactions are represented, and checking outcome residual patterns by history and treatment. For decision-making, inspect the implied action regions as well as outcome prediction error: a small average prediction error does not guarantee a correct sign near a boundary. Compare fitted rules’ estimated values using an appropriate evaluation procedure, while recognizing that an evaluation estimate also has sampling uncertainty and assumptions.

These checks can reveal warning signs but cannot prove that the conditional mean model is correct. A model may fit observed outcomes adequately and still extrapolate poorly into histories with sparse treatment overlap. The projection weights $p(H)\{1-p(H)\}$ are small near deterministic treatment assignment, so the contrast is weakly informed there. Conversely, in a multivariable model whose baseline is restricted, misspecification can distort contrasts even away from such histories.

A correctly specified parametric Q-model targets the true conditional mean within its model. A more flexible outcome-regression approach can reduce functional-form error but still depends on adequate data support and estimation. Blip-focused methods such as A-learning target treatment contrasts more directly, but they do not make a misspecified contrast model correct. No method can recover a reliable decision in regions where the observed data provide essentially no information about one treatment option.

Common Pitfall

Common mistake: “The residual $Y-E[Y\mid H]$ has mean zero given $H,A$, so it is orthogonal to every function of $H,A$.” This is generally false. In fact, $$ E[Y-E[Y\mid H]\mid H,A]=\{A-p(H)\}\delta(H), $$ which need not be zero. For example, if $p(H)=1/2$, $\delta(H)=2$, and $A=1$, the conditional expectation is $1$. To derive the contrast projection correctly, decompose $Y-E[Y\mid H]$ into $Y-E[Y\mid H,A]$ plus $\{A-p(H)\}\delta(H)$. Only the first component has conditional mean zero given $(H,A)$; minimizing over the baseline then yields the weighted contrast criterion.

Check Your Understanding

Answer each question, then check the solutions below.

In the unrestricted-baseline setting, what determines the weight assigned to a history in the contrast projection?
Suppose the true blip is $-2$ at a history and the fitted blip is $0.1$. What action does the fitted rule take, and what is its conditional value loss?
Why does the weighted projection formula not generally apply when the baseline model is restricted?
If the true blip equals zero at some histories, what value penalty arises from choosing the opposite action there?

Solutions. (1) The population weight is $P(H=h)p(h)\{1-p(h)\}$, interpreted through the expectation over $H$; it reflects both how common the history is and treatment overlap. (2) The fitted rule treats, while the optimal rule does not; the conditional loss is $|-2|=2$. (3) A restricted baseline cannot absorb arbitrary differences in $E[Y\mid H]$, so the fitted contrast may also respond to baseline-model error and treatment composition. (4) None: the two actions have equal conditional value when the blip is zero.