Treatment-Covariate Interactions In Q Models

Why Interactions Matter in a Q-Model

A treatment that improves average outcomes can still be a poor choice for some histories and a good choice for others. A Q-model can represent this heterogeneity by allowing the predicted outcome under treatment to depend on covariates measured before treatment. The central task is not merely to include a treatment indicator, but to let the treatment contrast change with history when the scientific setting supports it.

Focus on a terminal treatment decision. Let \(H\) denote the observed history just before treatment \(A\), and let \(Y\) be the subsequent outcome, with larger values preferred. The terminal-stage Q-function is the conditional mean outcome given history and treatment. Treatment-covariate interactions extend its model so that two histories can have different predicted advantages for treatment.

For causal interpretation, assume consistency, conditional exchangeability \(Y^a \perp A \mid H\), and positivity: each treatment being compared has positive probability at histories in the target population. Under these conditions, \(E[Y^a\mid H=h]=E[Y\mid A=a,H=h]\) for histories where the conditional quantities are defined. A correctly specified Q-model for the observed conditional mean can therefore represent history-specific causal means. These assumptions do not follow from adding interaction terms; they are needed to interpret the fitted comparisons causally.

Binary Covariate: What the Interaction Coefficient Encodes

Suppose \(A\) and a covariate \(X\) are binary, coded as 0 and 1. The model below includes a treatment main effect and a treatment-by-covariate interaction:

The interaction lets the treatment contrast differ between \(X=0\) and \(X=1\). The coefficient \(\beta_a\) is the contrast at the reference value \(X=0\), while \(\beta_{ax}\) is how much that contrast changes when \(X\) changes from 0 to 1. The interaction coefficient alone is not the treatment effect in either subgroup unless the other relevant terms are accounted for.

To verify the result, substitute each of the four possible \((x,a)\) pairs into the model. At \(X=0\), subtract the fitted value for \(A=0\) from that for \(A=1\): \((\beta_0+\beta_a)-\beta_0=\beta_a\). At \(X=1\), the same subtraction is \((\beta_0+\beta_x+\beta_a+\beta_{ax})-(\beta_0+\beta_x)=\beta_a+\beta_{ax}\). This also shows how a fully interacted model for binary \(A\) and \(X\) represents four cell means: its four coefficients can be recovered from those means by subtraction. The treatment contrast changes sign or magnitude across the two histories only if the corresponding fitted cell means support that conclusion. \(\square\)

Continuous Covariates, Centering, and Decisions

For a continuous covariate \(Z\), a linear interaction model might be \(\hat Q(z,a)=\hat\beta_0+\hat\beta_z z+\hat\beta_a a+\hat\beta_{az}az\). The fitted treatment contrast then varies linearly with \(z\). This is a modeling assumption: it says the contrast changes at a constant rate on the scale of the conditional mean. It may be useful over a restricted range, but it should not be treated as a universal feature of the data.

Centering can make a main effect easier to interpret without changing fitted predictions. If \(W=Z-c\), then \(Z=W+c\). Substitution into the model gives

The coefficients on the intercept and treatment term change, but the fitted value for every \((Z,A)\) stays the same because the equation is an identity. In the centered model, the treatment coefficient is the fitted contrast at \(Z=c\); in the original model, it is the contrast at \(Z=0\). Thus, interpreting a main effect without checking the covariate coding and reference value can be misleading.

From Fitted Q-Values to a History-Specific Rule

For a terminal decision with two available actions, a fitted rule can be defined by choosing the action with the larger fitted Q-value at the observed history. The rule must be measurable in \(H\), so that it specifies an action from information available before treatment. More importantly, its optimality claim requires the fitted Q-function to represent the relevant conditional means; a convenient interaction model alone does not establish this.

To prove this, write the value of any rule \(d\) as \(V(d)=E[Y^{d(H)}]\). By consistency and conditional exchangeability, conditioning on \(H\) identifies the conditional mean outcome under the rule as \(Q(H,d(H))\); hence the law of iterated expectations gives \(V(d)=E[Q(H,d(H))]\). By definition of \(d^*\), for every history where the conditional means are defined, \(Q(H,d^*(H))\geq Q(H,d(H))\). Integrability permits taking expectations of both sides, yielding \(V(d^*)\geq V(d)\). The argument compares the true Q-values; when estimated Q-values are used, the resulting decision is a plug-in rule and need not be optimal if the model or estimates are inaccurate. \(\square\)

This result explains why interactions matter for decisions: a model that forces a constant contrast may choose the same action at every history, even when the true conditional means favor different actions. It does not imply that every fitted interaction is useful or causal. A decision based on a noisy, misspecified, or confounded fit can be worse than a simpler rule.

More Than Two History Groups

Interactions can also be constructed for a categorical history variable with several levels. For example, use indicator variables for the non-reference levels and interact each indicator with treatment. The resulting fitted treatment comparisons are allowed to differ across levels, while the reference group’s comparison is represented by the treatment main effect.

Diagnostics, Alternatives, and Limitations

A no-interaction model is a simpler alternative: it assumes that treatment has the same conditional mean contrast at every history on the modeled scale. A fully stratified approach estimates separate conditional means in each history group and can avoid imposing a linear interaction pattern, but may use information less efficiently and become unstable in small groups. A regression model with interactions shares information across histories according to its chosen functional form. These are modeling choices, not guarantees of causal validity.

Before using an interaction-based rule, check that treatment and covariate coding match the intended reference groups, and calculate fitted Q-values for plausible histories rather than interpreting coefficients in isolation. Examine whether the model’s predicted outcomes are plausible across the observed covariate range, whether important history groups have sparse treatment data, and whether the fitted contrast changes in a way supported by the data. Extrapolated thresholds, extreme predictions, and large changes driven by a few observations are warnings. Lack of treatment overlap at some histories undermines the comparison there; adding an interaction cannot create information where a treatment is absent.

The rule is also limited by the chosen outcome scale, functional form, and available history variables. Linear interactions impose a specific pattern of change, and omitted effect modifiers or nonlinear relationships may not be represented. As in the introductory material on Q-functions and optimal actions, the decision is defined by comparing conditional values; here the important extra caution is that those values depend on a fitted model. Estimating a flexible model does not remove the need for identification assumptions, adequate data, and careful assessment of predictions.