Reference Coding And Q-Model Contrasts

Why Reference Coding Needs Care

A linear Q-model can represent a different conditional mean outcome for each available action, while sharing information across actions through common regression terms. Reference coding expresses each non-reference action as a difference from one chosen reference action. That makes the model compact, but it also creates a common interpretive trap: a treatment coefficient by itself is generally not the treatment contrast for every history.

The contrast can depend on history whenever the model includes treatment-by-history interactions. To choose an action correctly, first recover the predicted conditional mean under each action, then compare those means. The reference action must be included in that comparison even though its reference-coded indicator is zero.

Binary Actions: From Coefficients to Conditional Means

Let \(H\) denote the history available when a decision is made, and let \(A\in\{0,1\}\) denote the action. Let \(x(H)\) be a vector of history features, such as a measured biomarker or an indicator for a clinical subgroup. For a response \(Z\), define the action-specific Q-function as \(Q(h,a)=E[Z^a\mid H=h]\), where \(Z^a\) is the response that would be observed under action \(a\). At a later decision stage, \(Z\) may be a stage-specific pseudo-outcome rather than the final outcome.

For a causal interpretation from observational data, consistency, conditional exchangeability \(Z^a\perp A\mid H\), and positivity \(P(A=a\mid H=h)>0\) for each action being compared are standard identification conditions. A correctly specified conditional-mean model and adequate support are also needed to use the fitted Q-model reliably. In a randomized study, exchangeability follows from the randomization design, subject to the design and analysis being aligned.

Set \(A=0\) and \(A=1\) separately. The resulting action-specific means and their difference are

so the treatment contrast, defined here as action 1 minus action 0, is

The common history component cancels in the contrast. Consequently, action 1 maximizes the model’s conditional mean when \(\Delta(h)>0\), action 0 maximizes it when \(\Delta(h)<0\), and both actions maximize it when \(\Delta(h)=0\). This comparison follows directly from the definition of a maximizer over the two available actions.

More Than Two Actions

For \(K\) categorical actions, label the reference action 0 and use one indicator for each of the \(K-1\) remaining actions. Let \(I(A=j)\) equal 1 when action \(j\) is taken and 0 otherwise. A linear reference-coded model can be written

The reference action’s mean is \(m_0(h)=\beta_0+\beta^\top x(h)\). For each non-reference action \(j\), define \(\Delta_j(h)=\gamma_{j0}+\gamma_j^\top x(h)\). Substitution gives \(Q(h,j)=m_0(h)+\Delta_j(h)\), whereas \(Q(h,0)=m_0(h)\). It is convenient to set \(\Delta_0(h)=0\); then all actions share the expression \(Q(h,j)=m_0(h)+\Delta_j(h)\).

Proof. For each action \(j\), the model gives \(Q(h,j)=m_0(h)+\Delta_j(h)\). The term \(m_0(h)\) is identical across all actions at the same history. Therefore, for any \(j,l\), subtracting gives \(Q(h,j)-Q(h,l)=\Delta_j(h)-\Delta_l(h)\). In particular, \(Q(h,j)\geq Q(h,l)\) exactly when \(\Delta_j(h)\geq\Delta_l(h)\). Maximizing the conditional means is therefore equivalent to maximizing the contrasts, with the reference action represented by contrast zero. Equality preserves ties. \(\square\)

Reference Choices and Coefficient Interpretation

Changing the reference action changes the coefficients used to describe the model, but not the action-specific means or the maximizing action, provided the model represents the same fitted means. This is a property of the parameterization, not evidence that one action is universally best.

Proof. Under reference \(r\), define \(\widetilde{\Delta}_j(h)=Q(h,j)-Q(h,r)\), with \(\widetilde{\Delta}_r(h)=0\). For any actions \(j,l\), \(\widetilde{\Delta}_j(h)-\widetilde{\Delta}_l(h)=[Q(h,j)-Q(h,r)]-[Q(h,l)-Q(h,r)]=Q(h,j)-Q(h,l)\). Thus every pairwise ordering is unchanged. In particular, the action with the greatest conditional mean remains the maximizer, and ties remain ties. \(\square\)

Centering history features also changes coefficient interpretation without changing the fitted contrasts. If \(u=x-c\), then \(x=u+c\); substituting this identity into the model changes the intercepts but retains the same predictions at each original history. In a binary model, the treatment main-effect coefficient is the contrast at \(x=0\). After centering at \(c\), the new treatment main-effect coefficient is the contrast at \(x=c\). This is one reason to report the contrast function, not just a coefficient table.

Diagnostics, Alternatives, and Limitations

Before using a fitted rule, calculate the action-specific predictions and their pairwise differences at representative histories. This simple check often catches errors that are hard to see in a coefficient table. For example, verify that the reference action has contrast zero, confirm the sign convention used for each difference, and check that a reported maximizer really has the largest predicted mean.

• Check the coding and rank. Confirm that each action indicator corresponds to the intended category and that the fitted design matrix identifies the model coefficients. Empty or redundant treatment-history combinations can make some effects unidentified.
• Check treatment support. If a treatment is rare or absent in histories where its rule would be applied, its conditional mean is weakly estimated or not identified there. A numerical prediction from extrapolation does not create data support.
• Check interactions and fit. If the treatment contrast varies with history, a model omitting the relevant interaction forces that contrast to be constant. Residual patterns, unstable estimates, or large changes under reasonable model specifications can signal inadequate fit.
• Check the scale. The rule maximizes the modeled conditional mean of the response on the scale represented by the Q-model. The interpretation depends on what response the model targets and how it is defined.

Reference coding is often compared with a cell-means parameterization, which models each action’s conditional mean directly, and with effect coding, which uses a different coefficient constraint. With a sufficiently flexible and identifiable model, these parameterizations can represent the same action-specific means; reference coding makes each non-reference coefficient a contrast against one chosen action. A pooled model with treatment-history interactions can share information across actions, whereas separate action-specific regressions fit each action’s mean separately. Neither parameterization nor fitting strategy removes the need for support and adequate specification.

A linear Q-model imposes linearity in its included history features. Its contrasts can therefore be wrong when the conditional means or treatment differences are inadequately represented, even if the coding is implemented correctly. The maximizing rule inherits that error. Inference about a fitted contrast must also account for the covariance among its coefficient estimates: for a contrast \(c(h)^\top\hat{\theta}\), its estimated variance is \(c(h)^\top\widehat{\operatorname{Var}}(\hat{\theta})c(h)\), not generally the variance of one coefficient alone.

Common Mistake

A frequent error is to declare the reference action optimal whenever all displayed treatment coefficients are negative, or to select the action with the largest single treatment coefficient. Those comparisons are not generally valid. With interactions, coefficients combine with the observed history; with multiple actions, each contrast is relative to the reference rather than necessarily relative to every other action. The correction is to substitute the history into every action-specific mean—or equivalently calculate every \(\Delta_j(h)\), including \(\Delta_0(h)=0\)—and then compare the resulting values. When two values tie, report both as maximizing actions rather than forcing a unique choice.