Categorical Treatments And Argmax Decisions

From a Categorical Q-Model to a Decision Rule

When treatment has more than two possible actions, the decision is not described by a single treated-versus-untreated contrast. At each history, we need to compare the conditional value of every available action. A fitted Q-model provides those action-specific predictions; an argmax decision selects an action with the largest predicted value.

Suppose the current history or covariate vector is \(H\), the available actions are the finite set \(\mathcal{A}=\{0,1,\ldots,K-1\}\), and larger outcomes are preferred. At a single decision point, write \(Y^a\) for the outcome under action \(a\). In a multistage setting, the same definitions apply to the outcome under action \(a\) followed by a specified continuation strategy.

For a causal interpretation of the observed-data regression, assume consistency, conditional exchangeability \(Y^a\perp A\mid H\), and positivity \(P(A=a\mid H=h)>0\) for each action and each history in the target population. These conditions give \(Q(h,a)=E[Y\mid H=h,A=a]\) wherever the conditional expectations are defined. If these conditions do not hold, a regression can still predict observed outcomes, but its predictions need not represent the outcomes under interventions.

Fitting and Interpreting a Model With Several Actions

A categorical action should be represented as a factor, not as a numerical dose, unless its numerical spacing and trend are scientifically justified. One common model uses action indicators and their interactions with covariates. For a scalar covariate \(X\), action 0 as the reference, and three actions, one example is:

The reference-action prediction is \(Q_\beta(x,0)=\beta_0+\beta_x x\). The other action predictions are \(Q_\beta(x,1)=\beta_0+\beta_x x+\beta_1+\gamma_1x\) and \(Q_\beta(x,2)=\beta_0+\beta_xx+\beta_2+\gamma_2x\). Thus each \(\beta_j\) is a contrast at \(x=0\), not necessarily a constant contrast across histories. The interaction coefficients allow the relative value of an action to vary with \(x\).

A fitted conditional-mean model can be estimated by regression appropriate to the outcome and sampling design. For example, least squares estimates conditional means for a continuous outcome under the usual model-fitting conditions; a correctly specified binary-outcome mean model can use a link function, with predictions returned to the outcome scale before comparing actions. Whatever estimator is used, decisions compare predicted Q-values on the same outcome scale.

Decision Regions From Pairwise Comparisons

The decision region for action \(a\) is the set of histories at which its predicted value is at least as large as that of every other action. For a fitted model, define this region by intersecting the pairwise inequalities:

When the fitted Q-model is linear in a scalar covariate, each pairwise difference is linear, so each comparison gives a threshold inequality. Intersecting these inequalities identifies the region; checking only comparisons against the reference action is not enough. An action must beat all the alternatives to be selected.

Proof. By definition, \(a^\star\) belongs to \(\operatorname{argmax}_{a\in\mathcal{A}}Q(h,a)\) exactly when its value is at least every value in the finite set. To see the decision implication directly, any other choice \(b\) has conditional mean \(Q(h,b)\leq Q(h,a^\star)\). Thus choosing an argmax gives a conditional mean no smaller than choosing any alternative at that same history. A maximum exists because a finite, nonempty set of finite real numbers has a largest member. \(\square\)

Why the Pointwise Comparison Matters

For a target distribution of histories, the value of a rule is the mean outcome under the action it assigns. If the causal Q-function is identified and integrable, this value is \(E[Q(H,d(H))]\). The pointwise comparison above explains why maximizing Q at each history is appropriate: it avoids choosing a lower-valued action for any history.

This derivation assumes both rules assign available actions and that their Q-values are integrable under the target distribution. It is a population statement about the true Q-function. Replacing \(Q\) by a fitted model creates an estimated rule, and the fitted rule need not maximize the true value.

Diagnostics, Alternatives, and Limitations

Useful checks focus on whether the action predictions and comparisons are meaningful in the histories where the rule will be used. Check that every action is coded correctly, that the model produces one prediction per action on a common outcome scale, and that comparisons are made across all actions. Examine action-specific sample sizes and covariate overlap: sparse action-history combinations make conditional predictions unstable, while histories with no observed support violate positivity for causal identification. Plotting fitted Q-values against continuous covariates can reveal implausible crossings or extrapolation beyond the observed data.

A common alternative is to fit separate outcome models by action and compare their predictions. This can be sensible, but may use data less efficiently and does not automatically solve sparse-data problems. A reference-action model instead estimates contrasts relative to one action; it is algebraically equivalent to action-specific predictions when the model is parameterized equivalently. For \(K\) actions, it requires \(K-1\) contrasts, and interactions are needed when contrasts vary across covariates.

A frequent mistake is to code actions 0, 1, and 2 as a single numeric predictor and interpret its slope as a treatment effect. That imposes a particular linear trend and treats the spacing between category labels as meaningful. Unless that structure is justified, use categorical indicators and compare the resulting action-specific predictions. Another mistake is to select the action with the largest coefficient: coefficients are generally contrasts against a reference, not the predicted values themselves.

Finally, decision regions inherit the limitations of the Q-model and the observed treatment support. Predictions may be unreliable for rare histories, and a region derived from a misspecified model may not reflect the best action under the true conditional means. The calculations here define how to make the comparison; they do not remove uncertainty in the fitted model or guarantee that the selected rule has high value.