Binary-Treatment Blip Functions In Q-Learning

From Two Q-Values to One Decision Contrast

At a decision point, a binary treatment offers two actions, coded 0 and 1. The Q-function assigns a conditional expected utility to each action at the current history, accounting for the rewards and decisions that follow. A decision can therefore be made by comparing two quantities. The blip function compresses that comparison into a single signed contrast: positive favors action 1, negative favors action 0, and zero means the two actions have equal Q-value at that history.

The contrast is conditional on history, not necessarily a population-average treatment effect. Different histories can have different blip values, and therefore favor different actions. The definition below uses the convention “action 1 minus action 0”; reversing the order reverses every sign-based rule.

For a terminal stage, \(Q_t(h,a)\) is simply the conditional expected terminal utility given the action. At an earlier stage, it includes the immediate reward and the value of subsequent decisions. Thus a blip is not necessarily the contrast in the immediate response alone. Its meaning depends on the Q-functions being compared, including their future-decision convention.

Why the Sign Selects an Action

At a fixed history, choosing action 1 gives value \(Q_t(h,1)\), while choosing action 0 gives value \(Q_t(h,0)\). Since the blip is their difference, the larger value can be selected without comparing two separate expressions each time.

Proof. If \(\Delta_t(h)>0\), then \(Q_t(h,1)-Q_t(h,0)>0\), so \(Q_t(h,1)>Q_t(h,0)\) and action 1 maximizes the Q-value. If \(\Delta_t(h)<0\), the inequality reverses and action 0 maximizes it. If \(\Delta_t(h)=0\), the two Q-values are equal, so both actions maximize them. This proves the rule. \(\square\)

The rule is pointwise: it applies separately at each history where the Q-values are defined. If histories are continuous, conditional expectations and Q-functions are generally defined only up to sets of probability zero. Consequently, a decision rule is interpreted almost surely under the relevant history distribution; its values at unsupported or probability-zero histories need not be identified by the data.

Blips in Treatment-Interaction Models

A treatment-interaction model makes the history dependence of the contrast explicit. For a scalar history summary \(x\), consider the working model $$ Q(x,a)=\beta_0+\beta_x x+a(\beta_a+\beta_{ax}x), \qquad a\in\{0,1\}. $$ Substituting \(a=1\) and \(a=0\), then subtracting, gives $$ \Delta(x)=\{\beta_0+\beta_x x+\beta_a+\beta_{ax}x\} -\{\beta_0+\beta_x x\} =\beta_a+\beta_{ax}x. $$ The terms common to both actions cancel. The interaction allows the preferred action to change across values of \(x\); without it, this model constrains the blip to the constant \(\beta_a\). This algebra describes the model’s implied contrast, whether or not the model is correctly specified.

Continuation Values Are Part of the Contrast

In a multistage problem, the current action can change both immediate reward and the distribution of later histories. The blip must compare the resulting total Q-values, including the continuation value under optimal later decisions. The following finite example makes that contribution explicit.

When Does the Contrast Have a Causal Interpretation?

The sign rule is algebraic once the Q-values are defined. Interpreting those Q-values as the outcomes that would result from intervening on treatment requires additional assumptions when treatment is assigned observationally. A standard identification argument uses consistency, sequential exchangeability, and positivity at each decision point.

Result and derivation. Under these assumptions, the observed-data recursion identifies the optimal Q-functions. Write \(R_t\) for the reward at stage \(t\), \(H_{t+1}\) for the next history, and \(V_{t+1}(h')=\max_{a'}Q_{t+1}(h',a')\), with terminal continuation value zero after the last reward. Then $$ Q_t(h,a)=E\!\left[R_t+V_{t+1}(H_{t+1})\mid H_t=h,A_t=a\right]. $$ To justify the recursion, begin at the final stage. Given \(H_T=h,A_T=a\), consistency equates the observed reward with the reward under action \(a\). Sequential exchangeability makes its conditional distribution among those receiving \(a\) equal to the corresponding intervention distribution, and positivity ensures that this conditional distribution is supported by observed data. Hence the conditional mean identifies the final-stage causal Q-value.

Now suppose the recursion identifies the optimal continuation value from stage \(t+1\) onward. At stage \(t\), conditional on \(H_t=h\) and setting \(A_t=a\), consistency and sequential exchangeability identify the distribution of \((R_t,H_{t+1})\) by its observed conditional distribution given \(H_t=h,A_t=a\). For each next history, the induction hypothesis identifies \(V_{t+1}(H_{t+1})\). Taking the conditional expectation of the immediate reward plus that continuation value therefore gives the causal expected utility for action \(a\) followed by optimal future decisions. This is exactly the displayed recursion. Backward induction establishes the claim for every stage, and the difference of the two identified Q-values is the causal blip. \(\square\)

Diagnostics, Alternatives, and Limitations

A model-based blip is useful because it turns fitted Q-values into an interpretable decision contrast. It also makes certain problems visible. Check that treatment coding matches the stated contrast, that the fitted Q-model includes the interactions needed to represent relevant history-dependent differences, and that both actions are represented at histories where a decision is to be made. In an observational data set, sparse or absent treatment groups can make one side of the contrast extrapolative. A numerically precise fitted blip does not repair a positivity violation.

A second warning concerns decisions near zero. If a blip estimate is small, modest estimation error can change its sign and hence the selected action. Inspect the scale and uncertainty of the contrast rather than treating every positive or negative fitted value as equally persuasive. Also check whether predicted Q-values and blips are plausible over the history range used for decisions; an interaction model can yield extreme extrapolations outside the data support.

Q-learning builds conditional Q-functions and derives a rule from their contrast. G-computation also evaluates strategies using conditional outcome models, while inverse-probability-weighted methods use treatment probabilities to reweight observed outcomes. Marginal structural models generally describe marginal contrasts across treatment strategies rather than a history-specific Q-contrast. These approaches target related causal questions but rely on different modeling and weighting structures; none makes unsupported treatment comparisons empirically identified.

A blip is only as meaningful as its Q-functions: it is not automatically an immediate treatment effect, a marginal effect, or a guaranteed benefit for every individual. The sign identifies the maximizing action for the defined conditional expected utility, subject to the assumptions and model used. If the conditional means are misspecified, the sign of the fitted contrast can select the wrong action even when the subtraction is correct.

Common Mistakes

• Reversing the contrast. With \(\Delta=Q(1)-Q(0)\), a positive value favors action 1. The opposite convention requires reversing the decision rule too.
• Calling a zero blip a positive or negative preference. Zero means the two Q-values tie; it does not uniquely determine an action.
• Using only immediate reward in a multistage setting. The Q-contrast includes how the action changes future histories and their optimal continuation values.
• Treating observed conditional means as automatically causal. Consistency, sequential exchangeability, and positivity are needed for the observational-data identification argument.