Two Different Targets in Plug-In Q-Learning
A Q-function answers a value question: at a given history, what is the expected outcome under each available action, with later decisions chosen optimally? A treatment rule answers a decision question: which action should be taken at that history? Plug-in Q-learning estimates the first object and selects the action with the largest estimated value. The two targets are related, but they are not interchangeable.
For example, if two actions have nearly equal true values, a very small estimation error can reverse their estimated ranking. The estimated values can therefore be close to the truth while the selected action is wrong. Conversely, an estimate can preserve the ranking of actions while getting their values badly wrong. To make the distinction precise, consider one decision stage with a history \(H\) and a finite action set \(\mathcal A\). Write \(Q(h,a)\) for the true conditional value of action \(a\) at history \(h\), and \(\hat Q_n(h,a)\) for its estimate based on a sample of size \(n\).
Value consistency: The estimated values are value-consistent in \(L^1\) if $$ \|\hat Q_n-Q\|_{1,H} = E_H\!\left[\max_{a\in\mathcal A}|\hat Q_n(H,a)-Q(H,a)|\right] \xrightarrow{p}0. $$ Here \(E_H\) averages over the target population distribution of histories, conditional on the fitted estimates. Decision consistency: The plug-in rule is decision-consistent if $$ P_H\{\hat d_n(H)\ne d^*(H)\mid \hat Q_n\}\xrightarrow{p}0. $$ The latter definition assumes that the optimal action is unique with probability one under the target distribution of \(H\).
These are population-history criteria: they ask how the fitted rule performs across histories drawn from the target population, not whether it classifies every conceivable history correctly. The value criterion uses the largest action-specific estimation error at each history; it does not merely average errors over the actions actually selected. The definition of decision consistency also makes clear why ties require care: when several actions are truly optimal, disagreement with one chosen label need not mean a worse decision. The consequences of ties are a separate issue.
How Value Errors Affect Action Rankings
The key quantity for a decision is not the absolute size of the Q-values but the separation between the best and second-best actions. Define the optimal-action gap at a history with a unique optimum by $$ \Delta(h)=Q(h,d^*(h))-\max_{a\ne d^*(h)}Q(h,a). $$ A large gap can protect the decision from estimation error. A small gap leaves the ranking vulnerable.
Proof. For any competing action \(a\ne d^*(h)\), the assumed error bound gives $$ \hat Q_n(h,d^*(h))\ge Q(h,d^*(h))-\varepsilon $$ and $$ \hat Q_n(h,a)\le Q(h,a)+\varepsilon. $$ Subtracting the second inequality from the first, $$ \hat Q_n(h,d^*(h))-\hat Q_n(h,a) \ge Q(h,d^*(h))-Q(h,a)-2\varepsilon \ge \Delta(h)-2\varepsilon>0. $$ Thus the estimated value of the unique optimal action exceeds that of every competitor, so the plug-in rule selects it. \(\square\)
The theorem gives a pointwise guarantee, but value consistency is usually an average-over-histories statement. The next result shows why that weaker kind of accuracy can still suffice for decision consistency, provided the true optimum is unique almost everywhere.
Proof. Let \(e_n(h)=\max_{a\in\mathcal A}|\hat Q_n(h,a)-Q(h,a)|\). The pointwise theorem implies that a decision error can occur only if \(\Delta(h)\le 2e_n(h)\). Fix any \(\delta>0\). At histories with \(\Delta(h)>\delta\), an error therefore requires \(e_n(h)\ge\delta/2\). Markov's inequality, applied over the population distribution of histories conditional on the fitted estimates, gives $$ P_H\{\hat d_n(H)\ne d^*(H)\mid\hat Q_n\} \le P_H\{\Delta(H)\le\delta\} +\frac{2}{\delta}\|\hat Q_n-Q\|_{1,H}. $$ Since the optimum is unique almost surely, \(\Delta(H)>0\) almost surely. Consequently \(P_H\{\Delta(H)\le\delta\}\to 0\) as \(\delta\downarrow0\): the events decrease to \(\{\Delta(H)=0\}\), a probability-zero event. Given any positive tolerance, first choose \(\delta\) so that the first term is below half that tolerance. For this fixed \(\delta\), value consistency makes the second term smaller than the other half with probability tending to one. This proves the claimed convergence in probability. \(\square\)
The proof does not assert that the estimated action is correct at every history in a finite sample. It says that errors become rare in the target population. The uniqueness assumption matters: it ensures that histories with arbitrarily small gaps become rare as the gap threshold shrinks. Value consistency alone does not distinguish between actions that are exactly tied.
Worked Examples
Worked Example: A Small Error Reverses a Close Decision
At one history, suppose two actions have true values \(Q(h,A)=10.00\) and \(Q(h,B)=10.04\). The unique optimal action is \(B\), and the gap is \(10.04-10.00=0.04\). Suppose the fitted values are \(\hat Q(h,A)=10.03\) and \(\hat Q(h,B)=10.01\).
The estimated error for \(A\) is \(|10.03-10.00|=0.03\), and for \(B\) it is \(|10.01-10.04|=0.03\). Thus the largest value error is \(0.03\), which is small relative to the values themselves. Nevertheless, the estimated ranking is reversed: \(10.03>10.01\), so the plug-in rule chooses \(A\).
The gap condition explains why the guarantee does not apply. Here \(2\varepsilon=0.06\) exceeds the true gap \(0.04\). The decision is wrong even though both conditional values are estimated within \(0.03\).
Worked Example: Good Average Value Accuracy With Some Wrong Actions
Suppose the population has two types of histories. Type 1 occurs with probability \(0.90\), and type 2 with probability \(0.10\). There are two actions, \(A\) and \(B\). At type 1, let their true values be \(Q(A)=0\) and \(Q(B)=4\). At type 2, let them be \(Q(A)=0\) and \(Q(B)=0.02\). Thus \(B\) is uniquely optimal at both types.
Suppose the fitted values are exact at type 1. At type 2, suppose they are \(\hat Q(A)=0.011\) and \(\hat Q(B)=0.009\). The estimated rule chooses \(A\) at type 2, so its population decision-error probability is \(0.10\).
At type 2, the absolute errors are \(0.011\) for \(A\) and \(|0.009-0.02|=0.011\) for \(B\). The maximum error there is \(0.011\); at type 1 it is zero. Therefore the population \(L^1\) error is $$ 0.90(0)+0.10(0.011)=0.0011. $$ The value errors are small on average, but the action is wrong for one tenth of the population because the gap there is only \(0.02\). This finite-sample example does not contradict the theorem: it illustrates that a small error norm alone does not promise a small decision error at a particular sample size.
Worked Example: A Correct Rule With Inaccurate Values
At every history, suppose the true values are \(Q(A)=2\) and \(Q(B)=1\). The optimal rule always chooses \(A\). Now consider fitted values \(\hat Q_n(A)=n+2\) and \(\hat Q_n(B)=n+1\).
For every \(n\), the fitted ranking is correct because \((n+2)-(n+1)=1>0\). Hence the decision-error probability is zero. But the absolute error for each action is \(n\), so the value error does not converge to zero; in fact it diverges. This example verifies that decision consistency does not imply value consistency.
A rule needs the right ordering to choose an action; it does not need numerically accurate values. A value estimate, by contrast, is intended to recover the scale and level of conditional outcomes as well as their ordering.
Interpretation, Diagnostics, and Limitations
Value estimation and action classification should be assessed according to the scientific target. If the aim is to report expected outcomes under each treatment, value accuracy matters directly. If the aim is to assign treatment, the ranking of the actions matters most. In plug-in Q-learning, one estimate is used for both purposes, but success at one purpose is not a substitute for success at the other.
A practical diagnostic is to examine estimated action gaps across histories. A small gap is a warning that modest changes in the fitted Q-values could alter the selected treatment. One can also perturb the fitted model in plausible ways and check whether the selected actions change in parts of the history space. Such instability is evidence that the decisions are sensitive to estimation, not proof that a particular action is wrong. Conversely, stable decisions do not establish that the estimated conditional values are accurate.
The main comparison is between evaluating value estimates and evaluating the induced rule. Value-oriented checks examine prediction errors or calibration of conditional outcomes. Decision-oriented checks examine which action is selected and how sensitive that selection is. Neither check answers every question: low prediction error need not mean stable decisions near small gaps, while a stable rule can coexist with large errors shared across actions.
The implication proved above is population-based and relies on a finite action set, convergence of the integrated maximum absolute error, and a unique optimum almost everywhere. It does not guarantee correct decisions at every history, nor does it say that finite-sample value accuracy will always yield a useful rule. In multiple stages, the same distinction applies to each stage-specific Q-function and its induced decision; errors at one stage can also affect the histories encountered later. These observations concern the targets and sensitivity of plug-in decisions, not a correction for misspecified Q-models.
Check Your Understanding
Answer each question, then compare with the solutions below.
- At a history, the true values are \(Q(A)=5.0\) and \(Q(B)=5.6\). If each estimated value has absolute error at most \(0.2\), what does the gap bound imply?
- Why does the value-to-decision consistency result require a unique optimal action almost everywhere?
- Give a reason that a decision-consistent sequence of rules need not be value-consistent.
- In a population with a positive-probability group whose action gap is very small, what should a small average Q-value error make you cautious about?
Solutions. (1) The gap is \(0.6\), while twice the maximum error is \(0.4\). Since \(0.6>0.4\), the estimated action must be \(B\). (2) Uniqueness ensures the gap is positive almost surely, so the population fraction of histories with gaps below a shrinking threshold tends to zero. With ties, an arbitrarily small perturbation can change the selected label without making the decision inferior. (3) Add the same diverging constant to every action value: the ranking, and hence the rule, is unchanged, but the estimated values are not close to the true values. (4) Some decisions may still be reversed in that group; average value accuracy does not by itself protect actions where the gap is small.