Nonregular Inference Near Treatment Ties

Why a Small Q-Value Gap Creates a Large Inferential Problem

A treatment rule chooses the action with the largest conditional value. When one action is clearly better, modest estimation error usually does not change the choice. When two estimated Q-values are close, however, a small perturbation can reverse their ordering. The Q-values may be estimated with ordinary, approximately normal errors while the resulting action has a nonregular sampling distribution.

The issue is not that every Q-value estimator is nonregular. It is the operation that turns those estimates into an action: taking an argmax. This tutorial studies uncertainty in that selected action, not uncertainty in the value achieved by a regime.

The earlier material on ties in optimal treatment decisions establishes that \(\mathcal A^*(x)\) can contain more than one action. Here the focus is what sampling variation does near such ties. For transparent calculations, begin with two actions and define the contrast \(\Delta(x)=Q(1,x)-Q(0,x)\). The estimated rule chooses action 1 when \(\hat\Delta(x)>0\), and action 0 when \(\hat\Delta(x)<0\).

Regular Q-Value Estimation, Irregular Action Selection

Consider independent and identically distributed observations, a fixed covariate value \(x\), and a contrast estimator satisfying \(\hat\Delta(x)\xrightarrow{p}\Delta(x)\). Assume also that, for some \(\sigma^2>0\), \(\sqrt n\{\hat\Delta(x)-\Delta(x)\}\xrightarrow{d}N(0,\sigma^2)\). These are assumptions about the contrast estimator; they do not follow merely from defining a Q-model. In applications, they require an appropriate estimation procedure and its regularity conditions.

For a nonzero fixed gap, consistency follows because the estimated contrast converges to a number with the same sign as the true contrast. At a tie, the limiting contrast error has positive probability on either side of zero. More generally, if the true contrast changes with sample size and \(\Delta_n(x)=h/\sqrt n\), and the same asymptotic approximation holds along this sequence of data-generating distributions, then \(\sqrt n\hat\Delta(x)\xrightarrow{d}N(h,\sigma^2)\). Consequently, \(P\{\hat a(x)=1\}\to\Phi(h/\sigma)\), where \(\Phi\) is the standard normal distribution function. A gap on the order of the estimator's standard error therefore does not force the selected action to be correct with probability approaching one.

This also explains a limitation of pointwise consistency. For each fixed nonzero gap, the estimated action is consistent under the stated assumptions. But over a parameter class that permits arbitrarily small gaps, the local-alternative calculation shows that no assertion of uniformly reliable action selection follows from that pointwise result. For any finite \(h\), \(\Phi(h/\sigma)<1\). Closeness to a tie must therefore be treated as an inferential concern, not dismissed because the Q-value estimates themselves look stable.

Confidence Statements About the Action

A confidence interval for \(\Delta(x)\) addresses a different question from a confidence statement about which action is optimal. If a valid interval for the contrast lies entirely above zero, it supports action 1 over action 0 at the stated confidence level; if it lies entirely below zero, it supports action 0. If it includes zero, the data do not resolve their ordering at that level. Reporting only the plug-in action discards this distinction.

With multiple actions, one can form a confidence set for the optimal action from simultaneous intervals for the Q-values. The intervals must cover all action-specific values jointly at the desired level; separate intervals with nominal coverage do not automatically provide that guarantee.

To verify the coverage claim, suppose all intervals cover and \(a^*\) is any truly optimal action. Then \(U_{a^*}(x)\ge Q(a^*,x)\). For every \(b\), \(L_b(x)\le Q(b,x)\le Q(a^*,x)\), because \(a^*\) is optimal. Hence \(U_{a^*}(x)\ge\max_b L_b(x)\), so \(a^*\in\mathcal C(x)\). The event that every optimal action is included therefore contains the event of simultaneous interval coverage, and has probability at least \(1-\alpha\). This argument does not require a unique maximizer.

Worked Decisions and Diagnostic Checks

Several checks help reveal when a decisive-looking rule may be fragile:

• Compare the gap with its uncertainty. A small estimated gap relative to its standard error is a direct warning that sampling noise can reverse the ranking.
• Inspect near-tie frequencies. Across bootstrap or resampling fits, frequent changes in the top-ranked action indicate instability. Such frequencies are diagnostics, not automatically calibrated confidence probabilities for the true action.
• Check the inferential target. A confidence interval for a Q-value contrast is not itself a confidence set for an action unless the comparison and coverage argument are made explicit.
• Account for multiple actions. Selecting the largest of many noisy estimates can produce an apparently clear winner. Pointwise intervals that ignore joint uncertainty can understate uncertainty about the maximizing action.

The smooth-contrast approach and the action-set approach answer complementary questions. A Wald interval for a contrast is useful when the target is a particular pairwise Q-value difference and its normal approximation is justified. A simultaneous Q-value confidence set communicates which actions remain compatible with being optimal while accounting for the set of actions. In contrast, simply reporting the empirical argmax gives a single operational decision, but supplies no uncertainty statement. These approaches do not repair a misspecified Q-model or weak information about some actions; their coverage depends on valid estimation and standard errors.

Common Mistake and Limitations

Wrong reasoning: “The Q-value estimator is asymptotically normal, so the estimated optimal action is also approximately normal and its standard error can be read from the Q-value fit.” This confuses a real-valued estimator with a discontinuous, categorical decision. An action has no ordinary normal standard error, and near a tie its sampling behavior can remain split between alternatives.

Correction: Make inference on a smooth contrast when the question concerns the ordering of two actions, or use a confidence set for the maximizer when several actions are possible. At a tie or near-tie, report unresolved alternatives rather than converting a small estimated ranking difference into a claim of certainty. With many actions, simultaneous coverage and the quality of the underlying Q-value estimates matter; a confidence set cannot make inadequate data or invalid modeling assumptions disappear.