Nonlinear Q-Models And Decision Boundaries

Why nonlinear treatment contrasts matter

A treatment effect need not change at a constant rate with a patient’s history. For example, the benefit of treatment might increase with a biomarker at first, then level off or reverse at high values. A linear Q-model can miss that pattern and impose a single straight decision boundary where the science suggests a curved one. Polynomial terms and spline bases let the treatment contrast vary nonlinearly with history, while keeping the treatment decision interpretable as a comparison of two fitted Q-values.

The central calculation is not to inspect every coefficient in a fitted model separately. It is to subtract the fitted Q-value under action 0 from the fitted Q-value under action 1. The sign of this difference determines the preferred action; its zero set is the decision boundary. Nonlinear terms can make that zero set curved, create several boundaries, or produce different decisions across regions of the same history space.

Q-model contrasts and decision boundaries

At stage \(t\), let \(H_t=h\) denote the observed history available just before treatment \(A_t\), and consider two possible actions \(a\in\{0,1\}\). The stage-specific action-value \(Q_t(h,a)\) is the expected outcome under action \(a\) at this stage and optimal treatment decisions thereafter. At the final stage, it is the conditional expected final outcome given the history and current action. In an estimated Q-learning analysis, these quantities are represented by fitted functions.

A useful parameterization makes the contrast explicit:

Here \(m_t(h)=Q_t(h,0)\), and \(\tau_t(h)=\Delta_t(h)\). Thus the main-effect function \(m_t\) affects the predicted value but cancels from the treatment comparison. One may represent \(\tau_t(h)\) using polynomial terms, spline basis functions, or interactions among history variables. Nonlinear terms in \(m_t\) alone do not change the boundary unless they also enter the treatment contrast.

For a causal interpretation, the action values must correspond to well-defined interventions, and the usual identification conditions for sequential treatment decisions are needed: consistency, sequential exchangeability given the recorded history, and positivity for the actions being compared at relevant histories. Estimation additionally requires an adequate model for the relevant Q-functions or an explicitly acknowledged approximation. Even under these conditions, a fitted boundary is meaningful only where histories are supported by the data.

Two basic consequences of the contrast

Proof. For action 1, \(Q_t(h,1)=m_t(h)+\tau_t(h)\); for action 0, \(Q_t(h,0)=m_t(h)\). Subtracting gives \(Q_t(h,1)-Q_t(h,0)=\tau_t(h)\). If this difference is positive, action 1 has the greater value; if negative, action 0 does; if zero, the two values are equal. This proves the rule. \(\square\)

A second useful fact explains how polynomial contrasts divide a one-dimensional history variable into decision regions.

Proof. A nonzero degree-\(k\) polynomial has at most \(k\) distinct roots, by the factor theorem: each distinct root contributes a distinct linear factor, and a product of more than \(k\) such factors cannot divide a degree-\(k\) polynomial. Consider an interval containing no root. Polynomials are continuous. If \(\tau\) had opposite signs at two points of that interval, the intermediate value theorem would imply a root between them, a contradiction. Therefore its sign, and hence the preferred action, is constant throughout the interval. If \(\tau\) is identically zero, every history in the interval is a tie instead. \(\square\)

This result concerns a single scalar history variable. With several history variables, the zero set may be a curve or surface, and it need not divide the supported history space into simple, easily labelled regions. The sign rule still applies point by point.

Worked examples

Interpretation, diagnostics, and alternatives

A boundary is history-dependent: it describes how the action comparison changes as the measured history changes, not a universal threshold that applies to every patient. With two history variables, for example, a boundary may say that the biomarker level at which action changes depends on age. Reporting just a single cutoff would then conceal part of the fitted rule. A clear presentation describes the contrast, the relevant range of histories, and the action on each side of the zero set.

Several diagnostics help reveal when a nonlinear rule is unreliable. First, plot or tabulate \(\hat{\tau}_t(h)\), not only the resulting action: a rule can flip because a tiny estimated contrast crosses zero. Second, check how many roots or sign changes occur within the data-supported range; extra crossings near sparse tails may reflect instability rather than a reproducible pattern. Third, examine the distribution of histories around each boundary. If one side has little or no data, the model’s decision there relies heavily on extrapolation. Finally, compare the fitted contrast with a simpler specification and check whether small changes to the basis, knot locations, or fitted coefficients move the boundary substantially.

Polynomial terms are compact and can represent smooth curvature, but high-degree polynomials may oscillate, especially near the edges of the observed range. Splines allow local flexibility and can keep the fitted contrast more stable across a broad range, but their shape depends on the basis and knot placement. A linear treatment contrast is easier to summarize and may be more stable, but it cannot represent multiple crossings or curved boundaries. These are modeling alternatives, not substitutes for the sign comparison: for any fitted Q-model, the optimal action still follows from the sign of its contrast.

The boundary is also not a direct measure of how much a patient benefits. A contrast close to zero means the fitted values are similar, even if the action label changes across a narrow boundary. Conversely, a large positive or negative contrast indicates a larger fitted value difference under the model. A boundary does not by itself establish that the treatment effects are estimated precisely, that the model is correctly specified, or that a recommended action is clinically feasible. These issues must be considered separately when interpreting a fitted regime.

Common pitfall

• Using the wrong sign or reading coefficients one at a time. A coefficient of a nonlinear term does not by itself identify the preferred action at a given history. Calculate \(\hat{\Delta}_t(h)=\hat Q_t(h,1)-\hat Q_t(h,0)\), substitute the history values, and inspect the resulting sign. Also check arithmetic when squaring or evaluating basis functions: for example, \(0.25(3^2)=2.25\), not \(2\).
• Assuming one crossing. A nonlinear contrast may have several roots. Solve for all roots in the range of interest and check the sign in each interval rather than extrapolating from one point.
• Treating a root outside support as an observed cutoff. The equation may yield a mathematical root, but it is not an empirically supported decision threshold if that history is outside the observed range.
• Calling every fitted zero a stable clinical boundary. A zero can be sensitive to model choice or estimation noise, particularly where observations are sparse. Report this limitation rather than presenting the fitted crossing as exact.