Diagnostic Checks for AIPW and TMLE Estimating Equations

Why Inspect Estimating Equations?

An AIPW or TMLE estimate can be returned by software even when a nuisance prediction was misaligned with the observations, a weight was coded with the wrong treatment group, or a numerical optimizer stopped before targeting converged. Inspecting the equations behind the estimate provides a direct way to find such problems. These checks are especially useful when cross-fitting or flexible nuisance models make the full computation difficult to inspect by eye.

A small empirical residual is evidence that a particular calculation satisfies a numerical equation; it is not evidence that the causal assumptions are true. In particular, influence-function and targeting checks cannot establish exchangeability, consistency, no interference, or positivity. They complement, rather than replace, checks of the data structure, overlap, nuisance predictions, and uncertainty calculations.

Setup and the Equations to Check

Let $O_i=(W_i,A_i,Y_i)$, $i=1,\ldots,n$, be independent observations, where $A_i\in\{0,1\}$ is treatment, $Y_i$ is the outcome, and $W_i$ is a vector of measured covariates. For the average treatment effect, define $g(W)=P(A=1\mid W)$, $Q(a,W)=E(Y\mid A=a,W)$, and $\psi=E\{Q(1,W)-Q(0,W)\}$. Identification as a causal effect requires consistency, conditional exchangeability given $W$, and positivity, $0<g(W)<1$ on the covariate values in the target population. Estimation and usual influence-function inference also require suitable regularity and finite-variance conditions.

Definition: The efficient influence function for the average treatment effect is $$ D^*(O)=\frac{A}{g(W)}\{Y-Q(1,W)\}-\frac{1-A}{1-g(W)}\{Y-Q(0,W)\}+Q(1,W)-Q(0,W)-\psi. $$ It has expectation zero under the true data distribution and the stated identification conditions. Its estimated version replaces $g,Q,\psi$ by estimates. The estimated influence-function mean is $n^{-1}\sum_{i=1}^n \widehat D_i$.

In an AIPW calculation, define each observation's uncentered contribution using estimated nuisance functions:

\widehat m_i= \widehat Q(1,W_i)-\widehat Q(0,W_i) +\frac{A_i}{\widehat g(W_i)}\{Y_i-\widehat Q(1,W_i)\} -\frac{1-A_i}{1-\widehat g(W_i)}\{Y_i-\widehat Q(0,W_i)\}.

The AIPW estimate is $\widehat\psi_{\mathrm{AIPW}}=n^{-1}\sum_i\widehat m_i$, and the corresponding centered estimated influence function is $\widehat D_i=\widehat m_i-\widehat\psi_{\mathrm{AIPW}}$. Thus its sample mean is exactly zero by construction, apart from rounding or a coding inconsistency. This identity is a useful implementation check, but it does not test whether the nuisance estimates are accurate or whether the causal assumptions hold.

For binary outcomes, a common TMLE fluctuation updates the initial outcome regression by a logistic model with an offset. With $\operatorname{logit}(x)=\log\{x/(1-x)\}$, the updated predictions are

\operatorname{logit}\{\widehat Q_\epsilon(A_i,W_i)\} =\operatorname{logit}\{\widehat Q(A_i,W_i)\}+\epsilon H_i,\qquad H_i=\frac{A_i}{\widehat g(W_i)}-\frac{1-A_i}{1-\widehat g(W_i)}.

The logistic fluctuation score for $\epsilon$ is $\sum_i H_i\{Y_i-\widehat Q_\epsilon(A_i,W_i)\}$. A converged maximum-likelihood fit with a finite solution makes this score approximately zero. The empirical efficient influence-function mean for the plug-in TMLE estimate then has a direct connection to that score: the first two terms of the influence function sum to the score, while the empirical mean of the updated outcome contrast minus the plug-in estimate is zero. Consequently, when the same observations and predictions are used in both calculations, the empirical influence-function mean should also be near zero.

Three Practical Diagnostics

The following checks are most informative when reported at the level where the estimator was actually computed: the full analysis sample, each validation fold, and, where useful, groups with different treatment probabilities. A single overall mean can conceal large positive and negative residuals that cancel.

Reconstruct the equation.
Compute the estimated influence-function contributions and targeting-score terms directly from the observation-level predictions. Verify treatment coding, denominators, and the use of the matching $Q(1,W_i)$ or $Q(0,W_i)$ prediction.

Check scale and concentration.
Record the mean, standard deviation, range, and largest absolute contributions. A near-zero mean can coexist with a few enormous terms caused by estimated propensities close to zero or one.

Check each fold and the update.
For cross-fitted estimates, inspect validation-fold estimates and influence-function means separately. For TMLE, check the score after targeting and confirm that the optimizer converged to a finite update rather than stopping at a boundary.

Worked Example: Checking an AIPW Influence-Function Mean

Consider four hypothetical observations with $\widehat g(W_i)=0.5$, $\widehat Q(1,W_i)=0.6$, and $\widehat Q(0,W_i)=0.4$ for everyone. The treated outcomes are 1 and 0; the control outcomes are 1 and 0. The estimated contrast is $0.6-0.4=0.2$. Calculate the AIPW contributions and their mean.

For a treated person with $Y=1$, the contribution is $0.2+1(1-0.6)/0.5=1.0$. For a treated person with $Y=0$, it is $0.2+1(0-0.6)/0.5=-1.0$. For a control person with $Y=1$, it is $0.2-1(1-0.4)/0.5=-1.0$. For a control person with $Y=0$, it is $0.2-1(0-0.4)/0.5=1.0$.

Therefore, $\widehat\psi_{\mathrm{AIPW}}=(1-1-1+1)/4=0$. The centered influence-function values are $1,-1,-1,1$, and their mean is $0/4=0$. The result illustrates why checking only the mean is not enough: contributions range from $-1$ to $1$, despite the mean being zero. In a real analysis, a large range could signal sampling variability, unstable weights, or an implementation error; it requires inspection, not automatic rejection.

Worked Example: Verifying a Logistic Targeting Score

Suppose four hypothetical observations have $\widehat g=0.5$ and initial outcome predictions $\widehat Q(1,W_i)=\widehat Q(0,W_i)=0.5$. Both treated observations have $Y=1$, and both controls have $Y=0$. The clever covariate is $H=2$ for treated observations and $H=-2$ for controls. At $\epsilon=0$, the score terms are

$$ 2(1-0.5)+2(1-0.5)+(-2)(0-0.5)+(-2)(0-0.5)=4. $$

The mean score is $4/4=1$, not zero. More generally, writing $q_t=\operatorname{expit}(2\epsilon)$ for the updated treated prediction gives a control prediction $q_c=\operatorname{expit}(-2\epsilon)=1-q_t$. The score sum is

$$ 2(1-q_t)+2(1-q_t)+(-2)(0-q_c)+(-2)(0-q_c) =4(1-q_t)+4q_c=8(1-q_t). $$

For every finite $\epsilon$, $q_t<1$, so this score is positive. It approaches zero only as $\epsilon$ tends to infinity. There is no finite maximizing fluctuation parameter: the data are completely separated in this one-parameter fluctuation. A diagnostic that merely reports “the optimizer stopped” would miss the issue. The diverging estimate or nonzero score should prompt an investigation of separation, numerical limits, and the fitted predictions; silently treating the update as converged is not justified.

Worked Example: A Finite Update and Its Residual Check

Keep the same four observations, initial predictions of 0.5, and propensity estimates of 0.5, but now let the treated outcomes be 1 and 1 and the control outcomes be 0 and 1. For a treated observation the score term is $2(1-q_t)$ for each of two people. The two control terms sum to $-2\{(0-q_c)+(1-q_c)\}$. Since $q_c=1-q_t$, the total score is

$$ 4(1-q_t)-2(2q_t-1)=6-8q_t. $$

Setting the score equal to zero gives $q_t=6/8=0.75$, so $\epsilon=\operatorname{logit}(0.75)/2=\log(3)/2\approx0.5493$, and $q_c=0.25$. The score terms are $0.5,0.5,0.5,-1.5$, which sum to zero. Their mean is also zero. With constant predictions across observations, the plug-in TMLE contrast is $0.75-0.25=0.50$; its empirical influence-function mean is zero because the targeting-score mean is zero and the centered plug-in contrast has mean zero. This calculation checks the score, update, and resulting contrast together.

Cross-Fitting and Fold-Specific Instability

Cross-fitting uses nuisance predictions for an observation that were estimated without using that observation, typically by dividing the sample into folds. This limits overfitting bias in suitable settings, but it does not guarantee that estimates are stable across folds. A pooled result can hide substantial variation, especially when folds contain few treated or control observations or when nuisance predictions differ considerably.

For AIPW, compute each validation-fold estimate by averaging its $\widehat m_i$ values. The overall estimate is the sample-size-weighted average of those fold estimates. Do not treat fold estimates as independent replications: their training samples overlap, and the folds are components of one cross-fitted procedure. Their purpose here is diagnostic, not a substitute variance estimator.

Worked Example: Detecting Cancellation Across Folds

Suppose two equal-sized validation folds each contain one treated and one control observation. In both folds, $\widehat g=0.5$, $\widehat Q(1,W)=0.6$, and $\widehat Q(0,W)=0.4$. In fold 1, the treated outcome is 1 and the control outcome is 0. Their AIPW contributions are $0.2+2(1-0.6)=1.0$ and $0.2-2(0-0.4)=1.0$, so the fold estimate is $1.0$. In fold 2, the outcomes are treated $0$ and control $1$, giving contributions $0.2+2(0-0.6)=-1.0$ and $0.2-2(1-0.4)=-1.0$, so its estimate is $-1.0$.

The pooled estimate is $\{2(1.0)+2(-1.0)\}/4=0$. Reporting only zero would hide the opposing fold results. With such small folds this is not evidence of a particular failure by itself, but it is a clear reason to examine fold composition, nuisance predictions, weights, and sensitivity to the fold assignment before interpreting the pooled number.

Interpreting Failures and Choosing the Right Check

The checks differ in what they can reveal. An AIPW influence-function mean centered by the AIPW estimate is an algebraic identity, so a nonzero value points first to inconsistent centering, missing observations, or a coding or rounding problem. A small value cannot validate the nuisance models: the estimating equation is centered even if the estimated contributions are highly variable. A TMLE targeting-score mean is more directly a convergence check for the chosen fluctuation, provided it is evaluated on the observations used to fit that fluctuation. For a cross-validated procedure, record exactly which observations were used in each targeting step; do not expect a score to vanish on an unrelated sample.

AIPW and TMLE have related but different diagnostics. AIPW's plug-in estimate is defined by the average of its uncentered contributions, making the centered influence-function mean zero by construction. TMLE updates the outcome predictions to satisfy a targeting equation and then forms a plug-in estimate. For a binary-outcome logistic fluctuation, checking both the targeting score and the updated predictions tests whether the numerical update accomplished its intended task. Neither approach's equation check replaces direct examination of estimated propensities, outcome predictions, or influence-function tails.

Common warning signs include a score that remains large after fitting, an optimizer reporting nonconvergence, an extremely large fluctuation parameter, and fold-specific estimates with opposing magnitudes. Also inspect the largest $|\widehat D_i|$, the distribution of $\widehat g(W_i)$, and the separate treated and control residual terms. Extreme weights can dominate the influence-function variance while their positive and negative contributions happen to cancel in the mean. Truncating propensities or changing the fluctuation can alter the estimator and its target; such changes must be documented and cannot be justified solely by obtaining a smaller residual.

Key takeaway: Verify the equation that the estimator is supposed to solve, at the same observations and with the same predictions used by the estimator. Then inspect the terms, folds, and numerical behavior: a small residual is a computational diagnostic, not proof of causal validity, model adequacy, or stable inference.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why is the centered AIPW influence-function mean exactly zero when $\widehat\psi_{\mathrm{AIPW}}$ is computed from the same $\widehat m_i$ values?
For the separated-data example, what is the logistic targeting score sum at $\epsilon=0$, and why does it indicate a problem?
Can an AIPW influence-function mean of zero rule out extreme estimated weights? Explain.
Two equal-sized folds give estimates $0.7$ and $-0.3$. What is the pooled estimate, and what diagnostic concern remains?
What must be matched when checking a TMLE targeting score in a cross-fitted analysis?

Solutions. (1) Since $\widehat D_i=\widehat m_i-\widehat\psi_{\mathrm{AIPW}}$, its mean is $n^{-1}\sum_i\widehat m_i-\widehat\psi_{\mathrm{AIPW}}=0$ by definition. (2) The score sum is $4$, with mean $1$; the nonzero value shows the update has not solved the score equation at its initial value. In that example the score approaches zero only at an infinite fluctuation parameter, so there is no finite solution. (3) No. Centering forces the mean to zero even if individual contributions are large; inspect the estimated propensities and contribution magnitudes. (4) The equal-weight pooled estimate is $(0.7-0.3)/2=0.2$, but the opposing fold-specific values warrant investigation for instability or fold-composition effects. (5) Check the score on the exact observations and with the predictions used in that targeting step; a score need not vanish on a different evaluation sample.