Diagnostics for Interventional Mediation Analyses

Why Diagnostics Matter

Interventional mediation analyses combine a model for the outcome with a model for the mediator distribution. The resulting estimate can be sensitive to both. A mediator model that misses important variation can generate the wrong mediator draws; an outcome model evaluated at poorly supported mediator values can rely heavily on extrapolation. A small number of observations or a particular modeling choice may also dominate the result.

Diagnostics help identify these vulnerabilities and clarify what they may mean for interpretation. They do not, by themselves, establish that the causal assumptions hold. In particular, good model fit cannot demonstrate that all relevant confounders were measured, and overlap in the observed data does not establish exchangeability.

Definition: Let $A$ denote exposure, $M$ a mediator, $Y$ the outcome, and $C$ baseline covariates. Write $\mu(a,m,c)=E(Y\mid A=a,M=m,C=c)$, and let $F_{M\mid A=a^*,C=c}$ be the conditional distribution of the mediator under exposure level $a^*$. The interventional mean is $$ \psi(a,a^*)=E_C\left[\int \mu(a,m,C)\,dF_{M\mid A=a^*,C}(m)\right]. $$ The integral is with respect to the mediator distribution, so it applies to continuous and discrete mediators. For a binary mediator, it is $$ \psi(a,a^*)=E_C\left[\sum_{m\in\{0,1\}}\mu(a,m,C)P(M=m\mid A=a^*,C)\right]. $$

In practice, analysts estimate the conditional outcome mean and mediator distribution, then average predictions over a target covariate distribution. The causal interpretation additionally requires appropriate identification assumptions, including consistency, conditional exchangeability for the exposure and mediator-outcome relations, and the relevant positivity conditions. The diagnostics in this tutorial assess aspects of model adequacy and data support; they cannot prove those causal assumptions.

Check the Mediator Distribution Model

The mediator model must represent the conditional distribution needed by the intervention, not just its average. For a binary mediator, this means checking predicted probabilities against observed frequencies across exposure and covariate patterns. For a continuous mediator, inspect conditional residual patterns and whether the assumed distribution captures spread, skewness, tails, and heteroskedasticity. A model can predict the conditional mean reasonably well while getting the distribution wrong.

Useful checks include plots of observed versus predicted values, residuals against fitted values and important covariates, and comparisons of observed and simulated quantiles or tail frequencies. For a binary mediator, calibration summaries can compare predicted probabilities with observed proportions in groups of similar predicted risk. These checks should be stratified or otherwise adjusted for important covariates where feasible: an overall match can conceal poor fit in a subgroup that contributes importantly to the estimand.

Worked Example: Detecting a Poorly Fitting Continuous Mediator Model

Suppose a hypothetical analysis models a continuous mediator $M$, measured in hours, with a normal linear regression conditional on $A$ and $C$. The fitted model predicts a residual standard deviation of 4.0 hours. In a validation plot, one participant has observed $M=15.0$ hours and predicted $M=6.0$ hours.

The residual is $15.0-6.0=9.0$ hours. Its standardized value, using the model's residual standard deviation, is $$ \frac{15.0-6.0}{4.0}=2.25. $$ This observation is 2.25 model standard deviations above its prediction. That alone does not prove misspecification: unusual observations can occur under a suitable model. But suppose the analysis also finds that 14 of 100 residuals exceed 1.96 in absolute value. Under a well-calibrated normal residual model, about $100\times 0.05=5$ such residuals would be expected on average. The observed count of 14 is a warning to inspect tail behavior, covariate patterns, and possible nonlinearity.

If the normal model underestimates the mediator's tails, simulated mediator values may be too concentrated. The resulting g-computation can underrepresent mediator values that materially affect predicted outcomes. Possible next steps include checking transformations, allowing nonlinear covariate relationships, or using a more suitable conditional distribution, then repeating the diagnostics.

Assess Overlap and Simulated Support

The relevant support question is whether mediator values drawn from $F_{M\mid A=a^*,C=c}$ are adequately represented among observations that inform the outcome model at $A=a,C=c$. If the intervention frequently draws mediator values rarely or never observed at exposure level $a$, then $\mu(a,m,c)$ must be estimated by extrapolation there. Positivity is a population condition; observed overlap checks are empirical diagnostics that can reveal potential problems but cannot certify population positivity.

Inspect mediator distributions across exposure levels within important covariate strata, rather than relying only on marginal histograms. Also inspect the simulated mediator draws used in g-computation: compare their ranges, quantiles, and unusual values with the observed mediator support in the outcome-model group. For a discrete mediator, check whether the intervention assigns appreciable probability to categories with sparse outcome data. For a continuous mediator, compare the full distributions, not only their means.

Worked Example: A Mediator Intervention With Weak Outcome Support

Consider a hypothetical binary mediator and two equally common baseline strata, $C=0$ and $C=1$. The target intervention draws $M$ from its distribution under $A=0$, while the outcome model is evaluated at $A=1$. In stratum $C=0$, suppose $P(M=1\mid A=0,C=0)=0.40$, and among $A=1$ observations the corresponding proportion is 0.35. In stratum $C=1$, the target probability is 0.70, but only 5 of 100 observations with $A=1,C=1$ have $M=1$.

Suppose the fitted outcome model predicts, in stratum $C=0$, $\mu(1,0,0)=10$ and $\mu(1,1,0)=18$ outcome units. Its contribution to the interventional mean is $$ (1-0.40)(10)+(0.40)(18)=6+7.2=13.2. $$ In stratum $C=1$, suppose it predicts $\mu(1,0,1)=14$ and $\mu(1,1,1)=22$. The corresponding contribution is $$ (1-0.70)(14)+(0.70)(22)=4.2+15.4=19.6. $$ Standardizing equally over the two strata gives $\psi(1,0)=(13.2+19.6)/2=16.4$ outcome units.

The calculation is arithmetically straightforward, but 70% of the $C=1$ mediator draws ask the outcome model to predict for $M=1$, a category represented by only 5 of 100 relevant observations. The estimate may therefore be highly dependent on how the model extrapolates from sparse data. The observed proportion is not zero, so this is not proof of a positivity violation; it is a substantial support warning. Report it, assess uncertainty and model sensitivity, and avoid presenting the estimate as equally data-supported across strata.

Simulated draws deserve their own check because a fitted distribution can generate values outside the range or pattern that was apparent in the raw data. Such draws may result from a misspecified distribution, an incorrect covariate pattern, or a genuine but sparsely observed tail. The response is not automatically to truncate draws: truncation changes the intervention distribution and hence can change the estimand. First identify why the draws occur and whether the outcome model has support there.

Check Influence and Model Sensitivity

An observation is influential when its inclusion has a substantial effect on the estimated mediation contrast. Influence can arise from an unusual outcome, an extreme mediator value, an unusual covariate pattern, or a combination of these. Examine case-deletion or leave-one-out changes in the final estimate, and investigate observations flagged by residual or leverage diagnostics. If using resampling, check whether estimates are unstable across resamples or whether some fits fail. Influence is a property of the estimate and analysis, not a reason on its own to delete a data point.

Model sensitivity asks whether substantively reasonable alternatives produce materially different answers. Candidate alternatives might change functional forms, include justified exposure-by-mediator or covariate interactions, or use a different plausible distribution for a continuous mediator. Keep the estimand, target population, and covariate standardization fixed when comparing specifications; otherwise, differences can reflect a changed question rather than a changed model. Report meaningful variation rather than selecting only the specification that gives the preferred result.

Worked Example: An Influential Observation and Model Sensitivity

Suppose a hypothetical analysis estimates an interventional contrast in systolic blood pressure, in mmHg, defined as $\psi(1,1)-\psi(1,0)$. The primary model gives $\psi(1,1)=128.6$ mmHg and $\psi(1,0)=122.8$ mmHg, so the estimated contrast is $$ 128.6-122.8=5.8\text{ mmHg}. $$

A leave-one-out check identifies one observation whose removal changes the estimates to 125.0 mmHg and 122.9 mmHg. The resulting contrast is $125.0-122.9=2.1$ mmHg, a change of $2.1-5.8=-3.7$ mmHg relative to the primary estimate. This is a substantial influence signal. The observation should be checked for data errors and unusual but valid characteristics; it should not be removed solely because its removal makes the result smaller.

Next, a reasonable alternative mediator model gives $\psi(1,1)=126.1$ mmHg and $\psi(1,0)=123.1$ mmHg, for a contrast of $126.1-123.1=3.0$ mmHg. The primary, leave-one-out, and alternative-model contrasts are therefore 5.8, 2.1, and 3.0 mmHg. Their spread indicates that the estimated magnitude is sensitive to the analysis, especially to one observation. This does not determine which estimate is correct; it motivates careful reporting and investigation of the data and model structure.

Link Each Diagnostic to Its Inferential Consequence

Diagnostic warning	Likely inferential consequence	Useful response
Mediator model misses conditional spread or tails	Intervention draws may misrepresent the mediator distribution, biasing the standardized outcome mean	Assess distributional alternatives and recalibrate predictions
Little overlap in mediator values at the outcome-model exposure level	Outcome predictions rely on extrapolation and may be unstable or biased	Describe affected strata, quantify uncertainty, and avoid overstating support
Simulated mediator draws extend into sparse regions	Some parts of the estimate depend on weakly informed predictions	Trace the draws to the fitted distribution and report their contribution
One or a few observations strongly shift the contrast	The estimate may have high sensitivity to individual records	Validate records and report influence analyses without automatic deletion
Reasonable models give different contrasts	Conclusions depend on modeling choices not settled by the data alone	Report the range and explain the assumptions behind each specification

These consequences are possibilities, not automatic diagnoses of bias direction. For example, poor overlap can make an estimate unstable, but it does not tell us whether the estimate is too high or too low. Likewise, a visually acceptable fit does not rule out unmeasured confounding. Diagnostics are most useful when they are linked explicitly to the part of the estimation procedure they assess.

A Practical Diagnostic Sequence

Check the fitted conditional distributions.
Compare observed and predicted mediator behavior across relevant exposure and covariate patterns, including spread and tails where applicable.

Check overlap for the estimand.
Ask whether mediator values drawn by the intervention are represented in the data that inform the outcome predictions at the required exposure and covariate values.

Inspect the actual simulated draws.
Look for implausible values and draws concentrated in sparse regions, and determine how much they contribute to the standardized estimate.

Assess influence and alternatives.
Evaluate case-deletion changes and compare scientifically defensible model specifications while holding the target estimand fixed.

Common Mistakes

Treating good fit as proof of causal identification. A well-calibrated mediator model does not establish absence of unmeasured confounding or verify consistency.
Checking only the mediator mean. Interventional g-computation uses a mediator distribution. Correct means with incorrect spread or tails can still produce poor estimates.
Checking marginal overlap only. Substantial overall overlap can conceal weak support within important covariate strata.
Deleting an influential observation automatically. First determine whether it is erroneous. If valid, its influence is a limitation to report and investigate, not a justification for selective removal.
Calling sensitivity analysis a search for the preferred estimate. Compare defensible models to understand dependence on assumptions; do not choose a specification solely because its result is convenient.

Key takeaway: Diagnostics do not certify an interventional mediation analysis. They show where the fitted distributions, available support, individual observations, or modeling choices may drive the estimate, so the resulting uncertainty and limitations can be interpreted transparently.

Check Your Understanding

Answer each question using the diagnostic logic developed above.

Why can a mediator model with accurate conditional means still give poor interventional estimates?
What does it suggest if mediator values frequently drawn under the reference distribution are rare among observations used to fit the outcome model?
In a binary-mediator analysis, what should be compared to assess calibration across covariate patterns?
A leave-one-out analysis changes an estimated contrast substantially. Why is automatic deletion of the observation inappropriate?
When comparing alternative mediator models, why should the target estimand and standardization population be held fixed?