Diagnose Failed Fits and Unstable Imputations

Why a sensitivity result needs a diagnostic check

A sensitivity analysis can produce different treatment-effect estimates because its missing-data assumptions differ. That variation is often the point of the analysis, not a defect. But estimates can also vary because an optimizer stopped at a poor solution, a parameter was weakly identified, an imputation chain mixed slowly, or too few imputations were generated. A numerical result alone does not tell these explanations apart.

Diagnosis therefore uses several kinds of evidence together: records from the fitting algorithm, the location and stability of parameter estimates, summaries of iterative simulation when used, the distributions of the completed data, and the Monte Carlo error from using a finite number of imputations. A warning is a reason to investigate, not automatically proof that a fit is invalid. Conversely, a software message saying “converged” is not sufficient evidence that the result is trustworthy.

Read the fitting record, not just the final estimate

For an optimizer-based selection or pattern-mixture fit, review the recorded convergence status, iteration count, objective-function value, gradient or score at the reported solution, and any warnings about the Hessian or parameter constraints. When the software provides multiple starting values, compare the resulting solutions. The essential question is whether the algorithm has reached a stable solution to the stated model, rather than merely whether it stopped.

An estimate on a constraint boundary deserves particular attention. For example, a variance estimate of zero or a probability estimated at its allowed maximum may be mathematically permitted, but standard approximations based on a well-behaved interior solution may not apply. A boundary estimate can signal weak information, a model feature unsupported by the data, or an optimization problem. It is not, by itself, evidence of software failure. Check whether the objective continues to improve toward the boundary, whether small changes in starting values change the answer, and whether standard errors or curvature diagnostics are unstable.

For simulation-based algorithms, such as iterative methods used to generate imputations, inspect trace summaries as well. A trace is the sequence of sampled values over iterations. It should fluctuate around a stable range rather than show persistent drift, long stretches with little movement, or distinct chains occupying different ranges. Summary statistics from the beginning of a chain are not a substitute for checking whether the chain has settled and explored the relevant distribution.

Inspect traces and the completed data

Trace summaries are relevant when the imputation procedure uses iterative random draws. For example, compare trace plots or iteration summaries for key model parameters, imputed-data summaries, and quantities directly related to the analysis. Look for sustained trends, unusually high autocorrelation, chains with incompatible ranges, or abrupt changes associated with initialization. A stable trace is useful evidence, but it does not prove that the model is correct or that the chain explored every relevant region.

Next examine the imputed values themselves. Check their support (the values the variable is allowed to take), ranges, means or quantiles, and distribution by treatment arm, visit, and missingness pattern as appropriate. Compare observed and imputed values in context, remembering that they need not have identical distributions: the missingness model and MNAR restriction can intentionally imply a difference. The key diagnostic question is whether the difference is consistent with the stated assumption and whether the resulting values are substantively possible.

A delta adjustment can shift imputed outcomes by design. If its direction or magnitude changes across the grid, the imputed distribution should generally reflect that change. A distribution that does not respond when it should, or shifts in the wrong direction, can indicate a coding or data-handling problem. Conversely, a distribution that moves coherently with delta is not evidence of failure merely because it differs substantially from observed outcomes. This distinction is especially important when interpreting a tipping-point display.

Quantify Monte Carlo error

With \(M\) imputations, the pooled point estimate is the average of the \(M\) completed-data estimates. Its Monte Carlo error describes how much that average might vary if the imputation process were repeated with another finite set of draws under the same model and assumptions. It is not the sampling uncertainty of the treatment effect in the target population, and it is not the difference between two MNAR scenarios.

For independent imputations with estimates \(Q_1,\ldots,Q_M\), let \(\bar{Q}\) be their mean and \(B\) their sample variance. The Monte Carlo standard error (MCSE) of \(\bar{Q}\) is estimated by \(\sqrt{B/M}\). In standard multiple-imputation pooling, \(B\) also contributes to the pooled variance, but its role there is different: it reflects between-imputation uncertainty, not just uncertainty in the finite-simulation average. If the draws are autocorrelated, the independent-draw formula can understate the MCSE; use an effective sample size or a method appropriate to the simulation procedure.

Distinguish failure from sensitivity

A practical diagnosis follows the evidence from computation to interpretation. First ask whether the model fit or imputation algorithm completed as intended. Then ask whether estimates and diagnostics are stable enough to interpret. Only after those checks should differences across sensitivity scenarios be interpreted as consequences of changing assumptions.

The alternatives are not interchangeable. A MAR analysis provides a reference under its own missingness assumption; an MNAR sensitivity grid evaluates specified departures or restrictions. Increasing \(M\) can reduce Monte Carlo error, but it cannot resolve a poorly identified model, repair incorrect code, or determine which MNAR assumption is true. Re-fitting from new starting values may reveal optimizer instability, but it does not quantify uncertainty due to missing outcomes. A sensitivity pattern that persists after computational checks should be reported as substantive dependence on assumptions.

Common mistakes and limits

• Treating “converged” as a guarantee. A status label can coexist with a boundary estimate, poor curvature, or unstable results. Review the diagnostics that support the claim.
• Calling every boundary estimate a failed fit. A boundary can be a legitimate feature of the constrained likelihood, though it may limit interpretation and invalidate routine interior approximations.
• Calling an observed–imputed difference a defect. Under MNAR assumptions, such a difference may be intentional. Check its direction, magnitude, support, and consistency with the chosen restriction.
• Increasing imputations to fix everything. More imputations reduce finite-simulation noise under suitable conditions; they do not fix misspecification, poor mixing, or an optimizer that failed.
• Confusing MCSE with the pooled standard error. MCSE describes numerical precision of the finite-imputation average. The pooled standard error describes inferential uncertainty under the model and imputation procedure.

These diagnostics cannot establish that an MNAR assumption is correct. They help establish whether the calculation is reproducible and precise enough to interpret conditional on that assumption. A stable computation can still rely on a scientifically implausible restriction, while a plausible restriction can be implemented incorrectly. Report material warnings, boundary behavior, imputation checks, and MCSE alongside the sensitivity results.