Fit Stability Under Small Data Perturbations

Why a Converged Fit May Still Be Fragile

A GLMM optimizer can report convergence even when a small, defensible change to the data leads to a noticeably different estimate. Convergence says that the algorithm met its stopping criteria for the data and model it was given; it does not establish that the estimated parameters are insensitive to the observed cases. Fit stability under small data perturbations is a sensitivity check: keep the model specification fixed, alter the data in a controlled way, refit, and compare the estimates.

This check is especially informative when the data contain a small number of influential clusters, an unbalanced design, or weak information about a variance component. It complements the earlier material on flat likelihood directions, curvature, and boundary estimates. It does not replace those diagnostics, and a large change does not by itself identify the cause.

Model and Meaning of a Perturbation

Write clusters as \(g=1,\ldots,G\) and observations within cluster \(g\) as \(i=1,\ldots,n_g\). A GLMM specifies a conditional outcome distribution from an exponential-family model, with linear predictor \(x_{ig}^{\mathsf T}\beta+z_{ig}^{\mathsf T}b_g+o_{ig}\), where \(\beta\) is the fixed-effect vector, \(b_g\sim N(0,D)\) is a random-effect vector, and \(o_{ig}\) is an offset when present. Outcomes are conditionally independent within and across clusters given the random effects, and the random effects for distinct clusters are independent. Fitting the GLMM uses the likelihood obtained by integrating over the random effects.

Let \(\hat{\theta}\) denote the fitted parameter vector, including the fixed effects and the random-effect covariance parameters being reported. For each perturbation \(r\), record \(\hat{\theta}^{(r)}\), the corresponding estimate from the refit. A direct comparison is \(\Delta^{(r)}=\hat{\theta}^{(r)}-\hat{\theta}\). For parameters with different units, also report a scale-aware measure, such as \(\Delta^{(r)}/SE(\hat{\theta})\) for a fixed effect whose standard error is meaningful, or compare a variance parameter on a stated scale. A relative percentage change can be misleading when the original estimate is close to zero.

A perturbation is not automatically small because it changes only one row: that row could be highly informative, and an individual row may not be the independent sampling unit. Conversely, deleting or resampling a whole cluster may be a substantial change. The analyst should state exactly what was changed and why that change is a relevant stress test.

Local Sensitivity: The Role of Curvature

The local likelihood argument explains why two fits can both converge while their estimates differ. Let \(U(\theta;D)\) be the score vector for parameter \(\theta\) from data \(D\), and suppose an interior maximum-likelihood estimate satisfies \(U(\hat{\theta};D)=0\). Define the observed information \(J=-\partial U/\partial\theta^{\mathsf T}\), evaluated at the original fit. For a small data change, suppose the score at the old estimate changes by \(\Delta U_D\). A first-order Taylor expansion gives

The first equality uses the fact that the original score is zero; the minus sign comes from the definition \(J=-\partial U/\partial\theta^{\mathsf T}\). Thus, when \(J\) has a direction with small curvature, its inverse can be large in that direction. A modest score change caused by a perturbation can then produce a substantial parameter change. This is a local approximation, not a formula for an arbitrary refit: it assumes a smooth likelihood, a nearby interior solution, and a sufficiently small perturbation. It can be unreliable near a boundary or when the optimizer moves to a different solution.

The argument motivates a practical interpretation, not a universal numerical cutoff. A change should be judged against the parameter’s units, its uncertainty, the scientific scale of interest, and the perturbation’s size. For example, a coefficient shift of 0.1 log-odds units may be minor in one application and consequential in another.

Cluster-Level Perturbations and Resampling

When clusters are the independent sampling units, perturbations should usually preserve the observations grouped within each cluster. One useful diagnostic is leave-one-cluster-out refitting: omit cluster \(g\), refit the same model, and compare each resulting estimate with the full-data estimate. This identifies whether any single cluster has unusual influence. It is a stress test, not an estimate of the effect of deleting a cluster from the target population.

A second approach is cluster resampling. Sample \(G\) clusters with replacement, including every selected cluster’s observations together, and refit the model to each resample. This preserves within-cluster groupings, unlike resampling individual observations as though they were independent. The resulting estimates describe sensitivity to the particular cluster composition in these resamples. With few clusters, a resample may contain few distinct clusters, and some refits may fail or be singular; those outcomes are informative limitations, not estimates to silently discard.

Choosing and Reporting Perturbations

A perturbation should reflect a credible source of data uncertainty or a targeted influence question. Examples include correcting a verified data error, shifting a measured continuous covariate within a plausible measurement-error range, omitting one cluster at a time, or resampling whole clusters. For a binomial response recorded as a count of events out of a known number of trials, a change of one event may be meaningful; for a binary response, the corresponding change must remain zero or one. Do not alter outcomes or covariates merely to create a preferred result.

For every analysis, hold the model formula, link, random-effects structure, estimation method, and convergence tolerances fixed where possible. Otherwise, differences may reflect a changed fitting procedure rather than the data perturbation. Record convergence status, boundary or singular estimates, and whether each parameter is estimable for every refit. If the model returns multiple numerical solutions or fails on particular resamples, report that behavior rather than comparing only successful fits without explanation.

A concise report states the original estimate, perturbation rule, number of refits, range or distribution of changes, and any failed or singular fits. For a fixed-effect coefficient, report its units (for example, log-odds units) and, when useful, its change relative to the original standard error. For a variance or standard-deviation estimate, state which scale is being compared. Avoid percentage changes when the original estimate is near zero, where a small absolute difference can produce a misleadingly large percentage.

Diagnostics, Alternatives, and Limits

A large perturbation effect is a signal to investigate, not a diagnosis. Check the affected records and cluster-level covariate and outcome patterns, and compare the result with the model’s curvature and boundary diagnostics. If the concern is random-effects complexity, the earlier material on refitting after random-effects simplification provides a related model-based comparison; changing the random-effects structure is not the same as perturbing the data. If the concern is optimizer behavior, rerunning the same data from different starting values or with another optimizer probes numerical reproducibility, not data sensitivity.

These checks have limits. Sensitivity depends on the chosen perturbation, and no single scheme covers every plausible source of uncertainty. Leave-one-cluster-out results can be noisy with few clusters, while cluster resampling can yield sparse or singular resamples. A large change can arise from genuine heterogeneity, limited information, a boundary, data error, or optimization problems; the perturbation analysis alone cannot distinguish them. A stable result under the tested changes does not prove the model is correctly specified or that all relevant perturbations would also be harmless.

Common Mistake

Wrong reasoning: “The optimizer converged for the original data and every perturbed data set, so the estimates are robust.” Convergence only indicates that each run met its numerical criteria; the estimates can still vary substantially across data sets. The correction is to compare the estimates directly, quantify their changes on relevant scales, and report the perturbation scheme and all important failures. Likewise, a changed estimate does not automatically mean the original fit is invalid: it shows dependence on the particular data alteration that was tested.