Information Criteria for Comparing MNAR Candidates

Why Use Information Criteria for MNAR Candidates?

When plausible MNAR models give different conclusions, a fit statistic can help organize the comparison. Akaike’s information criterion (AIC) and the Bayesian information criterion (BIC) combine a measure of fit with a penalty for model complexity. A lower value is preferred within a carefully defined set of comparable candidates. These criteria do not determine which missing-data assumption is true.

For an MNAR model, the likelihood is typically an observed-data likelihood: it accounts for the outcomes and missingness information actually observed while integrating over outcomes that are missing. The model’s assumptions about those missing outcomes affect this likelihood and the resulting estimates. Information criteria compare the candidate likelihoods and their parameter counts; they cannot use unobserved outcomes to confirm that a candidate’s extrapolation is correct.

Definition: For a candidate model fitted by maximum likelihood, let $\ell$ be its maximized log-likelihood, $k$ the number of estimated parameters in that likelihood, and $n$ the number of independent observational units represented in the likelihood. Define $$ \mathrm{AIC}=-2\ell+2k,\qquad \mathrm{BIC}=-2\ell+k\log(n). $$ When candidates are comparable, smaller AIC or BIC indicates a preferred balance of fit and complexity under that criterion.

The log-likelihood is usually negative, so a larger (less negative) $\ell$ indicates better fit to the data under that likelihood. The term $-2\ell$ converts this fit measure to a deviance scale; the added term penalizes model complexity. AIC’s penalty is $2k$, whereas BIC’s is $k\log(n)$. For $n>e^2$, the BIC penalty per parameter exceeds the AIC penalty.

In longitudinal studies, $n$ is generally the number of independent participants, not the total number of outcome measurements or the number of observed participant-visits. BIC’s sample-size convention should match the independent units that contribute to the likelihood. The same convention must be used for every candidate in the comparison.

Conditions for a Meaningful Comparison

AIC and BIC are meaningful only when the quantities being compared refer to the same observed-data target. Candidate models should be fitted to the same participants and observed data, use likelihoods on the same scale, and represent the same likelihood contribution—not, for example, one candidate’s observed-data likelihood against another’s complete-data likelihood. Differences in the handling of missing outcomes may be part of the candidate assumptions, but the likelihood being scored must still be defined comparably.

The parameter count must include all freely estimated parameters that contribute to the candidate likelihood, including outcome-model, missingness-model, variance, and covariance parameters as applicable. Do not count a parameter fixed in advance as estimated; do not omit an estimated nuisance parameter merely because it is not central to the scientific question. The reported log-likelihood should be the maximized value from a converged fit. If a model is fitted by a method for which the reported objective is not a comparable maximum likelihood, the usual formulas may not be appropriate without a justified adaptation.

These criteria also rely on regularity conditions for their usual theoretical motivation: the model should be sufficiently well behaved, with an identifiable parameterization and a genuine maximum away from problematic boundaries. Mixtures, weak identification, or parameters on boundaries can make the usual penalty less reliable. Such issues deserve investigation rather than being hidden by a single score.

How the Penalties Arise

AIC’s penalty has a predictive interpretation. Even when a model is correctly specified and regular, fitting parameters to a data set tends to make its in-sample fit look more favorable than its fit to new data. In the usual large-sample approximation, estimating $k$ parameters contributes an optimism of about $k$ to the log-likelihood scale. Multiplying by $-2$ gives an optimism correction of about $2k$, leading to AIC.

BIC uses a stronger complexity penalty that increases with sample size. Its usual large-sample form is motivated by an approximation to a model’s integrated likelihood under regularity conditions and specified prior assumptions. Here, the practical point is that BIC may favor a simpler candidate more strongly than AIC. Neither criterion is a probability that a model is true, and the BIC calculation alone does not constitute a full model-evidence analysis.

Justification: Suppose two regular candidates are fitted to the same data, and the more complex candidate adds $\Delta k$ parameters while improving the maximized log-likelihood by $\Delta\ell>0$. The complex candidate has lower AIC exactly when $$ -2(\ell+\Delta\ell)+2(k+\Delta k)<-2\ell+2k \quad\Longleftrightarrow\quad 2\Delta\ell>2\Delta k. $$ For BIC, the corresponding condition is $2\Delta\ell>\Delta k\log(n)$. Thus a better fit must compensate for the criterion’s complexity penalty.

Calculating and Interpreting the Scores

Worked Example: A Small AIC Difference

Suppose two MNAR candidates are fitted to the same observed outcomes and missingness records from $n=240$ independent participants. Both use comparable maximized observed-data likelihoods and satisfy the fitting assumptions for this comparison. Candidate A has $\ell=-512.4$ and $k=6$; Candidate B has $\ell=-508.9$ and $k=10$.

For A, $-2\ell=1024.8$, so $$ \mathrm{AIC}_A=1024.8+2(6)=1036.8. $$ For B, $-2\ell=1017.8$, so $$ \mathrm{AIC}_B=1017.8+2(10)=1037.8. $$ A has the lower AIC by $1.0$, despite B’s better unpenalized fit. The improvement in B’s fit does not quite compensate for its four additional parameters under AIC.

For BIC, $\log(240)\approx5.4806$. Thus $$ \mathrm{BIC}_A=1024.8+6(5.4806)\approx1057.7,\qquad \mathrm{BIC}_B=1017.8+10(5.4806)\approx1072.6. $$ BIC also favors A, by about $14.9$ points. The numerical sizes of AIC and BIC are not themselves meaningful in isolation; it is their differences for these comparable candidates that are useful.

A score difference is usually reported relative to the lowest-scoring candidate: $\Delta_i=\mathrm{IC}_i-\min_j(\mathrm{IC}_j)$, where $\mathrm{IC}$ is AIC or BIC. A difference of zero marks the best score in the specified set; positive differences indicate worse scores relative to that minimum. Such differences are not p-values, do not test a null hypothesis, and do not give the probability that one candidate is correct.

Worked Example: AIC and BIC Can Rank Candidates Differently

Consider two candidates fitted to the same observed data from $n=100$ independent participants. Their observed-data likelihoods are comparable and maximized. The simpler candidate has $\ell=-205$ and $k=5$; the more complex candidate has $\ell=-201$ and $k=8$. The complex candidate improves the log-likelihood by $4$, adding three parameters.

For AIC, $$ \mathrm{AIC}_{\text{simple}}=410+2(5)=420,\qquad \mathrm{AIC}_{\text{complex}}=402+2(8)=418. $$ AIC favors the complex candidate by $2$ points because $2\Delta\ell=8$ exceeds its added penalty $2\Delta k=6$.

Since $\log(100)\approx4.6052$, the BIC values are $$ \mathrm{BIC}_{\text{simple}}=410+5(4.6052)\approx433.0, $$ $$ \mathrm{BIC}_{\text{complex}}=402+8(4.6052)\approx438.8. $$ BIC favors the simpler candidate by about $5.8$ points: the improvement of $8$ on the $-2\ell$ scale is smaller than the added BIC penalty $3\log(100)\approx13.82$. This disagreement is a consequence of the criteria’s different penalties, not a calculation error.

A Better Score Does Not Verify an MNAR Extrapolation

In an MNAR model, the distribution of an outcome among participants with missing outcomes is not directly observed. A selection model or pattern-mixture model supplies assumptions to extrapolate into that unobserved part of the data. Different extrapolations may yield different treatment-effect estimates even when the candidates fit the observed information similarly.

Information criteria can rank candidates by their penalized fit to the observed data under the specified likelihood. They cannot establish that missing outcomes follow the assumed distribution, that the missingness mechanism has the assumed form, or that a preferred treatment-effect estimate is unbiased. A lower score is evidence of a better score under the chosen criterion and candidate set—not proof of a correct missingness mechanism.

Worked Example: A Lower Score Is Not a Mechanism Test

Suppose two plausible MNAR candidates are fitted to the same observed records for $n=180$ participants. Both fits converge, use the same observed-data likelihood target, and count all estimated parameters. Candidate S, with an outcome-dependent selection model, has $\ell=-367.2$ and $k=9$. Candidate L, with a lagged-outcome dropout dependence, has $\ell=-369.0$ and $k=7$.

Their AIC values are $$ \mathrm{AIC}_S=734.4+18=752.4,\qquad \mathrm{AIC}_L=738.0+14=752.0. $$ AIC favors L by $0.4$ points. With $\log(180)\approx5.1930$, their BIC values are $$ \mathrm{BIC}_S=734.4+9(5.1930)\approx781.1,\qquad \mathrm{BIC}_L=738.0+7(5.1930)\approx774.4. $$ BIC favors L by about $6.8$ points.

These results say that L has the lower score under both criteria for this fitted comparison. They do not show that dropout truly depends on the lagged outcome rather than the current outcome, because the missing outcomes needed to check that distinction are unavailable. If the candidates imply different treatment effects, that disagreement remains relevant to interpretation even though one candidate has lower scores.

Diagnostics, Alternatives, and Limitations

Before interpreting a ranking, check the mechanics of the comparison. Verify that all candidates used the same analysis population, outcome and missingness records, and observed-data likelihood definition. Confirm convergence and inspect warnings, parameter estimates, standard errors, and boundary estimates. Recalculate the parameter count and the $n$ used for BIC. A small arithmetic or bookkeeping error can change the ranking when score differences are small.

AIC and BIC are not substitutes for examining whether candidate models reproduce relevant features of the observed data or for asking whether their extrapolations are scientifically plausible. Residual checks, posterior predictive checks, and artificial-masking calibration address different questions about model adequacy. Cross-validation for MNAR models can assess certain forms of predictive performance, but its limits matter: held-out observed data do not directly reveal the outcomes missing in the target population. These approaches complement, rather than certify, an MNAR assumption.

Approach	What it contributes	What it cannot establish
AIC or BIC	A fit-complexity ranking for comparable likelihood-based candidates	That the MNAR mechanism or missing-outcome extrapolation is true
Observed-data checks	Whether fitted candidates reproduce examined features of observed records	How unobserved outcomes are distributed
Cross-validation	Predictive assessment for held-out information under the validation design	Direct validation of systematically missing outcomes
Sensitivity analysis	How conclusions change across plausible extrapolation assumptions	A data-only resolution of assumptions not identified by observed data

A common mistake is to say, “The model with the lowest BIC is the correct MNAR model.” The reasoning fails because BIC ranks the specified candidates using their fitted likelihood and penalty; the data do not reveal the missing outcomes needed to verify the extrapolation. The correction is to state the ranking narrowly, report the score differences and candidate set, and separately describe the assumptions and their consequences for the scientific estimate.

Key takeaway: Compare AIC or BIC only for candidates fitted to the same observed-data target with consistent likelihoods, parameter counts, and sample-size conventions. A lower score is a relative ranking—not validation of an MNAR assumption.

Check Your Understanding

Answer each question, then compare your work with the solutions.

A model has $\ell=-80$, $k=4$, and $n=50$. Calculate AIC and BIC. Solution: $\mathrm{AIC}=-2(-80)+2(4)=160+8=168$. Since $\log(50)\approx3.9120$, $\mathrm{BIC}=160+4(3.9120)\approx175.6$.
Two candidates use the same data and comparable likelihoods. Candidate A has AIC $310$, and Candidate B has AIC $314$. What does the difference establish? Solution: A has the lower AIC by $4$ points among these candidates. The difference is not a p-value and does not establish that A is true or that its MNAR extrapolation is correct.
A complex candidate adds two parameters and improves the maximized log-likelihood by $1.5$. Which has lower AIC? Solution: Here $2\Delta\ell=3$ and $2\Delta k=4$. Because $3<4$, the complex candidate’s AIC is higher by $1$, so AIC favors the simpler candidate.
Why is the number of observed visits generally not the $n$ for BIC in a longitudinal study with independent participants? Solution: Repeated visits within a participant are not separate independent participants. Under the stated convention, $n$ counts the independent units represented in the likelihood—here, participants—and the same convention must be applied to every candidate.
A candidate has lower BIC but assumes a different distribution for missing outcomes. Does its lower BIC verify that distribution? Solution: No. BIC gives a penalized ranking based on the specified observed-data likelihood and complexity. The distribution of outcomes that are missing is not directly observed, so the score cannot by itself verify that extrapolation.