Worked Diagnostic Analysis of Rare Binary Outcomes

Why Rare Binary Outcomes Need More Than One Diagnostic

When events are rare, a model can predict a low overall event rate and still miss important patterns. It might underpredict events in a subset of risk groups, or fail to reproduce how events are distributed across clusters. Conversely, a cluster with several events can look surprising when judged against a single fitted probability even if it is not unusual under the full model. A useful assessment therefore compares event calibration, cluster-level discrepancies, and simulated data patterns.

The analysis below uses a hypothetical binary outcome recorded for individuals within clusters. It assesses a fitted logistic random-intercept GLMM; it does not treat a single residual, tail probability, or simulation summary as a stand-alone pass-or-fail test.

Model and assumptions: For individual $i$ in cluster $j$, $Y_{ij}$ is 1 if the event occurs and 0 otherwise, with $$ Y_{ij}\mid b_j \sim \operatorname{Bernoulli}(p_{ij}), \qquad \operatorname{logit}(p_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j,\qquad b_j\sim N(0,\sigma_b^2). $$ The model assumes the specified conditional mean and link are suitable, clusters are independent, and outcomes are conditionally independent given the random effects and covariates. The random effects are assumed to follow the stated normal distribution. A diagnostic conclusion is conditional on these assumptions and on the adequacy of the fitted parameter estimates.

The probability $p_{ij}$ is conditional on the cluster effect $b_j$. The population-average probability for a randomly selected cluster is generally not the same as the probability calculated by setting $b_j=0$; averaging over the random-effects distribution changes the probability because the inverse-logit function is nonlinear. For diagnostics, it matters whether replicated data hold the fitted cluster effects fixed or draw new effects from their distribution.

Grouped Residual Calibration

A direct calibration check compares event counts with their fitted expectations across groups of similar predicted risk. Suppose a group contains $m$ binary outcomes with fitted probabilities $p_1,\ldots,p_m$. Its observed event count is $O=\sum_iY_i$, and its fitted expected count is $E=\sum_i p_i$. Under conditional independence, the variance of that count is

\operatorname{Var}(O\mid \mathbf{p})=\sum_{i=1}^{m}p_i(1-p_i).

This follows because each Bernoulli outcome has variance $p_i(1-p_i)$, and the conditional covariances are zero. A descriptive standardized discrepancy is $(O-E)/\sqrt{\sum_i p_i(1-p_i)}$. With rare events, that quantity is not automatically standard normal: a group may have only a few expected events, making a normal approximation unreliable. It is best read alongside the actual counts and predictive simulations.

Worked Example: Comparing Observed and Fitted Events by Risk Group

Consider eight hypothetical clusters, each with 20 individuals. The observed event counts are $0,0,0,1,1,1,2,4$. A fitted model gives conditional probabilities, constant within each cluster, of $0.025, 0.030, 0.035, 0.040, 0.045, 0.050, 0.060,$ and $0.080$, respectively. For a compact calibration check, group the first four clusters as lower predicted risk and the last four as higher predicted risk.

In the lower-risk group, the observed count is $O=0+0+0+1=1$. The expected count is $E=20(0.025+0.030+0.035+0.040)=20(0.130)=2.60$. The variance is $20[0.025(0.975)+0.030(0.970)+0.035(0.965)+0.040(0.960)]=2.513$ (rounded), so the standardized discrepancy is $(1-2.60)/\sqrt{2.513}=-1.009$.

In the higher-risk group, $O=1+1+2+4=8$, and $E=20(0.045+0.050+0.060+0.080)=4.70$. Its variance is $20[0.045(0.955)+0.050(0.950)+0.060(0.940)+0.080(0.920)]=4.410$, giving $(8-4.70)/\sqrt{4.410}=1.571$. Across all clusters, the observed total is 9 and the expected total is 7.30. The model therefore underpredicts the observed total in this hypothetical sample, particularly in the higher-risk group, but the discrepancies are not decisive by themselves: there are only eight clusters and few events. A simulation check can show whether comparable discrepancies commonly occur under the fitted model.

Grouping is a summary, not a substitute for inspecting the underlying predictions. Results can change with the number and boundaries of risk groups, and grouping discards individual-level information. Very small groups can have event proportions of zero or one by chance; groups should therefore be large enough to be interpretable, and the number of events and expected events should be shown. A calibration pattern is more persuasive when it is coherent across sensible groupings and consistent with other checks.

Cluster-Level Discrepancies in Sparse Data

The event count $S_j=\sum_iY_{ij}$ within a cluster is a useful discrepancy statistic. It retains information that a pooled event total can hide: two datasets with the same total can differ substantially in whether events are spread across clusters or concentrated in a few. Under the fitted model, the distribution of $S_j$ depends on the individuals’ fitted probabilities and on how the cluster effect is handled. For a cluster with a common fixed probability $p$, the conditional count has a binomial distribution, but that simplification does not generally apply when probabilities differ or the cluster effect is integrated out.

Worked Example: Interpreting a Cluster With Three Events

In a separate hypothetical cluster of 20 individuals, suppose the fitted conditional probability is $p=0.03$ for each individual and three events are observed. Under conditional independence, $S_j\sim\operatorname{Binomial}(20,0.03)$. The probability of at least three events is

\begin{aligned} P(S_j\geq 3\mid p=0.03) &=1-\sum_{k=0}^{2}{20\choose k}(0.03)^k(0.97)^{20-k}\\ &=1-\{0.543794+0.336368+0.098830\}\\ &\approx 0.021008. \end{aligned}

Thus, if the probability really were fixed at 0.03 and the outcomes conditionally independent, three or more events would occur in about 2.10% of such clusters. This is a useful indication that the count merits attention, not a complete model-level conclusion. If many clusters were examined, the chance that at least one looks this unusual would be greater than the chance for a prespecified single cluster. Moreover, uncertainty in fitted parameters and in the cluster effect is not represented by this simple calculation. A simulation from the fitted GLMM can account for the relevant structure more directly.

Pearson residuals can also flag individual outcomes or groups, but a residual’s scale does not erase the discreteness of a rare binary response. For a single outcome, the Pearson residual is $(Y_{ij}-p_{ij})/\sqrt{p_{ij}(1-p_{ij})}$; when $p_{ij}$ is small, an event can produce a large value. That is not automatically evidence of a bad fit. Inspect whether events occur in clusters or covariate regions systematically, and compare chosen discrepancies with their simulated distributions.

Conditional and Unconditional Predictive Checks

A predictive check asks whether the observed data feature is unusual relative to replicated datasets generated under the fitted model. A conditional replication holds the estimated random effects for the observed clusters fixed and simulates new outcomes from the corresponding fitted probabilities. It checks the outcome model given those fitted cluster effects. An unconditional replication draws new cluster effects from the fitted random-effects distribution, then generates outcomes. It checks whether the model’s population distribution of clusters can reproduce the observed pattern, including between-cluster variation.

Neither scheme answers every question. Conditional checks can look reassuring when estimated random effects have adapted to unusual observed clusters. Unconditional checks can show broad variability while obscuring a problem in a particular observed cluster. The replication scheme should match the diagnostic question, and important conclusions should not depend on an unexplained switch between schemes.

Worked Example: Assessing Simulated Rare-Event Patterns

Return to the hypothetical eight-cluster data with observed total $T_{\mathrm{obs}}=9$ and maximum cluster count $M_{\mathrm{obs}}=4$. Suppose 2,000 conditional replications were generated from the fitted cluster-specific probabilities. Across these replications, the 5th and 95th percentiles of the event total are 3 and 13, and 623 replications have at least 9 events. Using the common plus-one estimate for a simulated upper-tail probability gives

\widehat{P}(T^{\mathrm{rep}}\geq 9) =\frac{623+1}{2000+1} =\frac{624}{2001} \approx 0.312.

The observed total is within the simulated central range and is not an unusual upper-tail total in this check. Suppose also that 365 replications have a maximum cluster count of at least 4. The corresponding estimate is $(365+1)/(2000+1)=366/2001\approx0.183$. The observed maximum is therefore not rare under this conditional replication scheme, although that does not establish that the model fits in every respect.

Now suppose unconditional replications, which also draw cluster effects, have a 5th-to-95th percentile range of 2 to 17 for the event total, and the observed total remains within that range. This is compatible with the model reproducing the overall event total and a moderately uneven cluster pattern. It does not prove that the random-effects distribution is correct, nor does it rule out miscalibration for a covariate subgroup. The simulation output is evidence about the selected discrepancies under the chosen fitted model, not a probability that the model is true.

Putting the Evidence Together

A supported assessment integrates several views of the same fitted model. For rare binary outcomes, report the number of events and clusters, the observed and fitted event totals, calibration by risk group, and cluster-level discrepancies. Then state which statistics were checked using conditional or unconditional replications and summarize where the observed values fall relative to those replicated values. If several features were examined, say so: selecting the most extreme result after looking at many diagnostics makes an isolated tail probability less informative.

In the worked dataset, the fitted total of 7.30 is below the observed total of 9, and the higher-risk group has a positive discrepancy. However, the conditional simulated event total is not extreme, and the maximum cluster count is also reproduced reasonably often. A measured conclusion is that these checks do not show a clear failure for the examined total and maximum-count summaries, while the higher-risk calibration pattern deserves follow-up. Additional covariate-specific checks or more data may be needed to determine whether that pattern is systematic.

Check the setup.
Confirm the outcome coding, fitted probabilities, cluster membership, and the intended conditional-independence and random-effects assumptions.

Summarize calibration and clustering.
Compare observed and expected events across meaningful risk groups, and inspect cluster counts rather than relying on the total alone.

Simulate relevant replications.
Use conditional replications for outcome behavior given observed fitted cluster effects and unconditional replications for variation among clusters.

State a bounded conclusion.
Describe which observed features were or were not unusual under the fitted model and identify what the checks do not establish.

Alternatives, Diagnostics, and Limitations

A grouped calibration table is easy to communicate but depends on grouping choices. Residual plots preserve more detail, but rare binary responses make residual distributions discrete and can produce large values for ordinary events. Simulation-based checks can compare complex summaries, such as the total number of events, the maximum cluster count, or the number of event-free clusters, while respecting the fitted design. Their conclusions still depend on the selected statistics, the replication scheme, and whether parameter estimation is represented appropriately. When a check uses a statistic affected by parameter fitting, refitting each replicated dataset may be necessary to mimic the full fitting procedure; otherwise the simulation may not reflect that estimation step.

Warning signs include a persistent observed-versus-expected gap across reasonable groupings, a concentration of events that repeatedly exceeds simulated cluster-level patterns, or a clear discrepancy in a prespecified predictive statistic. Before attributing such a pattern to the model, check for data or coding errors, influential observations, omitted predictors, dependence not represented by the model, and problems in fitting or simulation. A plot that looks irregular is a reason to investigate, not proof of a particular cause.

The main limitation is information: a rare outcome may yield few events even in a substantial dataset, and a small number of clusters gives weak evidence about the random-effects distribution. Diagnostic checks can have low power to identify some forms of misspecification, while many checks create opportunities to notice chance extremes. A non-extreme diagnostic is not proof of adequacy; an extreme one is not, by itself, proof of which assumption failed. Compare the checks with the scientific purpose of the model and avoid treating a diagnostic tail area as a conventional hypothesis test unless the testing procedure was explicitly designed and calibrated for that use.

Common pitfall: “The overall observed event total is inside the simulation interval, so the model fits.” This ignores where events occur. A model can reproduce the total while concentrating events in the wrong clusters or risk groups. Check both aggregate and cluster-level patterns, and describe the conclusion as limited to the features actually examined.

Check Your Understanding

Answer each question, then compare with the solutions below.

A fitted model predicts 6 events in a risk group, and 10 are observed. What can be concluded from the difference alone?
What feature of an unconditional replication distinguishes it from a conditional replication?
For a cluster of 20 outcomes with common fitted probability 0.03, what is the probability of at least three events?
In 1,000 simulations, 49 replications have a discrepancy statistic at least as large as the observed statistic. Calculate the plus-one upper-tail estimate and state its interpretation.
Why does an event total inside a predictive range not establish that the model reproduces the cluster pattern?

Solutions. (1) The observed count exceeds its fitted expectation by 4, but the difference alone does not show whether the discrepancy is unusual. Compare its variability and inspect simulation-based checks that reflect the fitted design and clustering. (2) An unconditional replication draws new random effects from the fitted random-effects distribution before simulating outcomes; a conditional replication holds the fitted effects for the observed clusters fixed. (3) The binomial calculation is $1-[0.97^{20}+20(0.03)(0.97)^{19}+{20\choose2}(0.03)^2(0.97)^{18}]\approx0.0210$, or about 2.10%, conditional on the stated probability and independence assumptions. (4) The plus-one estimate is $(49+1)/(1000+1)=50/1001\approx0.0500$. About 5.00% of replications produced a discrepancy at least this large; this is a simulation-based diagnostic summary, not the probability that the model is true. (5) The same total can arise from different allocations of events across clusters. A total-only check does not assess whether the model reproduces the observed concentration or spread of events among clusters.