Choosing Discrepancies for Predictive Checks

Why the Choice of Discrepancy Matters

A predictive check compares a feature of the observed responses with the same feature in data simulated from a fitted model. The feature being compared is the discrepancy: it might be the total number of events, the number of unusually large counts, or the largest cell-level residual. No one summary can reveal every way a model may fail. A total can look typical even when events occur in the wrong cells; a tail count can miss a mismatch near the center of the response distribution.

For binomial and Poisson GLMs, the design values and exposures are usually held fixed while responses are simulated. If \(Y_i\) is the response in cell \(i\), the fitted model specifies its conditional distribution given those fixed quantities. A useful check begins by asking what pattern would be concerning in the application, then chooses a statistic that would respond to that pattern.

Definition: A discrepancy statistic is a numerical summary \(T(Y)\) of the observed responses, often calculated using fitted model quantities as well as the responses. A predictive check compares \(T(Y)\) with \(T(Y^{\mathrm{rep}})\), where \(Y^{\mathrm{rep}}\) is generated from the fitted model under the same design. Large values are evidence against the model only when large values represent the kind of mismatch the statistic was designed to detect.

In a parametric bootstrap check, simulate responses from the fitted model, refit the model to each simulated data set using the same fitting procedure, and calculate the same discrepancy for each refit. The resulting bootstrap distribution accounts approximately for the fitting process; it is not generally an exact finite-sample reference distribution when parameters are estimated. For a discrepancy where larger values are more concerning, a common Monte Carlo tail estimate from \(B\) replicates is

\hat p_{\mathrm{boot}}= \frac{1+\sum_{b=1}^{B} I(T_b^{\mathrm{rep}}\ge T_{\mathrm{obs}})}{B+1}.

This tail estimate is meaningful only after the statistic and its direction have been specified. When comparing a directional count, such as the number of unusually large responses, the relevant tail is the upper tail of that count. When comparing a two-sided measure of cell departures, define it so that larger values indicate more extreme departures.

What Totals Can and Cannot Show

For independent responses conditional on the design, totals give a compact check on the overall amount of response. In a binomial GLM, write \(Y_i\sim\operatorname{Bin}(m_i,p_i)\), with \(m_i\) trials in cell \(i\). In a Poisson GLM, write \(Y_i\sim\operatorname{Pois}(\lambda_i)\). The following result gives the basic reference calculations when the model parameters are fixed.

Result: If the responses are conditionally independent, then for binomial cells \(E[\sum_iY_i]=\sum_i m_i p_i\) and \(\operatorname{Var}(\sum_iY_i)=\sum_i m_i p_i(1-p_i)\). If the responses are independent Poisson variables, then \(\sum_iY_i\sim\operatorname{Pois}(\sum_i\lambda_i)\). If all binomial cells share the same probability \(p\), their total is \(\operatorname{Bin}(\sum_i m_i,p)\).

For the binomial mean and variance, linearity of expectation gives the sum of the means. Independence makes each covariance between distinct responses zero, so the variance of the sum is the sum of the variances. When the probabilities are equal, the total counts successes across all \(\sum_i m_i\) independent Bernoulli trials, each with probability \(p\); hence it has a binomial distribution. For independent Poisson variables, the probability generating function of the total is the product of the individual generating functions:

\prod_i E[t^{Y_i}] =\prod_i \exp\{\lambda_i(t-1)\} =\exp\{(t-1)\sum_i\lambda_i\}.

This is the probability generating function of a Poisson variable with mean \(\sum_i\lambda_i\), proving the claim. In either GLM, a total is an informative summary of overall event volume. But it discards which cells contributed those events. In particular, binomial totals with differing \(p_i\) are not generally binomial with a single common probability, and matching the total does not establish that the cell-specific predictions are adequate.

Worked Example: A Binomial Total Conceals Cell Departures

Suppose four independent binomial cells each have \(m_i=20\), with fitted probabilities \(0.20,\ 0.30,\ 0.40,\ 0.50\). The observed counts are \(8,\ 2,\ 8,\ 10\). Compare the total with the cell-level departures.

The fitted expected counts are \(20(0.20)=4,\ 20(0.30)=6,\ 20(0.40)=8,\) and \(20(0.50)=10\). Thus the observed and expected totals both equal \(28\). A total-events check alone reports no departure from the fitted mean.

The binomial variances are \(20p_i(1-p_i)\), namely \(3.2,\ 4.2,\ 4.8,\ 5.0\). The Pearson cell discrepancy, the sum of squared residuals divided by their fitted variances, is

X^2=\frac{(8-4)^2}{3.2}+\frac{(2-6)^2}{4.2} +\frac{(8-8)^2}{4.8}+\frac{(10-10)^2}{5} =5+3.81+0+0=8.81.

The first two cells depart in opposite directions: their standardized residuals are \(4/\sqrt{3.2}\approx2.24\) and \(-4/\sqrt{4.2}\approx-1.95\). The total cancels these departures, while the cell-level statistic registers both. Whether \(8.81\) is unusually large must be judged against the bootstrap distribution for the fitted model, not from this calculation alone.

Tail Counts Target Specific Response Patterns

A tail count records how many cells have responses in a specified region of their fitted distributions. For example, it can count large binomial proportions, unusually large Poisson counts, or responses in either tail according to a specified cell-specific rule. Such a statistic focuses attention on rare or extreme observations that may have little effect on a total.

Result: Suppose \(Y_i\) are independent and \(A_i\) is a specified event for cell \(i\). Define \(T=\sum_i I(Y_i\in A_i)\) and \(q_i=P(Y_i\in A_i)\). Then \(E[T]=\sum_i q_i\) and \(\operatorname{Var}(T)=\sum_i q_i(1-q_i)\).

To verify this, let \(Z_i=I(Y_i\in A_i)\). Since \(Z_i\) is Bernoulli with probability \(q_i\), \(E[Z_i]=q_i\) and \(\operatorname{Var}(Z_i)=q_i(1-q_i)\). Therefore \(E[T]=\sum_i E[Z_i]=\sum_i q_i\). The indicators are independent because each is a function of a different independent response, so all their pairwise covariances are zero. Consequently, \(\operatorname{Var}(T)=\sum_i\operatorname{Var}(Z_i)=\sum_iq_i(1-q_i)\). This calculation applies directly when the probabilities are fixed. With estimated GLM parameters, use the bootstrap distribution to assess the statistic, since the fitting step can affect its distribution.

Worked Example: A Poisson Tail Count Detects Clustering

Consider ten independent Poisson cells, each with fitted mean \(\lambda_i=0.5\). The observed counts are \(5,0,0,0,0,0,0,0,0,0\). Compare the total, a large-count tail count, and the cell-level Pearson discrepancy.

The observed total is \(5\), exactly the fitted expected total \(10(0.5)=5\). Define the tail count as the number of cells with \(Y_i\ge3\). Its observed value is \(1\). For a Poisson variable with mean \(0.5\), \(P(Y_i\ge3)=1-e^{-0.5}(1+0.5+0.5^2/2)\approx0.01439\). The expected tail count under the fixed fitted means is therefore \(10(0.01439)\approx0.144\).

The Pearson cell discrepancy is much larger than the total alone suggests:

X^2=\frac{(5-0.5)^2}{0.5} +9\frac{(0-0.5)^2}{0.5} =40.5+4.5=45.

The observed high-count cell is unusual under its fitted distribution, and the other nine cells are all below their fitted means. The total nevertheless equals its expected value because the high count offsets those low counts. The tail count makes the high-count feature explicit; the cell-level statistic also records the departures in the other cells. Their bootstrap reference distributions determine how unusual the observed values are after fitting.

Cell-Level Discrepancies and Their Interpretation

A cell-level discrepancy retains information about where the model misses. Common choices include the Pearson sum \(X^2=\sum_i (Y_i-\hat\mu_i)^2/\widehat{\operatorname{Var}}(Y_i)\), the largest absolute standardized residual, or the number of cells whose standardized residual exceeds a chosen threshold in absolute value. These answer different questions. A sum can accumulate many moderate discrepancies; a maximum emphasizes one striking cell; a threshold count measures how often a specified degree of departure occurs.

The statistic should reflect the response scale and the concern. For binomial cells, Pearson standardization uses fitted variance \(m_i\hat p_i(1-\hat p_i)\); for Poisson cells, it uses \(\hat\lambda_i\). These standardizations can help compare cells with different fitted means, but they do not make their distributions identical, particularly when counts are sparse. A bootstrap check is valuable precisely because it can reproduce the discreteness and differing means that make simple reference approximations unreliable.

Worked Example: Choosing a Binomial Extreme-Cell Check

Suppose there are 20 independent cells, each with \(Y_i\sim\operatorname{Bin}(4,0.25)\). The observed counts consist of one cell with \(Y_i=4\), sixteen cells with \(Y_i=1\), and three cells with \(Y_i=0\). Compare the total with a statistic targeting perfect-success cells.

The observed total is \(4+16(1)+3(0)=20\), equal to its fitted expected total \(20[4(0.25)]=20\). Define \(T\) as the number of cells with all four trials successful, so \(T=\sum_i I(Y_i=4)\). The observed value is \(1\). In each cell, \(P(Y_i=4)=0.25^4=0.00390625\), so the expected count under the fixed fitted probabilities is \(20(0.00390625)=0.078125\).

A cell-level Pearson calculation also shows the concentration of the events. Each cell has fitted mean \(4(0.25)=1\) and variance \(4(0.25)(0.75)=0.75\). The cell with four successes contributes \((4-1)^2/0.75=12\). Each of the sixteen cells with one success contributes zero, and each of the three zero-count cells contributes \((0-1)^2/0.75=4/3\). Hence \(X^2=12+3(4/3)=16\).

The total check misses the concentration because it equals its fitted expectation. The extreme-cell count is aimed at perfect-success cells, while \(X^2\) also registers the three low cells. If the probability \(0.25\) was estimated from these data, it should be re-estimated in each bootstrap replicate before assessing the observed \(T\) or \(X^2\).

Choosing a Summary Without Overclaiming

A good discrepancy is not simply a statistic that produces a small tail probability. It is a statistic selected because it is sensitive to a failure that matters. Before simulating, state the suspected pattern and the summary that would register it. For instance, a total targets overall event volume; a high-tail count targets too many unusually large responses; a cell-level maximum targets a single severe mismatch; and a Pearson sum targets accumulated departures across cells.

A common mistake is to treat a total close to its fitted expectation as evidence that the entire response model fits. The reasoning fails because positive and negative cell departures can cancel in the total. The correction is to pair the total with at least one summary that retains distributional or cell-level information, chosen for the scientific concern. The reverse mistake is to treat one extreme cell as decisive without comparing it with the bootstrap distribution: even a correctly specified model can generate an occasional extreme response, especially across many cells.

Predictive checks are also limited by the summaries chosen. If a tail threshold is too high, the count may be nearly always zero and have little ability to distinguish models. If it is too low, it may mostly track the overall total. A useful analysis can examine a small set of scientifically motivated discrepancies, report their definitions and observed values, and compare each with its corresponding bootstrap distribution. Such checks diagnose particular patterns; they do not prove that the model is correct when no chosen summary looks unusual.

Check Your Understanding

For each question, focus on what the proposed summary can detect and what information it discards.

In independent binomial cells with differing fitted probabilities, why can matching the observed total to its fitted expectation still conceal cell-level lack of fit?
For a Poisson GLM, describe a response pattern that a count of cells with \(Y_i\ge3\) is designed to detect.
What is the difference in emphasis between a maximum absolute standardized residual and the sum of squared Pearson residuals?
Why should a bootstrap replicate generally be refitted when checking a discrepancy for a model with estimated parameters?
If a tail count is zero in the observed data, does that alone show that the model fits? Explain what reference comparison is needed.