Residual Diagnostics with Sparse and Small Samples

Why Sparse Data Need Careful Residual Reading

Residual diagnostics compare observed responses with what a fitted model predicts. In large, informative data sets, plots can reveal patterns that deserve investigation. With sparse cells, rare outcomes, or only a few observations, however, the available residual values may be few, discrete, and strongly asymmetric. A plot can look irregular even when the model is appropriate; it can also look quiet simply because the data contain little information.

The key distinction is between a residual’s reference behavior and its appearance in one small sample. A residual may have mean zero and variance one under a specified model without having a bell-shaped distribution. Those moments alone do not make a normal reference pattern reliable. The calculations below show how this issue arises for binomial and Poisson responses.

Definition: For an observed response \(y_i\) with fitted mean \(\hat{\mu}_i\) and fitted variance \(V(\hat{\mu}_i)\), the Pearson residual is \(r_{P,i}=(y_i-\hat{\mu}_i)/\sqrt{V(\hat{\mu}_i)}\). The signed deviance residual is the square root of the observation’s deviance contribution, with the sign of \(y_i-\hat{\mu}_i\). These quantities measure different aspects of discrepancy; neither makes a sparse discrete response continuous.

For a binomial observation with \(n_i\) trials and fitted success probability \(\hat p_i\), the fitted mean and variance are \(n_i\hat p_i\) and \(n_i\hat p_i(1-\hat p_i)\). For a Poisson observation with fitted mean \(\hat\mu_i\), the variance is \(\hat\mu_i\). These formulas require a non-boundary fitted probability or a positive fitted Poisson mean so that the residual denominator is defined. Boundary fits require separate care; one should not treat a zero denominator as an ordinary residual.

Unit Variance Does Not Mean Normality

The following result gives an exact moment statement when the model’s mean and variance are known. It is useful as a baseline, but it is not a normality theorem and does not describe every residual calculated after fitting model parameters to the same data.

Theorem: If \(Y\) has finite mean \(\mu\) and finite, positive variance \(V\), then \(R=(Y-\mu)/\sqrt V\) has \(E[R]=0\) and \(\operatorname{Var}(R)=1\). These facts do not imply that \(R\) is normally distributed.

Proof. Linearity of expectation gives \(E[R]=(E[Y]-\mu)/\sqrt V=0\). Subtracting a constant does not change variance, and dividing by \(\sqrt V\) divides variance by \(V\), so \(\operatorname{Var}(R)=\operatorname{Var}(Y)/V=1\). The derivation establishes only the first two moments; it makes no claim that the distribution of \(R\) is continuous or normal. \(\square\)

Worked Example: A Rare Bernoulli Outcome

Suppose a single trial has a specified success probability \(p=0.02\), and let \(Y\) be its binary outcome. The fitted probability is treated as known here, so the residual is \(r_P=(Y-p)/\sqrt{p(1-p)}\).

If \(Y=0\), then \(r_P=-0.02/\sqrt{0.02(0.98)}=-0.02/0.14=-1/7\), approximately \(-0.143\). If \(Y=1\), then \(r_P=0.98/0.14=7\). The residual therefore takes only two values: \(-1/7\) with probability \(0.98\), and \(7\) with probability \(0.02\).

Its mean is \(0.98(-1/7)+0.02(7)=-0.14+0.14=0\). Its second moment is \(0.98(1/49)+0.02(49)=0.02+0.98=1\), so its variance is one. Yet its distribution is plainly not a bell curve: almost every observation gives a small negative residual, while the rare success gives a large positive one. A symmetric, approximately normal reference shape would be a poor description of this single-observation residual.

The same discreteness is visible in deviance residuals. For a Bernoulli response with specified probability \(p\), the signed deviance residual is \(-\sqrt{2\log(1/(1-p))}\) when \(Y=0\), and \(+\sqrt{2\log(1/p)}\) when \(Y=1\). At \(p=0.02\), these are approximately \(-0.201\) and \(2.797\). The exact values differ from the Pearson residuals, but the basic limitation remains: there are only two possible outcomes.

How Often Will a Small Sample Contain a Rare Outcome?

A sparse diagnostic can be uninformative simply because a small sample often contains no rare events. This is a property of the sampling model, not by itself evidence for or against a fitted model.

Result: If \(n\) independent Bernoulli trials each have success probability \(p\), then the probability of no successes is \((1-p)^n\). The probability of at least one success is at most \(np\).

Derivation. By independence, the probability that all \(n\) trials fail is the product of their failure probabilities: \(P(\text{no successes})=(1-p)^n\). For the bound, write the event of at least one success as the union of the \(n\) individual success events. The union bound gives \(P(\text{at least one success})\leq \sum_{j=1}^n P(\text{trial }j\text{ succeeds})=np\). This bound is useful when \(np\) is small; it is not generally an equality. \(\square\)

Worked Example: No Rare Events in Eight Trials

Suppose eight independent observations each have a predicted event probability of \(0.05\). The probability of seeing no events is \((1-0.05)^8=0.95^8\approx0.6634\). Thus, even under the stated probabilities, a sample with no events occurs about two-thirds of the time.

The probability of at least one event is \(1-0.95^8\approx0.3366\), which is below the union-bound limit \(np=8(0.05)=0.4\). If the eight observations all have outcome zero, there are no observed positive cases in this sample to compare against the model. That absence alone is not surprising under these probabilities. If the probabilities came from a model fitted using these same eight outcomes, the calculation would need to account for the fitting procedure; this example treats them as specified predictions.

Sparse Grouped-Binomial Cells

When each row records a number of successes out of \(n_i\) trials, the possible responses are still discrete. Small trial counts make that discreteness especially visible. The Pearson residual standardizes by the binomial standard deviation; the deviance residual instead reflects the likelihood discrepancy from the fitted model. Neither calculation creates information that the cell does not contain.

Worked Example: Four Trials with a Low Success Probability

Consider one grouped observation with \(n=4\), fitted success probability \(\hat p=0.10\), and hence fitted mean \(0.4\) and variance \(4(0.10)(0.90)=0.36\). The standard deviation is \(0.6\). Compare the residuals for \(y=0\) and \(y=1\).

For \(y=0\), the Pearson residual is \((0-0.4)/0.6=-0.667\). Its deviance contribution is \(2[0\log(0/0.4)+4\log(4/3.6)]\), where the zero-count term is defined by its limiting value \(0\). This is \(8\log(10/9)\approx0.8429\), so the signed deviance residual is \(-\sqrt{0.8429}\approx-0.918\).

For \(y=1\), the Pearson residual is \((1-0.4)/0.6=1\). The deviance contribution is \(2[1\log(1/0.4)+3\log(3/3.6)]\). Numerically, the bracket is \(0.9163+3(-0.1823)\approx0.3693\); the contribution is approximately \(0.7387\), and the signed deviance residual is \(+\sqrt{0.7387}\approx0.859\).

These values are fully determined by the small count. Under the model, \(P(Y=0)=0.9^4=0.6561\) and \(P(Y=1)=4(0.1)(0.9^3)=0.2916\). Thus, \(P(Y\leq1)=0.9477\). The two residuals above represent very common outcomes, not automatically a failure of fit. Looking at one such row in isolation cannot establish whether the overall model is adequate.

Small Poisson Means and Uneven Residuals

When a Poisson mean is small, zero counts are common and positive counts are relatively rare. Pearson residuals can then be concentrated near one value with occasional larger values. Deviance residuals also reflect the imbalance, although their numerical values differ. The following example calculates both directly.

Worked Example: A Poisson Mean of 0.2

Suppose \(Y\sim\operatorname{Poisson}(0.2)\), with the mean treated as specified. Since the Poisson variance equals its mean, the Pearson residual is \((Y-0.2)/\sqrt{0.2}\). For \(y=0\), this is \(-\sqrt{0.2}\approx-0.447\). For \(y=1\), it is \(0.8/\sqrt{0.2}\approx1.789\).

The probabilities are \(P(Y=0)=e^{-0.2}\approx0.8187\) and \(P(Y=1)=0.2e^{-0.2}\approx0.1637\); the remaining probability, about \(0.0175\), is for counts of two or more. Consequently, most observations produce the same negative Pearson residual, while larger positive residuals occur rarely.

For the deviance residual, \(y=0\) gives \(-\sqrt{2[0- (0-0.2)]}=-\sqrt{0.4}\approx-0.632\), using \(y\log(y/\hat\mu)=0\) at \(y=0\). For \(y=1\), the residual is \(+\sqrt{2[\log(1/0.2)-(1-0.2)]}\) \(=+\sqrt{2(1.6094-0.8)}\approx1.272\). Both residual definitions show an uneven, discrete reference pattern. A normal-looking residual histogram should not be expected from a small collection of observations with this mean.

What to Conclude from a Sparse Diagnostic

Residual plots remain useful, but their evidential strength depends on how much information the data provide. With rare outcomes, several observations may share the same response and fitted mean, so their residuals can pile up at a small number of values. With only a few observations, a single rare count can dominate the plot. These features may be consequences of the response distribution rather than signs of a systematic model defect.

Near-boundary fitted probabilities require additional caution. In the Bernoulli Pearson residual, the positive value for a success is \(\sqrt{(1-p)/p}\), which grows as \(p\) approaches zero; the negative value for a failure grows in magnitude as \(p\) approaches one. These large values reflect the model assigning low probability to the observed outcome. But a very small fitted probability, a data error, a misspecified mean structure, and an unusual chance outcome are different explanations that a residual alone cannot distinguish. At an exact boundary, standardization may be undefined or degenerate.

Randomized quantile residuals can spread the discrete probability mass through randomized positions within its cumulative-probability jumps. That can help display discrete responses on a continuous scale, but randomization does not add observations or remove sampling uncertainty. In a tiny sample, changing the randomization can change the plot; conclusions should not rest on one realization.

Key takeaway: In sparse or small samples, interpret residuals against the response distribution and the amount of information available. A discrete, asymmetric, or visually irregular pattern is not by itself evidence of misspecification, and a quiet plot is not proof of adequacy.

Common Mistakes

Assuming unit variance implies normality. The exact moment result establishes a mean and variance, not a bell-shaped reference distribution. The rare Bernoulli example has variance one but only two possible residual values.
Treating every large residual as a model failure. A rare outcome can produce a large standardized residual even when the stated probability model is correct. Check how improbable the outcome is under the fitted model and consider the small-sample context.
Calling an all-zero sample surprising without calculating its probability. With eight independent event probabilities of \(0.05\), no events occur with probability about \(0.663\), so that observation is ordinary under the specified probabilities.
Expecting randomization to solve sparsity. Randomized residuals alter the display of discrete values; they do not make a small sample informative or eliminate uncertainty.

Check Your Understanding

Use the calculations and distinctions in this tutorial to answer the following.

Why does a Pearson residual with mean zero and variance one not necessarily have an approximately normal distribution?
For ten independent Bernoulli trials with success probability \(0.03\), write the exact probability of observing no successes.
In the grouped-binomial example, why is an observed count of zero not automatically evidence of poor fit?
For a small Poisson mean, why can most Pearson residuals be identical or nearly identical?
What does randomization change in a discrete residual display, and what limitation does it leave unchanged?