Residual Normality, Outliers, and Innovation Checks

Why Check Residual Normality?

An ARMA model describes a time series through its systematic dependence and a sequence of innovations. When the model is correctly specified with Gaussian innovations, those innovations are independent normal disturbances with constant variance. Checking whether fitted residuals look compatible with that assumption helps assess the model’s distributional adequacy. It can matter especially for likelihood-based inference and probability calculations that rely on Gaussian errors.

A residual diagnostic does not prove that the innovations are Gaussian. The innovations are unobserved; residuals are estimated from the data and depend on estimated model parameters. Moreover, a non-normal-looking residual distribution may be a symptom of something other than non-Gaussian innovations. A model may leave a changing mean, serial dependence, or clusters of unusually large values in its residuals. These features call for different investigations.

Definition: For an ARMA model, an innovation is the new, unobserved disturbance at time \(t\) in the model’s representation. A fitted residual, written \(\hat{\varepsilon}_t\), is the sample-based estimate of that disturbance after fitting the model. A standardized residual is a residual divided by its estimated standard deviation; in the constant-variance case, \(z_t=\hat{\varepsilon}_t/\hat{\sigma}\).

For the diagnostic discussion here, the target model has Gaussian white-noise innovations: \(\varepsilon_t\) are independent, each with distribution \(N(0,\sigma^2)\). The ARMA mean and dependence structure are assumed to be correctly specified, and the fitted parameters satisfy the model’s stationarity and invertibility requirements where those are needed for the chosen representation. In practice, fitted residuals only approximate the innovations, particularly in finite samples and near the beginning of a series. Use residual diagnostics as evidence about the fitted model, not as direct observations of the innovations.

Graphical Checks: Histogram and Normal Q–Q Plot

A residual histogram gives a rough view of shape. A distribution that is approximately symmetric and bell-shaped is compatible with normality, while strong skewness, heavy tails, or multiple peaks may be reasons to investigate further. The display depends on choices such as bin width, however, and a small sample can look irregular even when the underlying distribution is normal. Overlaying a fitted normal curve can help, but it does not turn the histogram into a formal test.

A normal Q–Q plot compares ordered standardized residuals with the corresponding theoretical quantiles of a standard normal distribution. If the residual distribution is close to normal, the points should lie roughly along a straight line. Pronounced curvature can indicate skewness; departures concentrated at both ends can indicate heavier or lighter tails than the normal distribution. A few points far from the line may be isolated extremes. The plot is a diagnostic comparison, not a test that certifies a model.

Both displays should be read alongside a plot of residuals over time. A histogram discards time order, and a Q–Q plot displays ranks rather than when values occurred. Thus neither can reveal a persistent run or a changing residual variance. Those patterns are important even if the overall histogram looks fairly normal.

Worked Example: Standardizing and Screening an Extreme Residual

Suppose a hypothetical fitted ARMA model has constant estimated innovation standard deviation \(\hat{\sigma}=2.0\) units. Among its fitted residuals is one value of \(6.4\) units. Standardize it and compare it with a common three-standard-deviation screening threshold.

Standardization. The standardized residual is \(z_t=\hat{\varepsilon}_t/\hat{\sigma}=6.4/2.0=3.2\). It is 3.2 estimated standard deviations above zero, so it exceeds the informal \(3\)-standard-deviation threshold.

If the standardized innovations were independent standard normal variables, the probability that one specified value lies more than three standard deviations from zero would be \(P(|Z|>3)\), approximately \(0.00270\). In a series of \(n=20\) such values, the expected number beyond that threshold would be \(20(0.00270)=0.0540\). The probability of at least one would be \(1-(1-0.00270)^{20}\), approximately \(0.0526\).

This residual merits inspection, but it is not by itself proof of a bad model or a data error. The calculation assumes independent standard normal values, whereas fitted residuals are estimated and may not satisfy that assumption exactly. The threshold is a screening heuristic, not a rule to delete the observation.

A Formal Normality Check

One commonly used omnibus test is the Jarque–Bera test. It summarizes two departures from normality: skewness and excess kurtosis. Let \(z_1,\ldots,z_n\) be standardized residuals, let \(\bar{z}\) be their sample mean, and define the moment-based skewness \(S\) and Pearson kurtosis \(K\) by

S=\frac{\frac{1}{n}\sum_{t=1}^{n}(z_t-\bar{z})^3} {\left[\frac{1}{n}\sum_{t=1}^{n}(z_t-\bar{z})^2\right]^{3/2}}, \qquad K=\frac{\frac{1}{n}\sum_{t=1}^{n}(z_t-\bar{z})^4} {\left[\frac{1}{n}\sum_{t=1}^{n}(z_t-\bar{z})^2\right]^2}.

For a normal distribution, the population skewness is zero and the Pearson kurtosis is three. The Jarque–Bera statistic is

JB=\frac{n}{6}\left(S^2+\frac{(K-3)^2}{4}\right).

Under the null hypothesis of normality, with independent observations and suitable large-sample conditions, \(JB\) has an approximate chi-square distribution with \(df=2\). The test is asymptotic: with a small sample, its p-value may be a poor approximation. For fitted time-series residuals, parameter estimation and any remaining dependence can also affect finite-sample calibration. If calibration is important, a parametric bootstrap that simulates from the fitted model and refits it can better reproduce the estimation step.

Justification: For independent normal observations, the large-sample distribution of \(\sqrt{n}(S,K-3)\) is approximately bivariate normal with mean \((0,0)\) and covariance matrix having diagonal entries \(6\) and \(24\), and off-diagonal entry \(0\). Standardizing the two components and summing their squares gives an approximate chi-square distribution with two degrees of freedom: \(nS^2/6+n(K-3)^2/24=JB\). This explains both the statistic’s formula and its reference distribution. The argument relies on the independent-normal large-sample approximation; it does not guarantee that the same calibration is exact for estimated ARMA residuals.

Worked Example: Calculating a Jarque–Bera Statistic

Suppose a hypothetical set of \(n=80\) standardized residuals has moment-based skewness \(S=0.50\) and Pearson kurtosis \(K=4.20\). Test the null hypothesis of normality using the large-sample Jarque–Bera reference distribution.

Hypotheses. \(H_0\) is that the innovations have a normal distribution; \(H_a\) is that their distribution is not normal. For this calculation, treat the residuals as sufficiently close to independent observations for the stated large-sample approximation.

Statistic. Substitution gives \(JB=\frac{80}{6}\left(0.50^2+\frac{(4.20-3)^2}{4}\right)\). Since \(0.50^2=0.25\) and \((1.20)^2/4=0.36\), the result is \(JB=13.3333(0.61)=8.133\).

For a chi-square distribution with \(df=2\), the upper-tail probability at \(8.133\) is approximately \(0.0171\). At significance level \(\alpha=0.05\), this p-value is below \(\alpha\), so reject \(H_0\) under the test’s approximation. The result is evidence against Gaussian residual shape; it does not identify the cause, establish that the true innovations are non-normal, or show that the null hypothesis has a 1.71% probability of being true.

Outliers and Persistent Residual Patterns

An extreme residual is an observation that is unusually far from the fitted value relative to the model’s estimated innovation scale. A large standardized residual can indicate a recording or measurement problem, an unusual but genuine event, or a model that fails to account for a relevant feature of the series. Check the original record and the context before deciding how to handle it. Deleting an observation solely because it makes a diagnostic plot look better can distort the analysis and conceal model failure.

The time order of residuals helps distinguish an isolated extreme from a persistent pattern. One unusually large value with otherwise unremarkable neighboring residuals may be an isolated event. Long runs of positive or negative residuals, repeated clusters of large magnitudes, or a sustained change in residual level or spread suggest that the model has left systematic structure unexplained. Possible explanations include an omitted mean change, remaining serial dependence, changing variance, or a change in the data-generating process. These are hypotheses to investigate, not conclusions established by a plot alone.

Worked Example: A Run Is Not an Isolated Outlier

Consider the following hypothetical standardized residuals from a fitted model, in time order: \(1.1, 1.3, 0.9, 1.5, 1.2, 1.0, 1.4, 1.1, -1.2, -1.4, -0.9, -1.3, -1.1, -1.5, -1.0, -1.2\). None exceeds the informal absolute-value threshold of 3. Assess what the sequence suggests.

The first eight values sum to \(9.5\), so their mean is \(9.5/8=1.1875\). The last eight sum to \(-9.6\), so their mean is \(-9.6/8=-1.200\). The sequence consists of a sustained positive run followed by a sustained negative run, not one isolated large residual.

Although every value is within three estimated standard deviations of zero, the ordered pattern is a reason to question whether the model has captured the series’ mean or dependence adequately. A histogram might show a roughly balanced collection of positive and negative values and conceal this ordering. The sequence alone does not determine the cause, but it warrants review of residual behavior over time and other model diagnostics.

Putting the Checks Together

Normality diagnostics address distributional shape, while checks for residual autocorrelation address dependence. They are complementary. A set of residuals can be approximately normal in its marginal distribution and still have serial dependence; conversely, it can have little apparent serial correlation and still show skewness or heavy tails. A normality-test p-value should therefore not be treated as an overall measure of ARMA adequacy. Interpret it with the residual time plot, the histogram or Q–Q plot, and the autocorrelation diagnostics considered in the earlier material on residual autocorrelation.

A useful diagnostic sequence is to examine residuals in time order, inspect their distribution with a histogram and normal Q–Q plot, and then use a formal normality test when its assumptions and sample-size limitations are understood. Investigate unusual observations in context. If the residuals show runs, clusters, or sustained changes, prioritize understanding those patterns rather than attributing every departure to non-normality. The goal is not to force the residuals to look normal by removing inconvenient values; it is to determine whether the fitted model is an adequate description for the intended analysis.

Common Mistakes

Treating a normality-test result as a model verdict. A small p-value indicates evidence against the test’s normality null under its calibration assumptions; it does not identify the cause or diagnose every aspect of the model.
Equating a large residual with a data error. A large residual is a prompt to investigate the observation and model, not an automatic reason to delete it.
Ignoring time order. A histogram and Q–Q plot do not show runs or variance clusters. Examine residuals as a time sequence as well.
Using an iid test without qualification. Normality-test reference distributions may not be exact for fitted residuals, especially with dependence, parameter estimation, or small samples.
Confusing absence of extreme values with adequacy. Persistent runs can be important even when every residual is within three standard deviations.

Key takeaway: Residual normality is one part of ARMA diagnostics. Use graphical displays and formal tests as evidence, inspect extreme observations without automatic deletion, and use the time ordering to recognize persistent patterns that may signal misspecification.

Check Your Understanding

Answer each question using the assumptions and distinctions developed above.

Why are fitted ARMA residuals not identical to the unobserved innovations?
What kinds of departures from normality may be suggested by curvature or tail departures in a normal Q–Q plot?
For \(n=100\), \(S=0.30\), and \(K=4.00\), calculate the Jarque–Bera statistic and state what additional information is needed to make a test decision.
Why is a residual beyond three estimated standard deviations a screening signal rather than an automatic deletion rule?
How might a persistent run of positive residuals differ from one isolated extreme, even if neither includes a residual beyond three standard deviations?