Innovation Residuals and Filtering Diagnostics

What an Innovation Residual Measures

A filter predicts observations using the information available so far. The difference between a new observation and that prediction is an innovation residual. Under a correctly specified model, this residual has predicted mean zero and a covariance fixed by the observation-noise model. These properties make residuals useful diagnostics: persistent means suggest bias, incorrect spread suggests a noise-scale problem, and dependence across time can signal model misspecification.

Residuals must be interpreted with care. Subtracting the predicted observation drift does not usually leave only observation noise. The residual also includes the effect of any error in the predicted hidden state. Consequently, the observation-noise variance alone is the full residual variance only when the state contribution is known exactly, or when additional assumptions establish that the state-error contribution vanishes.

Work on a filtered probability space \((\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})\) carrying a two-dimensional Brownian motion \(\mathbf{B}_t=(B^1_t,B^2_t)\) with independent components. Consider the scalar linear Gaussian model \(dX_t=aX_t\,dt+q\,dB^1_t\) and \(dY_t=H X_t\,dt+\sqrt{R}\,dB^2_t\), where \(R>0\), and the initial state is Gaussian and independent of \(\mathbf{B}\). The signal and observation stochastic integrals are Ito integrals; their integrands are adapted, progressively measurable, and satisfy \(\mathbb{E}\int_0^T H_s^2\,ds<\infty\) on every finite horizon. Here the constant integrands \(q\) and \(\sqrt R\) satisfy this condition. The observation filtration is \(\mathcal{F}^Y_t=\sigma(Y_s:0\le s\le t)\), completed in the usual way. Write \(m_t=\mathbb{E}[X_t\mid\mathcal{F}^Y_t]\) for the filter’s conditional mean.

For this correctly specified linear Gaussian model, the innovations result states that \(I_t/\sqrt R\) is a Brownian motion with respect to the observation filtration. This is a standard result from the earlier material on the innovations process; its filtering hypotheses include the correct signal and observation model and the conditional-mean filter. Thus its increments have zero conditional mean, variance \(R(t-s)\), and are independent over disjoint time intervals. This statement concerns the exact model residuals, not arbitrary residuals computed after fitting parameters to the same data.

Predictions Under the Correct Model

Proof. By definition, \(r_{s,t}=I_t-I_s\). The innovations result gives that \(I/\sqrt R\) is Brownian in the observation filtration. Therefore \(I_t-I_s\), conditional on \(\mathcal{F}^Y_s\), is centered normal with variance \(R(t-s)\). Its conditional mean and conditional second moment are consequently \(0\) and \(R(t-s)\). Brownian increments on disjoint time intervals are independent, so the corresponding residual increments are independent. Dividing each increment by the positive constant \(\sqrt{R(t-s)}\) gives a standard normal random variable and preserves independence. \(\square\)

The proposition supplies predictions, not a guarantee that a finite dataset will have an exactly zero sample mean or exactly zero sample autocorrelations. Random samples fluctuate. Diagnostics therefore compare observed patterns with these model-based predictions, accounting for the number and spacing of observations.

Residual Decomposition: Include State-Prediction Error

Suppose the actual observation equation includes a deterministic offset \(b\,dt\), while the filter subtracts only its modelled drift \(Hm_u\,du\). On an interval \([s,t]\) the resulting residual is

This identity follows by integrating the observation equation and subtracting the filter’s predicted drift. Taking expectations gives the exact mean formula

Thus the offset contributes \(b(t-s)\), but that contribution need not be the entire residual mean. If the filter updates its state estimate using observations affected by the offset, its state-prediction error may itself have nonzero mean. Likewise, the residual variance in general includes the variance of the integrated state error and its covariance with observation noise; it cannot be obtained by adding the observation-noise variance alone without justified assumptions.

Proof. Exact knowledge of the state contribution means that subtracting the predicted observation drift leaves \(r_{s,t}=b(t-s)+\sqrt R(B^2_t-B^2_s)\). Brownian increments have mean zero and variance \(t-s\). The deterministic term has mean \(b(t-s)\) and zero variance. Hence the residual mean is \(b(t-s)\), while its variance is \(R(t-s)\). \(\square\)

Diagnosing Noise Scale and Parameter Errors

When a filter predicts variance rate \(\widehat R\) but the actual observation-noise variance rate is \(R\), standardized residuals formed using the filter’s scale have noise variance \(R/\widehat R\), provided the state contribution is known exactly. A value above one indicates that the actual noise is larger than predicted; a value below one indicates that it is smaller. With uncertain state, this comparison alone is insufficient: state-prediction error also affects the total residual spread.

Parameter misspecification can instead produce a time-varying residual mean. A clean way to isolate this effect is to use a known deterministic signal trajectory, so no hidden-state estimation error is being mistaken for an observation-noise effect. More generally, if the true signal is \(X_u\) and the filter predicts \(\widehat X_u\), the mean contribution from their difference over an interval is \(\mathbb{E}[\int_s^t H(X_u-\widehat X_u)\,du]\), in addition to any omitted offset.

Using Residuals Responsibly

For observations at times \(0=t_0<t_1<\cdots<t_n\), compute each residual by subtracting the integral of the filter’s predicted observation drift over that interval. Under the correct model, its predicted mean is zero and its variance is \(R(t_k-t_{k-1})\). Standardize by the square root of that variance before comparing residuals from intervals of different lengths. If the model predicts correlated observation noise or a different observation structure, the appropriate covariance prediction must replace this scalar formula.

Residual whiteness means zero covariance between distinct increments in the model’s prediction. In the linear Gaussian setting here, the exact innovations are Gaussian, so zero covariance across disjoint increments accompanies independence. For non-Gaussian residuals, zero covariance alone does not imply independence. Also, a sample autocorrelation close to zero is not proof of independence, and a nonzero sample autocorrelation can occur by chance in a finite sample.

A useful diagnostic sequence is to examine the residual mean, scale, and temporal pattern separately. A stable mean shift is consistent with bias, but can also arise from a persistent state-prediction error. A scale mismatch can indicate incorrect observation-noise variance, but state uncertainty must be included before assigning that cause. Serial structure can indicate missed dynamics, delayed observations, or an inadequate noise model. These checks identify tensions between data and model; they do not by themselves identify a unique cause.

Common mistake. It is wrong to say that an omitted offset \(b\) makes every residual mean exactly \(bh\), without qualification. The correct mean is \(bh+\mathbb{E}[\int_s^t H(X_u-m_u)\,du]\); only when that integrated state-error term has mean zero does the simpler answer follow. It is also wrong to call \(Rh\) the full residual variance under noise misspecification when state error remains. That is the variance of the noise component; the full variance requires the state-error variance and covariance terms, unless the state contribution is known exactly.