Process and Observation Noise in Kalman State Estimates

Why the Two Noise Covariances Matter

A Kalman filter combines a model-based prediction with a noisy measurement. The balance depends on two different covariances: process-noise covariance, which represents uncertainty accumulated while the PK state evolves between observations, and observation-noise covariance, which represents measurement error. Increasing the first generally makes the filter more willing to adjust its state to new data; increasing the second generally makes it less willing.

This tutorial treats both covariances as specified inputs. It examines what their values imply for gains, filtered states, and uncertainty; it does not estimate them from data. The formulas apply exactly to a linear Gaussian state-space model. For an extended Kalman filter (EKF), the corresponding update uses a local linearization, so its uncertainty statements are approximate.

Prediction and Measurement Update

Consider states \(X_k\) at observation times \(t_k\). In a linear Gaussian model, write the transition and observation equations as

X_k=F_kX_{k-1}+b_k+W_k,\qquad Y_k=H_kX_k+V_k.

Here \(X_k\) is a state vector, \(Y_k\) is an observation vector, \(F_k\) is the transition matrix, and \(b_k\) contains known inputs such as a dose. The process noise \(W_k\) has mean zero and covariance \(Q_k\); the observation noise \(V_k\) has mean zero and covariance \(R_k\). Assume the initial state is Gaussian, both noise sequences are Gaussian, and process noises, observation noises, and the initial state are mutually independent. Assume also \(Q_k\) and \(R_k\) are positive semidefinite, with \(R_k\) positive definite for the inverse formulas below.

Let \(m_{k-1}\) and \(P_{k-1}\) be the filtered mean and covariance at the previous observation. The predicted state mean and covariance are

m_k^-=F_km_{k-1}+b_k,\qquad P_k^-=F_kP_{k-1}F_k^{\mathsf T}+Q_k.

The superscript minus denotes a prediction made before using \(Y_k\). Define the innovation, its covariance, and the Kalman gain by

\nu_k=Y_k-H_km_k^-,\qquad S_k=H_kP_k^-H_k^{\mathsf T}+R_k,\qquad K_k=P_k^-H_k^{\mathsf T}S_k^{-1}.

The filtered mean and covariance are

m_k=m_k^-+K_k\nu_k,\qquad P_k=P_k^--P_k^-H_k^{\mathsf T}S_k^{-1}H_kP_k^-.

Thus \(Q_k\) affects the update through \(P_k^-\), while \(R_k\) enters directly into \(S_k\). In an SDE-based PK model, the transition covariance \(Q_k\) represents the accumulated process uncertainty over the interval from \(t_{k-1}\) to \(t_k\). It is not automatically the instantaneous diffusion covariance; its value depends on the transition over that interval.

Definition: The innovation is the observed value minus its predicted mean, \(\nu_k=Y_k-H_km_k^-\). The innovation covariance \(S_k\) is the predicted covariance of that difference. Under the stated linear Gaussian assumptions, \(Y_k\mid Y_1,\ldots,Y_{k-1}\) has mean \(H_km_k^-\) and covariance \(S_k\).

What the Gain Does

The gain determines how much of the innovation is added to the predicted state. In a scalar model with \(H_k=1\), predicted variance \(P^-\), and assay variance \(R\), the gain and filtered variance simplify to

K=\frac{P^-}{P^-+R},\qquad P=\frac{P^-R}{P^-+R}.

A larger \(P^-\) relative to \(R\) assigns more weight to the observation. A larger \(R\) relative to \(P^-\) assigns less. This interpretation is relative: multiplying both variances by the same positive constant leaves the scalar gain unchanged, although it scales the filtered variance.

Result: In the scalar case \(H=1\), \(P^-=FP_{\mathrm{old}}F^{\mathsf T}+Q\) reduces to \(P^-=F^2P_{\mathrm{old}}+Q\). For \(R>0\), the gain \(K=P^-/(P^-+R)\) increases with \(Q\) and decreases with \(R\), provided \(P_{\mathrm{old}}\) and \(F\) are fixed.

To verify the directions, differentiate with respect to the variances:

\frac{\partial K}{\partial Q} =\frac{R}{(P^-+R)^2}>0,\qquad \frac{\partial K}{\partial R} =-\frac{P^-}{(P^-+R)^2}\leq 0.

The derivatives use \(\partial P^-/\partial Q=1\). They show how the scalar gain responds, not how far the filtered mean moves: that also depends on the observed innovation, whose sign and size can vary.

Worked Example: More Process Noise, More Measurement Influence

Suppose a concentration-like scalar state has predicted mean \(10\), prior filtered variance \(P_{k-1}=4\), transition coefficient \(F=0.8\), and observation \(Y_k=12\) with \(H=1\). Compare process variances \(Q=1\) and \(Q=0.25\), holding assay variance \(R=4\) fixed.

For \(Q=1\), the predicted variance is \(P^-=0.8^2(4)+1=3.56\). The innovation is \(12-10=2\), and \(S=3.56+4=7.56\). Therefore \(K=3.56/7.56\approx0.4709\), and the filtered mean is \(10+0.4709(2)\approx10.942\). The filtered variance is \(3.56(4)/7.56\approx1.884\).

For \(Q=0.25\), \(P^-=2.56+0.25=2.81\), \(S=2.81+4=6.81\), and \(K=2.81/6.81\approx0.4126\). The filtered mean is \(10+0.4126(2)\approx10.825\), and the filtered variance is \(2.81(4)/6.81\approx1.651\).

With the larger process variance, the filter treats the prediction as less precise, uses more of the positive innovation, and retains greater filtered uncertainty. The filtered mean changes by \(0.117\) between these choices; it does not change by the same amount as the gain because the innovation is multiplied by the gain.

Covariance Effects in Multiple PK States

For a vector state, a measurement of one compartment can update other compartments when their predicted errors are correlated. The gain is a matrix, and its entries depend on the full predicted covariance and observation covariance. Consequently, increasing a covariance does not generally increase every entry of the gain: the scalar directional rule is not a componentwise rule for matrices.

Worked Example: A Central-Compartment Assay Updates Two States

Let the predicted mean for central and peripheral states be \(m^-=(10,4)^{\mathsf T}\), with predicted covariance \(P^-=\begin{pmatrix}4&1\\1&2\end{pmatrix}\). A central-state assay has \(H=(1,0)\), measured value \(Y=12\), and variance \(R=1\). The off-diagonal covariance represents correlated prediction errors.

The predicted measurement is \(Hm^-=10\), so the innovation is \(2\). The innovation variance is \(S=HP^-H^{\mathsf T}+R=4+1=5\). The gain is \(K=P^-H^{\mathsf T}/5=(4,1)^{\mathsf T}/5=(0.8,0.2)^{\mathsf T}\).

The updated mean is \(m=m^-+K(2)=(11.6,4.4)^{\mathsf T}\). In particular, the peripheral state increases by \(0.4\), even though it was not directly observed; its prediction error was correlated with the central-state error.

The updated covariance is \(P=P^--P^-H^{\mathsf T}S^{-1}HP^-\). Here the subtracted matrix is \(\begin{pmatrix}0.8\\0.2\end{pmatrix}(4,1) =\begin{pmatrix}3.2&0.8\\0.8&0.2\end{pmatrix}\), giving \(P=\begin{pmatrix}0.8&0.2\\0.2&1.8\end{pmatrix}\). Both marginal variances decrease, but the peripheral variance decreases only because its state error was correlated with the measured state error.

A specified process covariance can create, remove, or change such correlations before an observation arrives. For example, if the preceding covariance is \(\operatorname{diag}(2,1)\), \(F=I\), and \(Q=\begin{pmatrix}2&1\\1&1\end{pmatrix}\), then the predicted covariance is \(\begin{pmatrix}4&1\\1&2\end{pmatrix}\), exactly the matrix used in the example. This illustrates why replacing a full \(Q\) with only its diagonal can alter updates to unobserved compartments.

Why More Uncertainty Cannot Improve Precision

There is a useful matrix statement about filtered uncertainty. For symmetric matrices, write \(A\preceq B\) when \(B-A\) is positive semidefinite. Consider a fixed observation matrix \(H\), a positive definite predicted covariance \(P^-\), and positive definite \(R\). The filtered covariance can also be written

P=\left((P^-)^{-1}+H^{\mathsf T}R^{-1}H\right)^{-1}.

This information-form identity gives a precise comparison of uncertainty for different covariance choices.

Result: For fixed \(H\), if \(P^-_1\preceq P^-_2\), then the corresponding filtered covariances satisfy \(P_1\preceq P_2\). For fixed \(P^-\), if \(0\prec R_1\preceq R_2\), then \(P_1\preceq P_2\). Thus increasing predicted uncertainty or assay-error covariance in this matrix order cannot reduce the filtered covariance.

To prove the first claim, inversion reverses the positive-definite matrix order, so \(P^-_2{}^{-1}\preceq P^-_1{}^{-1}\). Adding the same positive semidefinite matrix \(H^{\mathsf T}R^{-1}H\) preserves that order. Inverting again reverses it, yielding \(P_1\preceq P_2\). For the second claim, \(R_1\preceq R_2\) implies \(R_2^{-1}\preceq R_1^{-1}\). Therefore \((P^-)^{-1}+H^{\mathsf T}R_2^{-1}H\preceq(P^-)^{-1}+H^{\mathsf T}R_1^{-1}H\). Inverting gives \(P_1\preceq P_2\). If the changed process covariance makes \(P^-\) only positive semidefinite, the covariance update remains valid in its non-inverse form, and the same monotonicity follows by continuity from \(P^-+\epsilon I\) as \(\epsilon\) decreases to zero. \(\square\)

This result concerns uncertainty, not the filtered mean. A different \(Q\) or \(R\) changes the gain and therefore can move the mean either upward or downward, depending on the innovation and state-observation relationships.

Extended Kalman Updates and a Nonlinear Assay

For a nonlinear observation \(Y_k=h(X_k)+V_k\), the EKF linearizes around the predicted mean. With \(H_k\) set to the Jacobian of \(h\) at \(m_k^-\), it uses the same innovation, gain, and covariance formulas with predicted observation \(h(m_k^-)\) and \(S_k=H_kP_k^-H_k^{\mathsf T}+R_k\). These are local Gaussian approximations, not exact conditional moments in general.

Worked Example: Assay Error in a Saturating Concentration Measurement

Suppose a positive concentration state has predicted mean \(c^-=8\) and variance \(P^-=4\). An assay follows the saturating response \(h(c)=10c/(5+c)\). The observed response is \(Y=7.2\), with assay variance \(R=0.25\).

At \(c^-=8\), the predicted response is \(h(8)=80/13\approx6.1538\). The derivative is \(h'(c)=50/(5+c)^2\), so the local observation coefficient is \(H=50/169\approx0.29586\). The innovation is \(7.2-6.1538\approx1.0462\).

The innovation variance is \(S=H^2(4)+0.25\approx0.6001\), and the gain is \(K=4H/S\approx1.972\). The EKF filtered mean is approximately \(8+1.972(1.0462)=10.063\). Its local covariance update gives \(P\approx4-KH(4)\approx1.666\).

The numerical state shift is large relative to the predicted concentration. That is a reason to check the local approximation: the Jacobian was evaluated at 8, while the updated state is near 10.1. A smaller \(R\) would increase the gain and could make this linearization concern more pronounced; a larger \(R\) would reduce the local measurement influence.

Diagnostics, Alternatives, and Limitations

Before interpreting a filtered trajectory, check that each covariance is on the scale and time interval it is meant to describe. \(Q_k\) is for the transition between observation times; changing the interval generally changes it. \(R_k\) should reflect assay error on the observation scale, not state-scale uncertainty unless the measurement equation converts scales appropriately. Covariances must be symmetric and positive semidefinite. A computed negative variance, materially asymmetric covariance, or singular innovation covariance indicates a modeling or numerical problem.

The innovations also provide a practical diagnostic. Under a correctly specified linear Gaussian model, each innovation has mean zero conditional on past observations and covariance \(S_k\). Standardized innovations, such as \(\nu_k/\sqrt{S_k}\) in a scalar model, should not show persistent bias or unmodeled serial patterns. Many unusually large standardized innovations can indicate an incorrect noise scale, a misspecified transition or assay model, an unmodeled dose event, or outliers. They do not by themselves identify which covariance is wrong.

The Kalman filter is the exact conditional Gaussian update when the model assumptions hold. The EKF is a computationally convenient alternative for nonlinear models, but its local linearization can be poor when uncertainty is broad or the observation function is strongly curved over the plausible state range. Particle filtering is an alternative for more general nonlinear, non-Gaussian state distributions; it is not a reason to treat a questionable EKF covariance as exact. In all approaches, supplying \(Q\) and \(R\) is a modeling decision here, not a conclusion established by the filter.

Do not confuse \(Q_k\) and \(R_k\). Process uncertainty enters the predicted state covariance; assay uncertainty enters the observation covariance.
Do not add \(R_k\) directly to the state covariance. It contributes through \(S_k=H_kP_k^-H_k^{\mathsf T}+R_k\).
Do not assume a larger \(Q_k\) must move the state estimate in a particular direction. It changes the gain; the innovation determines the direction of the update.
Do not read an EKF covariance as an exact nonlinear posterior variance. It describes uncertainty under a local approximation.

Key takeaway: With fixed model structure, greater process uncertainty generally weakens confidence in the prediction, and greater assay error weakens confidence in the measurement. Their effects pass through the innovation covariance and gain; the resulting filtered mean depends on the observed innovation as well as on those covariances.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a scalar update, \(P^-=3\), \(R=1\), predicted mean \(5\), and observation \(7\). Find the gain, filtered mean, and filtered variance.
Keeping \(P^-=3\) fixed, what happens to the scalar gain if \(R\) increases from 1 to 3? Calculate both gains.
Why can an assay of the central compartment change the filtered peripheral-compartment mean?
In a nonlinear EKF update, why should a large state shift prompt a check of the linearization?

Solutions. (1) \(K=3/(3+1)=0.75\), so the filtered mean is \(5+0.75(7-5)=6.5\), and the filtered variance is \(3(1)/(3+1)=0.75\). (2) The gain changes from \(3/(3+1)=0.75\) to \(3/(3+3)=0.5\); the less precise assay receives less weight. (3) The predicted central and peripheral state errors can be correlated, so learning about the central error also changes the conditional mean of the peripheral state. If their covariance is zero and the observation measures only the central state, that indirect mean update is zero. (4) The EKF uses a Jacobian evaluated at the predicted mean. A large shift may place the updated state in a region where the measurement function has different curvature or slope, making that local approximation unreliable.