EKF Updates with Nonlinear Concentration Measurements

Why a Nonlinear Assay Changes the Update

In a linear Gaussian state-space model, the measurement update conditions a Gaussian state distribution on a linear observation. Many concentration assays do not have that form: their expected response may saturate at high concentration, transform concentration to a signal, or combine concentrations through a nonlinear calibration curve. The extended Kalman filter (EKF) handles such an observation by replacing it locally with a linear function, then applying a Gaussian update to that approximation.

The result depends on where the assay is linearized. The observation Jacobian describes local sensitivity at the predicted state mean, while the innovation covariance accounts for both uncertainty in the predicted state and assay error. These quantities are useful diagnostics as well as ingredients in the update.

The predicted distribution used for the update is usually supplied by the state-prediction step. For the derivation below, assume it is Gaussian, \(X\sim N(m^-,P^-)\), with \(P^-\) positive definite. The assay noise is Gaussian as specified above. These assumptions make conditioning on the linearized observation exact for that surrogate model. They do not make the update exact for the original nonlinear observation.

The Local Measurement Model and EKF Formulas

A first-order Taylor expansion of \(h\) around \(m^-\) gives

For an observed value \(z\), define the innovation \(r=z-h(m^-)\). Under the linearized model, \(Z-h(m^-)\) is approximately \(H(X-m^-)+V\). Its mean is zero and its covariance is \(S=HP^-H^\mathsf T+R\), called the innovation covariance. Since \(R\) is positive definite, \(S\) is positive definite and invertible.

Here \(K\) is the Kalman gain, \(m^+\) is the updated state mean, and \(P^+\) is the updated state covariance. The gain maps the measurement discrepancy into a state correction. Its size and direction depend on prior state uncertainty, assay sensitivity, and measurement noise.

Why the Formulas Hold for the Linearized Model

The Taylor approximation defines an affine observation model. Conditional on the Gaussian predicted state and independent Gaussian assay error, that model is jointly Gaussian with the state. The following derivation makes clear both why the update has its familiar form and what its exactness applies to.

To derive the result, write \(u=X-m^-\). The prior density contributes a quadratic term \(u^\mathsf T(P^-)^{-1}u\) to the negative twice-log-density. The observation likelihood contributes \((r-Hu)^\mathsf T R^{-1}(r-Hu)\). Collecting the terms that depend on \(u\) gives

Completing the square shows that the conditional covariance is \(C=((P^-)^{-1}+H^\mathsf T R^{-1}H)^{-1}\), and the conditional mean displacement \(\delta\) solves \(((P^-)^{-1}+H^\mathsf T R^{-1}H)\delta=H^\mathsf T R^{-1}r\). Set \(\delta=P^-H^\mathsf T S^{-1}r\). Substitution into the left side gives \(H^\mathsf T S^{-1}r+H^\mathsf T R^{-1}HP^-H^\mathsf T S^{-1}r\). Since \(HP^-H^\mathsf T=S-R\), this equals \(H^\mathsf T S^{-1}r+H^\mathsf T R^{-1}(S-R)S^{-1}r=H^\mathsf T R^{-1}r\). Thus the displacement is \(Kr\).

The covariance identity \(C=P^- -P^-H^\mathsf T S^{-1}HP^-\) follows by multiplying the right-hand side by \((P^-)^{-1}+H^\mathsf T R^{-1}H\), which yields the identity matrix. Therefore \(C=P^+\), proving the stated formulas for the affine model. Since the original model uses \(h(X)\), not its affine approximation, this proof does not establish exact conditional moments for the original nonlinear model. \(\square\)

State Uncertainty and a Correlated Two-Compartment Example

The covariance update has a useful equivalent form. In the affine model, the updated error is the prior error after applying the gain, minus the part explained by the observation. More explicitly, if the prior estimation error is \(e^-=X-m^-\), then the updated error is \(e^+=(I-KH)e^- - KV\). Independence of \(e^-\) and \(V\) gives

The first expression, often called the Joseph form, makes nonnegative covariance contributions visible and can be more stable in numerical computation. The second expression makes the uncertainty reduction transparent. For any vector \(a\), \(a^\mathsf T(P^- -P^+)a=(HP^-a)^\mathsf T S^{-1}(HP^-a)\geq0\). Thus the update cannot increase uncertainty in any linear combination of states in this affine Gaussian model.

Diagnostics, Alternatives, and Limits

A small residual does not by itself guarantee a reliable update. The EKF uses the slope at one point to represent the assay over the range of plausible states. That approximation is suspect when the prior uncertainty spans strong curvature, a saturation plateau, a threshold, or a physical boundary. Useful checks include the predicted response \(h(m^-)\), the innovation \(r\), the innovation standard deviation \(\sqrt{S}\), and whether the updated state remains physically plausible. The standardized innovation \(r/\sqrt{S}\) is a discrepancy scaled by both predicted state uncertainty and assay error; a large magnitude signals tension with the predicted measurement distribution, though it does not identify the cause by itself.

For example, let \(C\) have predicted mean 1 and variance 0.04, use \(h(C)=C/(1+C)\), and observe \(z=0.04\) with \(R=0.0001\). Here \(h(1)=0.5\), \(H=1/4\), and \(r=-0.46\). The innovation variance is \(S=(1/16)(0.04)+0.0001=0.0026\), so \(\sqrt S\approx0.05099\), and the standardized innovation is about \(-9.02\). The gain is \(K=(0.04)(0.25)/0.0026\approx3.84615\), yielding \(m^+=1+3.84615(-0.46)\approx-0.76923\). The covariance becomes \(P^+=0.04-(0.01)^2/0.0026\approx0.001538\). The very small updated variance does not rescue the physically implausible negative mean: this is a warning that the local approximation is inappropriate for this observation.

The assay response at \(C=0\) is zero. The observed value \(0.04\) is therefore four assay-error standard deviations above that baseline, because \(\sqrt R=0.01\). This is distinct from the innovation at the predicted mean: relative to the predicted response of \(0.5\), the residual is \(-0.46\). Keeping the baseline residual, the prior-prediction innovation, and their different uncertainty scales separate prevents misstatements about standardized discrepancies.

When \(h\) is linear, the ordinary Kalman filter gives the exact Gaussian update under the same Gaussian assumptions. For a genuinely nonlinear \(h\), the EKF is a first-order, moment-based approximation. Iterated linearization can reconsider the expansion point, while sigma-point methods approximate nonlinear transformations using selected points rather than one Jacobian. Direct integration of the nonlinear likelihood is another option when accuracy matters and the state dimension permits it. These alternatives do not make the EKF formulas exact for the original nonlinear model.

The EKF is most defensible when the predicted state is reasonably concentrated around \(m^-\), the assay function is smooth over that region, and the resulting update respects the model's physical constraints. It can mislead when the response is nearly flat, when multiple state values produce similar assay responses, or when a local Gaussian update crosses a boundary such as zero concentration. In those cases, inspect the observation model and uncertainty assumptions rather than interpreting a numerically precise \(P^+\) as proof of a reliable approximation.