Sensitivity Derivatives In Nonlinear Models

Why Prediction Sensitivities Matter

In a nonlinear model, a small change in a parameter does not generally produce the same change in the predicted response at every setting. The derivative of the prediction with respect to a parameter describes that local change. Collecting these derivatives across observations gives a sensitivity matrix; its cross-products determine how much information the Gaussian observations provide about the model parameters.

The focus here is the mean model. The observations are independent Gaussian measurements around differentiable predictions, and their common variance is fixed and known. Under those conditions, Fisher information is a scaled cross-product of sensitivities. The calculation is local: derivatives are evaluated at a specified parameter vector, and describe the model near that value.

Assumptions and the Information Formula

Suppose the design settings \(x_1,\ldots,x_n\) are fixed, and the observation at setting \(x_i\) is \(Y_i=f(x_i,\boldsymbol{\theta})+\varepsilon_i\). Assume the errors are independent and \( \varepsilon_i\sim N(0,\sigma^2)\), where the same known variance \(\sigma^2\) applies to every observation and does not depend on \(\boldsymbol{\theta}\). The mean function must be differentiable with respect to the parameters at the value being considered. Write \(f_i(\boldsymbol{\theta})=f(x_i,\boldsymbol{\theta})\), and let \(\mathbf{y}\) and \(\mathbf{f}(\boldsymbol{\theta})\) be the vectors of observations and predicted means.

Under these assumptions, the log-likelihood, apart from no omitted parameter-dependent terms, is \(-\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_i(y_i-f_i(\boldsymbol{\theta}))^2\). Differentiating it gives the score vector. If \(J\) is the sensitivity matrix and \(\mathbf{r}=\mathbf{y}-\mathbf{f}(\boldsymbol{\theta})\) is the residual vector, then \(\mathbf{U}(\boldsymbol{\theta})=\sigma^{-2}J^\mathsf{T}\mathbf{r}\). This expression follows because the derivative of \(-\frac12(y_i-f_i)^2\) with respect to \(\boldsymbol{\theta}\) is \((y_i-f_i)\nabla f_i\).

The matrix product explains the role of the derivatives. Its \(j,k\) entry is \(\sigma^{-2}\sum_{i=1}^n(\partial f_i/\partial\theta_j)(\partial f_i/\partial\theta_k)\). A diagonal entry sums squared sensitivities to one parameter; an off-diagonal entry sums products of sensitivities to two parameters. The factor \(\sigma^{-2}\) gives less information to noisier observations. The result is a \(p\times p\) matrix, even though \(J\) is \(n\times p\).

Why the Cross-Product Is Fisher Information

The course material on likelihood scores and expected information defines Fisher information as the expected outer product of the score, equivalently as the negative expected Hessian under regularity conditions. Here the score expression gives a direct justification. Since \(J\) depends on the fixed settings and parameters, not on the random errors, and the errors have mean zero and covariance matrix \(\sigma^2 I_n\),

The same result follows from the expected negative Hessian. Let \(H_i\) be the \(p\times p\) Hessian matrix of second derivatives of \(f_i\). Differentiating the score shows that the observed information, the negative Hessian of the log-likelihood at the observed data, is \(\sigma^{-2}\sum_i\{(\nabla f_i)(\nabla f_i)^\mathsf{T}-r_iH_i\}\). Since \(E[r_i]=0\), taking its expectation removes the residual-times-curvature term and leaves the sensitivity cross-product. This distinction matters: the observed information for a particular data set need not equal \(J^\mathsf{T}J/\sigma^2\), even though its expectation does.

Worked Examples

Practical Checks and Common Failure Modes

The formula is simple, but the derivatives often are not. A reliable calculation starts by writing each prediction as a function of the full parameter vector, differentiating while holding the other parameters fixed, and only then substituting the parameter values. Keep a clear distinction between a parameter and a setting: a derivative with respect to a parameter measures local prediction change as that parameter changes, not as the design setting changes.

• Check dimensions. An \(n\times p\) matrix multiplied as \(J^\mathsf{T}J\) must produce \(p\times p\) information. A result with \(n\) rows and \(n\) columns usually signals a reversed product.
• Check the variance factor. The information is divided by \(\sigma^2\), not multiplied by it. Larger observation variance reduces information for the same sensitivities.
• Check the derivative signs and chain rule. A sign error can change off-diagonal entries. Differentiating a composite prediction requires the derivative of its inner function as well as its outer function.
• Check what is held fixed. The formula assumes the settings are fixed and the variance is parameter-independent. If the variance depends on parameters or differs by observation, the likelihood derivatives and information calculation must reflect that model instead.

Analytic derivatives are often the clearest option for a smooth model. Automatic differentiation can calculate the same derivatives while reducing manual algebra. Finite differences approximate a derivative by small changes in a parameter, so their results can be distorted by step sizes, rounding, or poorly scaled parameters; they are useful as a numerical check, not a substitute for checking the model assumptions. The expected-information cross-product is also not a replacement for observed information when the purpose is to describe curvature in a particular realized data set.

A common mistake is to treat \(J^\mathsf{T}J/\sigma^2\) as a universal information formula. It is the expected information for the stated Gaussian, fixed-variance mean model. Another is to interpret a large derivative as a guarantee of precise estimation: sensitivities must be considered together across parameters, and the information calculation alone does not check whether the model is scientifically appropriate or its assumptions are credible. Finally, derivatives at one parameter value describe local response changes there; for a strongly curved nonlinear mean function, they need not describe behavior far from that value.