Two Information Quantities, Two Stages of an Experiment
When planning a QSP experiment, information calculations describe what a design is expected to provide under a specified model and parameter value. After measurements are collected, the likelihood has a particular shape for the responses that actually occurred. These are related but distinct quantities: expected information averages over possible responses, while observed information is calculated from the realized responses.
This distinction matters especially for nonlinear ordinary differential equation (ODE) models. Their predictions can curve as parameters change, so the observed likelihood curvature depends not only on prediction sensitivities but also on how far the measurements fall from the predictions. Expected information is useful for prospective design comparison; observed information describes local curvature in the likelihood for the data in hand.
A design calculation must specify a parameter value, or a distribution over plausible parameter values, because information generally varies with \(\theta\). A local expected-information calculation uses a planning value such as \(\theta_0\), along with planned doses, sampling times, measured outputs, and an observation-error model. It is not an information value guaranteed to be obtained. Observed information, in contrast, requires actual responses and is usually evaluated at a stated parameter value, often the fitted estimate.
Gaussian Observations of a Nonlinear ODE Model
Suppose an ODE system has state \(x(t;\theta)\), governed by \(dx/dt=f(x,\theta,t)\), and the predicted measurement at observation \(i\) is \(m_i(\theta)\). Assume the observed response is \(Y_i=m_i(\theta)+\epsilon_i\), where the errors are independent normal variables with mean zero and known variances \(\sigma_i^2\). The model is assumed correctly specified, the prediction is twice differentiable in the parameters, and the observation schedule and error variances are fixed independently of the responses.
Write \(r_i=y_i-m_i(\theta)\) for the residual, \(J_i=\partial m_i/\partial\theta\) for the row vector of sensitivities, and \(H_i=\partial^2m_i/\partial\theta\partial\theta^\mathsf{T}\) for the prediction Hessian. The Gaussian log-likelihood, omitting constants that do not depend on \(\theta\), is:
The score (the gradient of the log-likelihood) is \(\sum_i J_i^\mathsf{T}r_i/\sigma_i^2\). Differentiating once more gives the observed information:
The first term is built from local prediction sensitivities. The second is a residual-weighted curvature correction: it vanishes if the residual is zero, but can either increase or decrease the observed curvature. Because the model is assumed to have mean prediction \(m_i(\theta)\), \(E_\theta(r_i)=0\). Taking the expectation of the observed-information expression therefore yields:
This is the expected Fisher information for this fixed-variance Gaussian model. The equality follows because the residual correction has expectation zero. It does not mean that the observed information for one realized data set must equal its expectation. The formula also relies on the stated likelihood and assumptions; if variance depends on parameters, or errors are correlated, the likelihood and information expressions must include those features.
Worked Example: Expected Information From a Clearance ODE
Worked Example: Expected Information From a Clearance ODE
Consider the hypothetical one-compartment decay model \(dC/dt=-kC\), with \(C(0)=D=10\) mg/L. Its prediction is \(m(t;k)=10e^{-kt}\), where \(k\) is a clearance-rate parameter in day\(^{-1}\). Suppose a single concentration is planned at \(t=1\) day, with independent normal measurement error of known standard deviation \(0.5\) mg/L. Calculate expected information at the planning value \(k_0=0.5\) day\(^{-1}\).
The sensitivity is \(J=\partial m/\partial k=-tDe^{-kt}=-t\,m(t;k)\). At the stated planning value and sampling time, \(m=10e^{-0.5}=6.0653\) mg/L and \(J=-6.0653\) mg/L per day\(^{-1}\). The expected information for this single observation is:
The units are inverse squared units of \(k\), hence day\(^2\). This is a local expected-information calculation before measurement. It describes sensitivity relative to the assumed measurement error at \(k_0\); it does not promise that the realized data will yield this exact likelihood curvature.
For this one-parameter model, \(\mathcal I_E(k)=D^2t^2e^{-2kt}/\sigma^2\). Differentiating its logarithm with respect to \(t\) gives \(2/t-2k\), so for \(t>0\) the local expected information is maximized at \(t=1/k\). At \(k_0=0.5\) day\(^{-1}\), that time is 2 days. This is a model- and parameter-specific design result, not a universal rule for sampling time.
Worked Example: Computing Realized Information
Worked Example: Computing Realized Information
Use the same hypothetical ODE, error standard deviation, sampling time, and evaluation value \(k=0.5\) day\(^{-1}\). Suppose the observed concentration is \(y=6.8\) mg/L. Calculate observed information at this stated parameter value and compare it with expected information.
The prediction is \(m=6.0653\) mg/L, so the residual is \(r=6.8-6.0653=0.7347\) mg/L. At \(t=1\) day, the first and second derivatives are \(J=-m=-6.0653\) and \(H=m=6.0653\), with units appropriate to first and second derivatives with respect to \(k\). Thus:
For clarity, retaining the unrounded values through the calculation gives a numerator of approximately \(32.332\ ({\rm mg/L})^2{\rm day}^2\), divided by \(0.25\ ({\rm mg/L})^2\), or about \(129.3\ {\rm day}^2\). This realized curvature is smaller than the expected value of \(147.2\ {\rm day}^2\) at the same parameter and design because the positive residual and positive prediction curvature make the residual correction subtract from the sensitivity term.
The calculation is at the specified \(k\), not necessarily at the maximum-likelihood estimate. If observed information is reported at an estimate, the prediction, residual, sensitivities, and Hessians must all be evaluated at that estimate. The numerical value alone is not a confidence interval and does not account automatically for nonlinearity away from that evaluation point.
Why Observed Curvature Can Behave Unexpectedly
Expected Fisher information for the Gaussian model above is a sum of positive-semidefinite sensitivity contributions. Observed information includes residual-weighted Hessians and need not be positive. A negative value in a one-parameter example means the log-likelihood is locally convex in that parameter at the evaluation point, rather than locally concave. It does not mean that expected Fisher information is negative or that the likelihood itself is invalid.
Worked Example: A Large Residual Reverses Local Curvature
Keep the one-observation ODE setup and evaluate at \(k=0.5\) day\(^{-1}\), but suppose the hypothetical observed concentration is \(14.0\) mg/L. The prediction remains \(6.0653\) mg/L, so \(r=7.9347\) mg/L. The expected information remains \(147.2\) day\(^2\), because it depends on the model sensitivity and error variance, not this response. The observed information is:
The negative result indicates local convexity of the log-likelihood at the specified \(k\). Here the response is far above the prediction, and the residual-curvature term exceeds the sensitivity term. This local result should prompt checks of the data, model assumptions, and evaluation point; it is not a reason to replace the expected design calculation with a negative “amount of information.”
For an ODE model, the sensitivity matrix is usually obtained by differentiating the state equations. If \(S_j(t)=\partial x(t;\theta)/\partial\theta_j\), then, for differentiable \(f\), it satisfies \(dS_j/dt=f_xS_j+f_{\theta_j}\), with initial conditions obtained by differentiating the initial state with respect to \(\theta_j\). These sensitivities feed into \(J_i\) through the observation function. Expected information therefore depends on the assumed parameter values, the ODE dynamics, and the planned observations, even before responses exist.
Set the planning parameter value or distribution, ODE inputs, observation times, measured outputs, and observation-error model.
Solve the ODE and sensitivity equations at the planning values, then form the prediction derivatives for each planned observation.
Use the model likelihood to average over possible responses. For independent Gaussian errors with fixed variances, sum \(J_i^\mathsf{T}J_i/\sigma_i^2\).
Use the realized responses and a stated parameter value in the negative log-likelihood Hessian, including residual-dependent terms.
Using the Distinction in Design and Analysis
Expected information is the natural quantity for comparing candidate designs before data are collected. For example, the design comparison can use a determinant, a trace, or a selected parameter-specific element of the expected-information matrix, provided the chosen criterion and parameter scale are appropriate. Such criteria compare model-based expectations at the stated planning values; they are not guarantees of realized precision. If parameters are uncertain before the experiment, one can evaluate expected information across plausible values and average or otherwise summarize it according to a stated design objective.
Observed information is a data-dependent description of local likelihood curvature. It can help explain why a particular data set is more or less informative than expected under the model. It is not interchangeable with expected information in a prospective design criterion. Nor should either matrix automatically be read as the exact uncertainty in a nonlinear model: using an inverse information matrix as an approximate covariance relies on local and sampling approximations. Profile likelihood, introduced in the earlier material on profile likelihood, can reveal likelihood shape beyond a local curvature summary.
Diagnostics should focus on whether the inputs and assumptions behind the calculation are credible. Check that the ODE solution and sensitivities are numerically stable, that units and parameter scales are consistent, and that the assumed observation variance and correlation structure match the measurement process. After data collection, inspect residuals and compare the observed curvature at relevant parameter values with the design expectation. A large difference may reflect random variation, a poor planning parameter value, model discrepancy, an outlying response, or a computational error; the information calculation alone cannot identify which explanation is correct.
A common mistake is to say, “The design had expected information of 147 day\(^2\), so the observed data must have supplied 147 day\(^2\).” That reasoning confuses an average over possible responses with one realized data set. The correction is to label each quantity, state the parameter value and design, and calculate observed information using the actual responses when discussing realized likelihood curvature. Another mistake is to omit the residual-Hessian term from observed information in a nonlinear model. That omission gives the sensitivity-based expected-information expression instead, not generally the realized negative Hessian.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For independent Gaussian observations with known, parameter-independent variances, why does the expected-information formula omit the residual-Hessian term?
- In the clearance example, what does an observed information value of \(129.3\ {\rm day}^2\) describe, and why is it not a guaranteed property of the sampling design?
- Can observed information be negative in a nonlinear model while expected Fisher information is positive? Explain.
- Before responses are collected, which quantity is suitable for comparing candidate sampling designs under a specified planning model?
Solutions. (1) The residual has mean zero under the correctly specified model, so the expectation of its product with the prediction Hessian is zero. (2) It is the negative log-likelihood curvature at the stated parameter value for the particular hypothetical observed response; other possible responses would give different residual corrections. (3) Yes. A sufficiently large residual can make the residual-curvature correction dominate, giving locally negative observed information, even though the expected Gaussian information is a sum of nonnegative sensitivity contributions. (4) Expected information, evaluated using the specified planning parameter value or parameter distribution, design, and observation-error model.