Worked Exponential Model Information

From an Exponential Response to Information

An exponential response can change quickly at early times and slowly at later times. The observation times therefore affect how strongly the data respond to each model parameter. Fisher information makes this connection explicit: it combines the model’s parameter sensitivities with the observation variances to quantify local precision.

Consider the mean response \(m(t;A,k)=A\exp(-kt)\), where \(A>0\) is the response at time zero and \(k>0\) is the decay rate. The parameters are \(\theta=(A,k)^\mathsf{T}\). For observations at times \(t_1,\ldots,t_n\), the sensitivity matrix has one row per observation and one column per parameter. Its entries are derivatives of the expected response with respect to the parameters.

The derivatives follow directly from the mean function:

The first derivative describes how a small change in amplitude changes the expected response. The second describes how a small change in the decay rate changes it; the factor \(t\) means that at time zero the response has no sensitivity to \(k\). The two derivatives generally have different units: if response is measured in concentration units and time in hours, they have units of concentration per concentration unit and concentration per inverse hour, respectively.

Why the Sensitivities Produce Fisher Information

Suppose \(Y_i=m(t_i;A,k)+\varepsilon_i\), with independent errors \(\varepsilon_i\sim N(0,\sigma_i^2)\). Assume the variances are known, do not depend on \(A\) or \(k\), and the observation times are fixed. These assumptions are important: parameter-dependent variances add terms to the information, and correlated errors require a covariance-aware expression.

Let \(g_i\) be the sensitivity column vector at \(t_i\). Apart from constants not involving the parameters, the log-likelihood is the negative weighted sum of squared residuals. Differentiating it gives the score:

Each residual has mean zero and variance \(\sigma_i^2\). Independence makes the expected products of residuals from different observations zero. Taking the expected outer product of the score therefore gives the information:

This is the expected Fisher information under the stated normal model. It depends on the observation times, the variances, and the parameter values at which the sensitivities are evaluated. It does not depend on the particular residuals observed. The information is a local measure: it describes how sharply the likelihood changes near the parameter values used to calculate \(J\), not how informative the model must be everywhere.

How Observation Time Affects Each Parameter

For one observation with variance \(\sigma^2\), the information contribution is the outer product of its sensitivity row, divided by \(\sigma^2\). Thus a time contributes strongly to a parameter’s diagonal information when the magnitude of that parameter’s derivative is large. The contribution to the off-diagonal entries records how the two sensitivities move together at that time.

Reading the Matrix and Checking the Calculation

The diagonal entries of \(I\) summarize information in the individual parameter directions, while the off-diagonal entries reflect the joint pattern of sensitivities. The units follow from the derivatives: information for \(A\) has inverse squared amplitude units, and information for \(k\) has squared time units. The inverse information has corresponding variance units. Comparing raw diagonal entries across parameters can therefore be misleading unless units and parameter scales are considered.

A practical calculation should be checked in several ways. First, verify each derivative analytically or by comparing it with a sufficiently small finite-difference approximation at representative values. Second, check that the information matrix is symmetric and that its quadratic form is nonnegative: for any vector \(v\), \(v^\mathsf{T}Iv=\sum_i(v^\mathsf{T}g_i)^2/\sigma_i^2\geq0\). Third, check units, time units, and whether the variance or standard deviation was used in the weight. A variance belongs in the denominator of the information contribution.

A frequent error is to calculate an off-diagonal entry by adding the sensitivities themselves. The correct entry is the sum of products, \(\sum_i (\partial m_i/\partial A)(\partial m_i/\partial k)/\sigma_i^2\). Adding raw derivatives is not the matrix product \(J^\mathsf{T}WJ\), and may not even have the units of an information entry. Another error is to treat a large diagonal entry as proof of precise joint estimation: information in one direction alone does not establish that the full parameter vector is locally well determined.

Analytic derivatives are usually the clearest approach for this simple model. Finite differences are useful as an independent diagnostic, but their accuracy depends on step size and numerical scaling. Likewise, expected information gives a model-based local summary, whereas observed information uses the realized data and can differ in a finite sample. Neither removes uncertainty about model misspecification, the assumed error distribution, or the parameter values used to evaluate the sensitivities.

The information matrix is useful because it links a proposed sampling schedule and an explicit response model to local precision before data are collected. It also reveals which parameter derivatives are driving the calculation. Its limits are equally important: it depends on the assumed model, error structure, and evaluation values; it is local rather than global; and small or weakly informative sensitivities can make inverse-information approximations unstable. In a practical analysis, these calculations should be interpreted alongside model diagnostics and scientific knowledge, not as a substitute for them.