Testing Whether an Effect Compartment Improves Fit

Why Compare the Models?

When plasma concentration changes before the measured response, an instantaneous concentration–response model can miss the timing of the effect. An effect-compartment model represents a delay by letting the hypothetical effect-site concentration \(C_e\) approach plasma concentration \(C_p\) over time. The question is not simply whether the delayed model can fit better: it has an additional parameter, \(k_{e0}\), and so can use extra flexibility to reduce residual error. A sound comparison considers the likelihood improvement, the structure of the residuals, and whether predictions improve in a meaningful way.

The models compared here use the same concentration input, response function, observations, and residual-error assumptions. Only the treatment of delay differs. This isolates the question of whether an effect-compartment delay is supported by the data; it does not establish a physiological mechanism for that delay.

Definition: In an instantaneous response model, the response at time \(t\) depends on the plasma concentration at that time, for example \(R(t)=E_0+S C_p(t)\). In a linear effect-compartment response model, \(C_e\) evolves according to \(\frac{dC_e(t)}{dt}=k_{e0}\{C_p(t)-C_e(t)\}\), and the response is \(R(t)=E_0+S C_e(t)\). Here \(E_0\) is baseline response, \(S\) is the response slope, and \(k_{e0}>0\) is the equilibration rate constant, with units of inverse time. The same comparison can be made with another specified response function, provided it is used consistently in both models.

Likelihood Comparison and Its Limit

For clarity, first suppose there is one response profile with \(n\) observations \(Y_i\) at times \(t_i\). Conditional on the concentration input and the model predictions, assume

Y_i=R(t_i;\theta)+\epsilon_i,\qquad \epsilon_i\overset{\text{ind}}{\sim}N(0,\sigma^2).

The errors are assumed independent, normally distributed, and of constant variance; the concentration input is treated as known; and both models use the same observations and likelihood. If observations are correlated, heteroscedastic, or repeated within subjects, the likelihood must represent those features rather than treating all errors as independent with one variance. Parameter estimates must also be obtained with adequate numerical optimization and the effect-site trajectory must be computed accurately from the concentration input.

Let \(SSE_I\) be the residual sum of squares under the instantaneous model, and \(SSE_E\) the residual sum of squares under the effect-compartment model. For either model, the Gaussian log likelihood is maximized over \(\sigma^2\) at \(\widehat{\sigma}^2=SSE/n\). Substituting this estimate into the log likelihood gives the likelihood-ratio statistic below, where the alternative is the effect-compartment model.

-2\log\left(\frac{L_I}{L_E}\right) = n\log\left(\frac{SSE_I}{SSE_E}\right).

A larger statistic means a greater improvement in maximized likelihood for the effect-compartment model. The formula does not, by itself, determine a p-value. In particular, the instantaneous model is the limiting case \(k_{e0}\to\infty\): the effect-site concentration approaches the plasma concentration increasingly quickly. This is not a finite value of \(k_{e0}\) under the effect-compartment model. Consequently, the usual regular likelihood-ratio result that the statistic has an approximate chi-square distribution with one degree of freedom does not automatically apply. Treating it as an ordinary one-parameter nested test can give an unjustified p-value.

A practical calibration is a parametric bootstrap under the instantaneous model. Fit the instantaneous model, simulate response datasets under that fitted model and its residual distribution, refit both models to every simulated dataset, and recalculate the statistic each time. The fraction of simulated statistics at least as large as the observed statistic estimates how often the apparent improvement would occur if the instantaneous model generated the data. This calibration should use the same fitting constraints, concentration inputs, sampling times, and error model as the analysis.

Worked Examples

Worked Example: Bootstrap Test for a Delayed Response

Hypothetical data. Suppose six response measurements are collected over a concentration profile that rises and then falls. Both candidate models use the same linear response function and an independent, constant-variance Gaussian error model. A numerical fit gives \(SSE_I=17.36\) response-units-squared for the instantaneous model and \(SSE_E=0.1980\) response-units-squared for the effect-compartment model. These are fitting results for this hypothetical dataset, not observations from a real study.

The observed likelihood-ratio statistic is

T_{\mathrm{obs}} =6\log\left(\frac{17.36}{0.1980}\right) =6\log(87.68) \approx 26.84.

To calibrate it, suppose 2,000 datasets are simulated under the fitted instantaneous model and both models are refit to every dataset. If 9 simulated statistics are at least 26.84, a finite-simulation estimate with the usual add-one correction is

\widehat{p}_{\mathrm{boot}} =\frac{9+1}{2000+1} =\frac{10}{2001} \approx 0.00500.

With a prespecified significance level of \(\alpha=0.05\), this result is evidence against the instantaneous model under the stated assumptions: statistics this large were uncommon in the bootstrap datasets generated under that model. It is not the probability that the instantaneous model is true, nor does it prove a physiological effect compartment exists. The bootstrap p-value also has Monte Carlo uncertainty because it is based on a finite number of simulations.

Worked Example: What Residuals Add to the Test

For the same hypothetical profile, suppose the response observations are \(10.0,\ 12.2,\ 15.8,\ 18.1,\ 16.2,\ 12.4\) response units. A fit produces instantaneous-model predictions \(11.5,\ 13.9,\ 17.5,\ 16.3,\ 13.9,\ 11.5\), and effect-compartment predictions \(10.2,\ 12.0,\ 15.6,\ 18.0,\ 16.4,\ 12.5\). Define each residual as observed minus predicted response.

The instantaneous residuals are \(-1.5,\ -1.7,\ -1.7,\ 1.8,\ 2.3,\ 0.9\) response units. Their signs change in a time-ordered pattern: the model tends to overpredict early and underpredict later. The effect-compartment residuals are \(-0.2,\ 0.2,\ 0.2,\ 0.1,\ -0.2,\ -0.1\) response units, which are smaller and show less obvious directional structure. The squared residuals for the instantaneous fit sum to

$$ (-1.5)^2+(-1.7)^2+(-1.7)^2+(1.8)^2+(2.3)^2+(0.9)^2 =17.37, $$

which is consistent with the reported \(SSE_I=17.36\) to rounding. The rounded effect-compartment residuals sum to \(0.18\) response-units-squared, close to the unrounded fit result \(SSE_E=0.1980\). Residual patterns support investigating delay because their direction changes systematically over time under the instantaneous fit. With only six observations, however, these plots and summaries are diagnostic clues, not decisive proof; the likelihood comparison and checks on additional or replicated data remain important.

Worked Example: A Fit Improvement That Does Not Ensure Better Prediction

Suppose a second hypothetical analysis has \(n=20\), \(SSE_I=40.0\), and \(SSE_E=36.0\), in squared response units. The likelihood-ratio statistic is

T=20\log\left(\frac{40.0}{36.0}\right) =20\log(1.111) \approx 2.107.

The effect-compartment model fits the observed data better, but the improvement is modest compared with the first example. A bootstrap calibration is still needed before assigning a p-value. Suppose, separately, that predictions at prespecified held-out times have root mean squared errors of \(1.80\) response units for the instantaneous model and \(1.75\) response units for the effect-compartment model. The held-out improvement is only \(0.05\) response units. It would be inappropriate to claim a substantial predictive gain from this small difference without considering its uncertainty and the clinical relevance of the response scale. In this example, the likelihood gain and the predictive evidence tell related but not identical stories.

Diagnostics and Predictive Checks

A likelihood comparison answers whether the observed data are less compatible with the instantaneous model than with the fitted effect-compartment model, relative to the assumed likelihood. It does not show that either model is adequate. Examine residuals against time, predicted response, and concentration, and inspect their sequence across rising and falling concentration phases. A persistent time trend, a changing residual spread, or runs of residuals with the same sign can indicate that the error model or the response model is inadequate. A few extreme residuals may also dominate the sum of squares and the likelihood ratio.

Prediction checks ask whether each model reproduces the aspects of the data it is intended to describe. Compare predicted and observed responses at informative times, particularly where concentration is changing rapidly and where rising and falling phases provide different timing information. When feasible, reserve observations or profiles for prediction assessment rather than using every measurement to estimate parameters. A model that lowers training residual error but fails to predict held-out responses has not demonstrated a reliable predictive advantage. Visual predictive checks for delayed concentration-effect data provide a complementary way to assess whether the model reproduces the observed response distribution and time course.

Specify comparable models.
Use the same response observations, concentration input, response function, and error assumptions. State initial conditions and parameter constraints before fitting.

Fit and calculate the likelihood improvement.
Check numerical convergence and calculate the likelihood-ratio statistic from the maximized likelihoods, or from the residual sums of squares when the shared Gaussian assumptions apply.

Calibrate the comparison.
Because the instantaneous model corresponds to an infinite equilibration rate, use a suitable calibration such as a parametric bootstrap instead of assuming an ordinary one-degree-of-freedom chi-square reference applies.

Assess adequacy and prediction.
Inspect residuals and evaluate predictive checks. Report evidence for improved fit separately from evidence for improved prediction or a meaningful scientific interpretation.

Alternatives, Limitations, and Common Pitfalls

The likelihood-ratio test is designed for a comparison framed through the fitted likelihoods. AIC is an alternative for comparing relative expected information loss while penalizing model complexity; lower AIC is preferred among the candidate models, but an AIC difference is not a p-value. Held-out prediction or cross-validation focuses more directly on predictive performance, although results depend on how observations or profiles are held out. For each approach, use the same underlying data structure and error model in both candidates.

These approaches have different targets. A small bootstrap p-value indicates that the observed likelihood gain would be unusual under the fitted instantaneous model and bootstrap assumptions. A lower AIC favors one candidate in the specified comparison. Better held-out prediction supports predictive usefulness in the validation setting. None alone establishes that the estimated \(k_{e0}\) is precise, that all relevant sources of uncertainty have been represented, or that the effect compartment corresponds to a literal anatomical site. A fitted delay may also reflect data-collection lag or other model misspecification, so interpretation should remain appropriately limited.

Common mistake: “The effect-compartment model adds one parameter, so the likelihood-ratio statistic must be compared with a chi-square distribution with one degree of freedom.” The correction is that the instantaneous model is obtained as \(k_{e0}\to\infty\), not by setting \(k_{e0}\) to an ordinary finite value in the effect-compartment model. The regular chi-square approximation therefore cannot simply be assumed. Calibrate the test appropriately, commonly by parametric bootstrap, and separately assess residual and predictive adequacy.

Key takeaway: A convincing case for an effect-compartment model requires more than a smaller residual sum of squares. Use a likelihood comparison with appropriate calibration, inspect whether the instantaneous model leaves systematic residuals, and determine whether the added delay improves predictions relevant to the intended use.

Check Your Understanding

Answer each question using the model assumptions and comparison principles above.

Why is the instantaneous model a limiting case of the effect-compartment model rather than an ordinary finite-parameter restriction?
For \(n=10\), \(SSE_I=12.0\), and \(SSE_E=8.0\), calculate the Gaussian likelihood-ratio statistic.
What does a parametric-bootstrap p-value of 0.02 mean when testing the instantaneous model at \(\alpha=0.05\)?
Name two residual patterns that could indicate model inadequacy even when the effect-compartment model has a larger likelihood.
How does AIC answer a different question from held-out prediction?

Solutions. (1) The effect-site concentration approaches plasma concentration as \(k_{e0}\) tends to infinity; there is no finite \(k_{e0}\) that gives exact instantaneous equilibration. (2) \(T=10\log(12.0/8.0)=10\log(1.5)\approx4.055\). This statistic alone does not give a valid p-value without an appropriate reference distribution or calibration. (3) Under the bootstrap procedure and its assumptions, statistics at least this extreme occurred about 2% of the time under the fitted instantaneous model; at the prespecified 0.05 level, this is evidence against that model. It is not the probability that the null model is true. (4) Examples include a systematic time trend or changing residual spread; runs of residuals with the same sign are another warning. (5) AIC compares penalized relative fit for candidate models, while held-out prediction evaluates performance on observations not used for fitting.