Marginal and Joint Calibration

Why Marginal Calibration Is Not Enough

Simulation-based calibration (SBC) asks whether an inference procedure recovers parameters in repeated datasets generated from its own model and prior. A coordinate-wise rank check is useful: for example, it can reveal systematic bias in estimates of clearance or volume. But a model can give apparently correct ranks for each parameter separately while getting their relationship wrong. That relationship matters in pharmacokinetics: posterior dependence between parameters can affect derived quantities such as exposure, and decisions based on several parameters at once.

The key distinction is between the marginal distribution of each parameter and the joint distribution of all parameters. Uniform coordinate-wise ranks check the former. They do not, by themselves, establish the latter. This tutorial makes that limitation precise and shows how to add joint checks without assuming that coordinate ranks should be independent.

The standard rank result assumes a proper prior, data generated from the stated likelihood, and exact independent draws from the posterior conditional on the simulated data. In a continuous model, ties occur with probability zero. When ties are possible, the randomized tie-breaking rule is needed for the discrete uniform result. A practical Markov chain output is generally dependent; treating its draws as independent does not automatically preserve the exact finite-sample rank result.

What Coordinate-Wise Uniformity Establishes

Under the stated assumptions, conditional on the simulated data \(y\), the generating value \(\theta^\ast\) and the posterior draws are identically distributed: each is a draw from the posterior given \(y\). This exchangeability gives the rank result.

Proof. Fix a coordinate \(j\). Conditional on \(y\), the \(M+1\) values \(\theta_j^\ast,\theta_j^{(1)},\ldots,\theta_j^{(M)}\) are independent and identically distributed. With probability one they are distinct. By exchangeability, each of the \(M+1\) values is equally likely to be the smallest, second-smallest, and so on. Therefore \(\theta_j^\ast\) has each rank \(0,\ldots,M\) with probability \(1/(M+1)\). This holds for every \(y\), so averaging over the simulated data leaves the same rank probabilities. \(\square\)

This result is coordinate-wise. It does not say that the vector \((R_1,\ldots,R_p)\) has independent uniform components. Its joint distribution can depend on the posterior dependence structure and on how ranks are constructed. In particular, checking a flat histogram for every coordinate does not test whether a posterior sampler has the right copula, covariance, or tail dependence.

A Concrete Dependence Failure

Consider two standardized PK parameters, such as transformed clearance and volume parameters. Suppose their exact posterior, conditional on the data, is bivariate normal with unit marginal variances and correlation \(\rho=0.8\). An approximate sampler that preserves both normal marginals but draws the two coordinates independently has the correct marginal distributions and the wrong joint distribution.

The joint error can be seen with just one posterior draw (\(M=1\)). For each coordinate, the rank is either 0 or 1. Define the event of concordant ranks as \(R_1=R_2\): both generating values exceed their posterior draws, or both are below them.

To verify the sign formula, write the pair as \(X\) and \(Y=rX+\sqrt{1-r^2}Z\), where \(X\) and \(Z\) are independent standard normal variables. Their joint density is rotationally symmetric in the \((X,Z)\)-plane, so the angle is uniform. The conditions \(X>0\) and \(Y>0\) define a wedge of angle \(\pi/2+\arcsin(r)\); hence their joint probability is \((\pi/2+\arcsin(r))/(2\pi)\). By symmetry, the probability of both coordinates being negative is the same. Adding those probabilities gives \(1/2+\arcsin(r)/\pi\).

This calculation also clarifies an important diagnostic principle: the correct joint distribution of coordinate ranks is not generally the product of their marginal uniform distributions. Even an exact sampler can produce dependent ranks. A joint check should compare observed patterns with an appropriate reference under the intended model and rank construction, rather than test whether all rank components behave as independent uniforms.

Joint Checks Using Projections

One practical way to probe dependence is to examine pre-specified scalar functions of the parameter vector. For any fixed vector \(a\), form the projection \(a^\mathsf{T}\theta\) and rank the generating projection among the posterior-draw projections. Under the exact SBC assumptions, the generating projection and posterior projections are exchangeable, so its rank is uniform (with randomized tie-breaking if necessary). A projection that combines parameters can be sensitive to dependence that coordinate-wise checks miss.

No finite set of projections proves that a joint distribution is correct. Projections are diagnostic directions, not a universal certificate. Choose clinically meaningful combinations in advance, such as a log-scale contrast between clearance and volume, and inspect several directions where dependence would affect interpretation. For broader checks, compare multivariate rank patterns or summaries against a reference generated using the exact SBC design. Do not assume a single, universally applicable multivariate rank definition.

Diagnostics, Alternatives, and Limitations

A useful assessment combines complementary views. Coordinate rank histograms are direct checks of marginal calibration. Joint rank-pattern or dependence summaries examine whether coordinates move together as expected. Projection ranks test selected combinations. These checks answer different questions; none should be presented as a substitute for the others. This complements the broader model-calibration and adequacy checks introduced in the earlier material on Calibration, Model Adequacy, and Model Comparison.

• Check the simulation design. The prior, likelihood, data generator, parameter transformations, and posterior computation must correspond to the model being assessed. A mismatch can create misleading ranks.
• Check rank ties and posterior computation. Discrete parameters or rounded output require randomized tie handling. Correlated Markov chain draws can invalidate the iid finite-sample argument; inspect chain behavior and do not interpret nominal rank counts as exact when its assumptions fail.
• Inspect dependence-sensitive quantities. Plot rank pairs or summaries of concordance, and use pre-specified projections or contrasts tied to PK interpretation. Examine whether conclusions change across relevant directions.
• Account for finite simulation. A finite number of SBC replicates produces noisy histograms and noisy joint summaries. Apparent irregularity is not automatically evidence of failure, and a smooth marginal histogram is not proof of correct dependence.

Marginal ranks remain valuable: a joint check does not make them unnecessary. Conversely, a finite collection of joint diagnostics cannot establish correctness in every direction or tail. SBC assesses an inference procedure under the specified generative model and prior; it cannot by itself show that those assumptions adequately represent a real PK study.