Calibration Diagnostics for Conditional Distributions

Why Calibration Must Be Checked Conditionally

A conditional distribution forecast makes a claim about the distribution of an outcome at each covariate value. For example, a forecast might assign different means and variances to patients with different measured characteristics. A diagnostic that pools every forecast and outcome together can miss systematic errors that cancel across covariate groups. The key question is therefore not only whether the forecasts look right on average, but whether they are reliable across the covariates on which they depend.

Let \(X\) be the covariate and \(Y\) the outcome. Write \(F_0(y\mid x)=P(Y\le y\mid X=x)\) for the true conditional CDF, and \(G(y\mid x)\) for a forecast conditional CDF. The forecast is exactly calibrated conditionally when it agrees with the true conditional law, for almost every covariate value under the distribution of \(X\). This is a population property; diagnostics based on finite data can provide evidence about it, but cannot verify it everywhere.

Definition: A conditional distribution forecast \(G(\cdot\mid X)\) is conditionally calibrated if \(G(y\mid X)=F_0(y\mid X)\) for every \(y\), almost surely in \(X\). A transform is marginally uniform if its distribution, after averaging over \(X\), is Uniform\((0,1)\). Marginal uniformity is weaker than conditional calibration.

The Probability Integral Transform for Continuous Outcomes

If a forecast CDF is continuous, its probability integral transform (PIT) for an observed pair \((X,Y)\) is \(U=G(Y\mid X)\). When the forecast is correct, \(U\), conditional on each covariate value, has a Uniform\((0,1)\) distribution. For the converse, assume that for almost every \(x\), \(G(\cdot\mid x)\) is continuous and strictly increasing over the outcome support. Then conditional uniformity of \(G(Y\mid x)\) implies that the true conditional CDF equals \(G(\cdot\mid x)\). Strict increase is a useful sufficient condition for this converse; without it, the transform can lose information on flat portions of the forecast CDF.

Theorem: Suppose that, for almost every \(x\), \(G(\cdot\mid x)\) is a continuous, strictly increasing CDF on the support of \(Y\mid X=x\). Then \(G(Y\mid X)\mid X=x\) is Uniform\((0,1)\) for almost every \(x\) if and only if \(F_0(\cdot\mid x)=G(\cdot\mid x)\) for almost every \(x\).

Proof. First suppose \(F_0(\cdot\mid x)=G(\cdot\mid x)\). For any \(t\in(0,1)\), continuity and strict increase give a unique \(y_t\) with \(G(y_t\mid x)=t\). Thus \(P(G(Y\mid x)\le t\mid X=x)=P(Y\le y_t\mid X=x)=F_0(y_t\mid x)=G(y_t\mid x)=t\). The probabilities at \(t=0\) and \(t=1\) are 0 and 1, so the conditional distribution is uniform.

Conversely, suppose \(G(Y\mid x)\mid X=x\) is uniform. For any \(y\) in the support, strict increase implies that \(G(Y\mid x)\le G(y\mid x)\) exactly when \(Y\le y\). Therefore \(F_0(y\mid x)=P(Y\le y\mid X=x)=P(G(Y\mid x)\le G(y\mid x)\mid X=x)=G(y\mid x)\). This holds at every \(y\) in the support, proving equality of the conditional CDFs. \(\square\)

Worked Example: A Correct Heteroscedastic Normal Forecast

Let \(X\) take values 0 and 1, and suppose \(Y\mid X=0\sim N(0,1)\) and \(Y\mid X=1\sim N(2,4)\). The second argument of \(N\) is the variance, so the conditional standard deviations are 1 and 2. Consider the forecast that uses these same two conditional distributions.

For \(X=0\), the forecast CDF at \(y\) is \(G(y\mid 0)=\Phi(y)\). For \(X=1\), it is \(G(y\mid 1)=\Phi((y-2)/2)\), where \(\Phi\) is the standard normal CDF. Since the forecast matches the stated conditional law in both groups, the theorem says the PIT is uniform conditional on either value of \(X\).

For an observed outcome \(Y=4\) when \(X=1\), the PIT is \(G(4\mid 1)=\Phi((4-2)/2)=\Phi(1)\approx0.8413\). This is one realized PIT value, not a test on its own. Across repeated observations with \(X=1\), the PIT values are uniform on \((0,1)\) if the forecast is correct. The different scales in the two groups matter: using \(\Phi(Y)\) for both groups would not be the specified forecast and would generally produce a nonuniform PIT for \(X=1\).

Randomized PIT for Discrete Outcomes

For a discrete outcome, the ordinary transform \(G(Y\mid X)\) has jumps and is not generally uniform even when the forecast is exactly right. The randomized PIT fills each jump with an independent uniform draw. Conditional on \(X=x\) and \(Y=y\), it is uniform over the interval from the left limit \(G(y-\mid x)\) to \(G(y\mid x)\).

Definition: Let \(V\sim\mathrm{Uniform}(0,1)\) be independent of \((X,Y)\). The randomized PIT is \(U=G(Y-\mid X)+V\{G(Y\mid X)-G(Y-\mid X)\}\). Here \(G(y-\mid x)=\lim_{z\uparrow y}G(z\mid x)\). The interval length is the forecast probability mass at the observed outcome; it is not the CDF value itself.

Worked Example: Checking a Two-Point Forecast

At a fixed covariate value \(x\), suppose the forecast puts probability \(q_1=0.5\) on \(y_1=0\) and \(q_2=0.5\) on \(y_2=1\). The forecast cumulative probabilities are \(c_0=0\), \(c_1=0.5\), and \(c_2=1\). The observed conditional probabilities are instead \(p_1=0.7\) and \(p_2=0.3\).

If \(Y=0\), the randomized PIT is \(U=0+0.5V\), uniform on \([0,0.5]\). If \(Y=1\), it is \(U=0.5+0.5V\), uniform on \([0.5,1]\). The first outcome occurs with probability 0.7 and the second with probability 0.3. For \(0<u<0.5\), the density of \(U\) is therefore \(0.7/0.5=1.4\); for \(0.5<u<1\), it is \(0.3/0.5=0.6\). These densities integrate to \(1.4(0.5)+0.6(0.5)=1\), but they are not the constant density 1 required for a uniform distribution.

For example, \(P(U\le0.25)=0.7(0.25/0.5)=0.35\), whereas a uniform random variable would have probability 0.25. Thus the randomized PIT detects the forecast's excessive probability on outcome 0.

Theorem: At a fixed \(x\), suppose the forecast has finite support \(y_1<\cdots<y_m\), with positive forecast masses \(q_j=G(y_j\mid x)-G(y_j-\mid x)>0\). Let \(c_0=0\) and \(c_j=\sum_{k=1}^{j}q_k\), so \(c_m=1\). If the true conditional probabilities on this same support are \(p_1,\ldots,p_m\), then the randomized PIT is uniform on \((0,1)\) if and only if \(p_j=q_j\) for every \(j\).

Proof. When \(Y=y_j\), the transform is \(U=c_{j-1}+Vq_j\), which is uniform on the interval \([c_{j-1},c_j]\), an interval of length \(q_j\). The event \(Y=y_j\) has probability \(p_j\). Consequently, the density of the mixture on the interior of that interval is \(p_j/q_j\). The intervals partition \([0,1]\) up to endpoints, which have probability zero. If \(p_j=q_j\) for all \(j\), each interval has density 1 and the full density is 1. Conversely, if the full distribution is uniform, its density on each interval must be 1 almost everywhere, so \(p_j/q_j=1\), hence \(p_j=q_j\). \(\square\)

In the theorem, the support is common to the forecast and the true conditional distribution, and every forecast mass is positive. If the true law assigns mass outside the forecast's support, the stated finite-support calculation does not apply; that discrepancy itself is a warning that the forecast support is inadequate. For a general discrete forecast, the same randomized construction uses the jump at the realized outcome, not the CDF value \(G(y\mid x)\) alone.

Marginal Uniformity Can Hide Conditional Failure

A pooled PIT histogram examines the distribution of \(U\) after mixing across covariates. Conditional calibration implies marginal uniformity, but a marginally uniform PIT does not imply conditional calibration. One group can have too many small PIT values while another has too few, with the errors canceling exactly in the pooled distribution.

Worked Example: A Uniform Pooled PIT From Two Miscalibrated Groups

Let \(X\) be 0 or 1 with probability one-half each. Forecast \(Y\mid X\) as Uniform\((0,1)\) in both groups, so \(G(y\mid X)=y\) for \(0\le y\le1\). Suppose the actual conditional PIT \(U=G(Y\mid X)=Y\) has density 2 on \([0,0.5]\) and 0 on \((0.5,1]\) when \(X=0\). When \(X=1\), let it have density 0 on \([0,0.5)\) and 2 on \([0.5,1]\). Each conditional density integrates to 1.

For any \(u\in[0,0.5]\), the pooled density is \(0.5(2)+0.5(0)=1\). For any \(u\in(0.5,1]\), it is \(0.5(0)+0.5(2)=1\). Thus the pooled PIT is exactly uniform. But at \(u=0.5\), \(P(U\le0.5\mid X=0)=1\) and \(P(U\le0.5\mid X=1)=0\), instead of 0.5 in either group. The forecast is badly miscalibrated conditionally despite passing the pooled uniformity check.

Covariate-Stratified Reliability Checks

A direct diagnostic is to compare the empirical fraction of PIT values below a threshold \(u\) with \(u\), separately within covariate strata. If \(B\) is a set of covariate values and the evaluation observations are independent draws from the target population, define \(\widehat R_B(u)=n_B^{-1}\sum_{i:X_i\in B}\mathbf{1}\{U_i\le u\}\), where \(n_B\) is the number of observations in the stratum. This estimates \(P(U\le u\mid X\in B)\). Under conditional calibration, that probability equals \(u\) for every stratum \(B\) and every \(u\in[0,1]\).

Result: If \(U\mid X\) is Uniform\((0,1)\) almost surely, then for every measurable covariate stratum \(B\) with \(P(X\in B)>0\), \(P(U\le u\mid X\in B)=u\) for every \(u\in[0,1]\).

Proof. Conditional uniformity gives \(P(U\le u\mid X)=u\) almost surely. Taking conditional expectation given the event \(X\in B\) yields \(P(U\le u\mid X\in B)=E[\mathbf{1}\{U\le u\}\mid X\in B]=E[P(U\le u\mid X)\mid X\in B]=E[u\mid X\in B]=u\). The stratum has positive probability, so this conditional expectation is defined. \(\square\)

In practice, choose strata that reflect meaningful covariate structure: categories for discrete predictors, or prespecified intervals for continuous predictors. A reliability plot displays \(\widehat R_B(u)\) against \(u\); the diagonal is the calibration target. PIT histograms within strata are another useful view. With discrete outcomes, use randomized PIT values and account for the added randomization when interpreting plots.

A finite collection of strata and thresholds is only a diagnostic, not a proof of conditional calibration. Miscalibration can vary within a bin, and small strata give noisy estimates. To make comparisons meaningful, assess forecasts on observations not used to fit them, or use an appropriate sample-splitting or cross-fitting procedure. These steps limit the danger that in-sample fit is mistaken for reliable performance on new observations.

Common Pitfalls and Interpretation

Treating marginal uniformity as conditional calibration. A pooled PIT histogram can look uniform because opposing errors cancel across covariate groups. Examine PIT behavior within relevant strata as well.
Using the CDF value as a discrete probability mass. The mass at \(y\) is \(G(y\mid x)-G(y-\mid x)\). Using \(G(y\mid x)\) as the interval length in a randomized PIT gives incorrect intervals.
Expecting the ordinary discrete PIT to be uniform. Atoms create jumps. Randomization across the forecast's jump interval is what produces uniformity when the forecast is correct.
Reading a noisy plot as a definitive test. A small sample can produce visible departures by chance, while broad strata can hide local failures. Treat plots as diagnostics and interpret them with sample size and stratum choice in mind.

Calibration diagnostics are valuable because they identify where a conditional distribution forecast is systematically too concentrated, too dispersed, or otherwise misaligned with outcomes. The PIT converts a full distributional forecast into a simple uniformity target, while covariate stratification checks whether that target holds across the conditions on which the forecast depends. Neither a good pooled histogram nor a few reassuring strata establish correctness everywhere; they are targeted checks of a population property.

Check Your Understanding

Answer each question by referring to the definitions and calculations above.

Under what regularity condition does uniformity of the continuous PIT imply equality between the forecast and true conditional CDFs?
A forecast has probability 0.4 at \(y=0\). What is the length of the randomized-PIT interval when \(Y=0\), and how is that length obtained from the CDF?
Why can a pooled PIT be uniform even when the forecast is miscalibrated at each of two covariate values?
What conditional probability does the empirical fraction \(\widehat R_B(u)\) estimate?
Why does checking calibration in a finite number of covariate strata not establish conditional calibration everywhere?