From a Conditional Estimand to an Error Measure
A conditional distribution estimate is useful only when its target and its assessment agree. If the outcome is binary, estimating a success probability may be enough to specify the entire law. For a continuous outcome, a conditional mean alone does not describe spread or multiple modes. A point-prediction error can therefore look small while the estimated distribution is poor, or vice versa.
Let \(X\) be a covariate and \(Y\) an outcome. At a covariate value \(x\), the target is the conditional law of \(Y\) given \(X=x\). In applications with continuous \(X\), this conditional law is defined up to sets of covariate values having probability zero; estimation at a particular \(x\) requires suitable support and regularity for the method being used. The examples below take a fixed covariate value or stratum and focus on choosing an estimand, estimator, and assessment appropriate to its outcome structure.
A scoring rule evaluates a forecast distribution using an observed outcome; its expected value under the true law describes its population performance. An error metric instead compares an estimated object with a target object, when that target is available or can be specified for analysis. These purposes are related but not interchangeable: a score is computed from forecasts and outcomes, while a distributional distance directly compares laws.
Worked Example: A Bernoulli Conditional Law
Worked Example: A Bernoulli Conditional Law
Suppose that in a fixed treatment-and-age stratum, \(Y\) indicates whether a patient responds to treatment. The estimand is \(p_x=P(Y=1\mid X=x)\), and the conditional law is Bernoulli with success probability \(p_x\). Among \(n=20\) patients in this stratum, 15 respond and 5 do not. Under an independent Bernoulli model with a common stratum probability, the likelihood is \(L(p_x)=p_x^{15}(1-p_x)^5\). Maximizing it gives \(\hat p_x=15/20=0.75\).
For an individual binary forecast \(q\), the logarithmic loss is \(-\log q\) when \(Y=1\), and \(-\log(1-q)\) when \(Y=0\). The Brier loss is \((Y-q)^2\). To compare forecasts \(q=0.70\) and \(q=0.75\) on a held-out set containing 3 responses and 1 nonresponse, calculate the average logarithmic loss at \(q=0.70\):
Thus the average loss is \(0.5685\) to four decimal places. At \(q=0.75\), it is
The smaller observed log loss favors \(q=0.75\) on this held-out set. The average Brier losses are \(\{3(0.30)^2+1(0.70)^2\}/4=0.1900\) for \(q=0.70\), and \(\{3(0.25)^2+1(0.75)^2\}/4=0.1875\) for \(q=0.75\); they favor the same forecast here. This comparison is a finite-sample result for these four outcomes, not a claim that either score must rank forecasts identically on every sample.
Why Proper Scores Target the Conditional Law
For a binary outcome, a forecast probability \(q\) can be assessed without first estimating a separate outcome model. If the true conditional success probability is \(p\), the expected Brier loss and expected logarithmic loss are minimized by forecasting \(p\). The next result gives the derivations and states the conditions for the logarithmic comparison.
Derivation. Since \(Y\) is 1 with probability \(p\) and 0 with probability \(1-p\),
The squared term is nonnegative and vanishes exactly at \(q=p\). For the logarithmic score, the expected loss at \(q\) is \(-p\log q-(1-p)\log(1-q)\). Subtracting its value at \(p\) gives the displayed expression. This expression is the Kullback–Leibler divergence between the Bernoulli laws with probabilities \(p\) and \(q\); its nonnegativity follows from the general density result below, or directly from \(\log t\leq t-1\). Equality requires the two Bernoulli laws to agree, hence \(q=p\). \(\square\)
More generally, let \(P\) and \(Q\) have densities \(p\) and \(q\) with respect to the same measure. If both expected negative logarithmic losses under \(P\), \(E_P[-\log p(Y)]\) and \(E_P[-\log q(Y)]\), are finite, then
Here finiteness ensures the difference is defined. Nonnegativity is Gibbs’ inequality: on the set where \(p>0\), \(q>0\) under the stated finite-loss condition, and \(\log(q/p)\leq q/p-1\). Integrating with respect to \(P\) gives
The final inequality holds because \(q\) integrates to one over the whole space. This proves the result. \(\square\)
Worked Example: A Gaussian Conditional Law
Worked Example: A Gaussian Conditional Law
At a fixed covariate value \(x\), suppose a working model treats the outcomes as independent draws from \(N(\mu_x,\sigma_x^2)\). The estimand is the pair \((\mu_x,\sigma_x^2)\), or equivalently the full Gaussian conditional law. Five observations in the stratum are \(1,2,3,4,5\). The likelihood maximizers are the sample mean and the variance with denominator \(n\):
The fitted conditional law is \(N(3,2)\), where 2 is the variance. For a held-out outcome \(y=4\), its negative log density, also called the continuous logarithmic score, is
Density scores can be negative for densities greater than one; that does not make them invalid. If the target law for comparison were \(N(2.8,1.5)\), the fitted mean error would be \(3-2.8=0.2\), and the variance error would be \(2-1.5=0.5\). Those parameter errors describe different features and use different scales. For a full-law comparison, the 2-Wasserstein distance between these one-dimensional Gaussian laws is
The example distinguishes fitting from assessment: the maximum-likelihood variance uses denominator \(n\), whereas an unbiased sample-variance estimate uses denominator \(n-1\). The appropriate choice depends on the estimation procedure and target; they are not interchangeable formulas.
Worked Example: A Multimodal Conditional Law
Worked Example: A Multimodal Conditional Law
Suppose a conditional outcome has two separated modes, represented by the mixture \(f_x(y)=w_x g_L(y)+(1-w_x)g_R(y)\), where \(g_L\) is uniform on \([-2.2,-1.8]\), and \(g_R\) is uniform on \([1.8,2.2]\). The component densities are known; the target is the mixing weight \(w_x\), which also determines the full law. In 10 observations, 7 fall in the left interval and 3 in the right. Each observation contributes a factor \(w_x/0.4\) if it falls left, and \((1-w_x)/0.4\) if it falls right. Therefore the likelihood is
The log-likelihood, apart from its constant term, is \(7\log w_x+3\log(1-w_x)\). Its derivative is \(7/w_x-3/(1-w_x)\), which vanishes at \(10w_x=7\), so \(\hat w_x=0.7\). Its second derivative is \(-7/w_x^2-3/(1-w_x)^2<0\) for \(0<w_x<1\), confirming a maximum.
At \(y=-2.1\), the fitted density is \(0.7/0.4=1.75\), so the negative log density is \(-\log(1.75)\approx-0.5596\). At \(y=2.0\), the density is \(0.3/0.4=0.75\), and the score is \(-\log(0.75)\approx0.2877\). For a density score, a negative value is possible and is not an error.
If the true mixing weight is \(0.6\), the estimated and true CDFs differ by \(0.1\) at zero: the CDF there equals the left-mode weight. The 1-Wasserstein distance is \(4|0.7-0.6|=0.4\). To see this, couple the common mass in each component identically; the remaining mass \(0.1\) must move between the two component intervals. Its least possible mean travel distance is the difference between their means, \(2-(-2)=4\), giving cost \(0.1(4)=0.4\). This metric captures the separation of the modes, whereas the weight error alone is \(0.1\).
Choosing What to Estimate and How to Assess It
The examples illustrate a practical sequence. First specify whether the target is a scalar feature, such as a conditional mean, or the whole law. Then select a model and estimator that represent the outcome structure. Finally, choose an assessment matched to the target: a proper score for observed outcomes, a parameter error for a parameter target, or a distributional metric for a law target.
| Outcome structure | Natural estimand | Possible assessment |
|---|---|---|
| Binary, Bernoulli | Conditional probability \(p_x\) | Logarithmic or Brier score; probability error if the target is known |
| Approximately Gaussian | Conditional mean and variance, or full Gaussian law | Continuous logarithmic score; parameter errors or Wasserstein distance |
| Separated modes or mixture | Mixing weights and component laws, or full mixture law | Held-out log score; CDF or Wasserstein error when the target law is available |
A common mistake is to assess a full conditional distribution solely by the error in its mean. For example, two distributions can have the same mean while one is concentrated near that mean and the other has two distant modes. A mean-only error is then zero even though the laws differ. The correction is to define the estimand before choosing the metric: use mean error when the mean is the target, but use a distributional score or distance when the full conditional law is the target.
Check Your Understanding
Work through these questions by identifying the target before choosing a calculation.
- In a Bernoulli stratum with 12 successes among 30 observations, what is the maximum-likelihood estimate of the success probability?
- Why does the expected Brier loss favor the true Bernoulli probability even when that probability is not known to the forecaster?
- For Gaussian outcomes, why should the variance with denominator \(n\) not automatically be described as an unbiased variance estimate?
- In the two-component mixture example, what does a mixing-weight error omit that the Wasserstein distance can reflect?
- Give an example of two conditional laws with the same mean but different distributional shape. Why would mean error fail to distinguish them?