Log Score, Conditional Likelihood, and KL Regret

Why Log-Score Regret Has a Distributional Meaning

A conditional distribution model assigns a distribution to the response $Y$ for each covariate value $X=x$. The log score evaluates the model by the log density it assigns to the observed response. Its appeal is not only computational: under suitable conditions, the increase in expected log-score loss from using a candidate distribution rather than the truth is exactly a conditional Kullback–Leibler (KL) divergence. This identity gives the log score a precise population interpretation and explains its connection to conditional likelihood.

The support condition matters. If the true conditional distribution puts positive probability where a candidate assigns density zero, the candidate receives infinite expected log loss there. Integrability matters too: an identity subtracting two risks is not meaningful if both risks are infinite. The statements below make these qualifications explicit.

Conditional Risk and Conditional KL Divergence

Let $(X,Y)$ have joint distribution $P$. Assume the covariate and response take values in standard Borel spaces, so regular conditional distributions can be used. Write $P_x$ for the conditional law of $Y$ given $X=x$. Suppose the true conditional laws and candidate conditional laws $Q_x$ have densities $p(y\mid x)$ and $q(y\mid x)$, respectively, with respect to the same sigma-finite measure $\nu$. The densities are assumed jointly measurable in $(x,y)$. Densities that differ only on a $\nu$-null set represent the same conditional law.

Definition: The conditional log-score loss of a candidate density $q$ at $(x,y)$ is $-\log q(y\mid x)$, using the natural logarithm and setting $-\log 0=+\infty$. Its risk is $R(q)=E[-\log q(Y\mid X)]$, when this expectation is defined. The conditional KL divergence is $$ D_{\mathrm{KL}}(P_x\|Q_x)=\int p(y\mid x)\log\!\left(\frac{p(y\mid x)}{q(y\mid x)}\right)\,d\nu(y) $$ if $P_x$ is absolutely continuous with respect to $Q_x$, and $+\infty$ otherwise. The expected conditional KL divergence is $E_X[D_{\mathrm{KL}}(P_X\|Q_X)]$.

Absolute continuity, written $P_x\ll Q_x$, means that every set assigned probability zero by $Q_x$ is also assigned probability zero by $P_x$. With densities, this is equivalent to $q(y\mid x)>0$ for $P_x$-almost every $y$. If this fails, the true distribution sometimes produces an outcome to which the candidate assigns zero density, so the candidate's log loss is infinite on that event.

Theorem: Suppose $E[|\log p(Y\mid X)|]<\infty$. If $P_x\ll Q_x$ for $P_X$-almost every $x$, and $E[|\log q(Y\mid X)|]<\infty$, then $$ R(q)-R(p)=E_X\!\left[D_{\mathrm{KL}}(P_X\|Q_X)\right]. $$ The right side is nonnegative, and it is zero exactly when $P_x=Q_x$ for $P_X$-almost every $x$. If conditional absolute continuity fails on a set of covariate values with positive $P_X$-probability, then $R(q)=+\infty$, given the assumed finite true risk. More generally, when the true risk is finite and candidate risk is allowed to be $+\infty$, the identity holds in the extended sense: failure of absolute continuity or infinite candidate risk gives infinite regret and infinite expected conditional KL divergence.

The finite-risk assumptions in the first statement ensure that the subtraction $R(q)-R(p)$ is legitimate and that the log-ratio can be integrated by subtracting the two log terms. The extended statement permits infinite candidate risk but retains a finite true risk; it does not license subtracting two infinite risks.

Deriving the Regret Identity

Under the finite-risk assumptions, $q(Y\mid X)>0$ almost surely, so the log ratio is defined at the observed pair. Pointwise, $-\log q(Y\mid X)+\log p(Y\mid X)=\log\{p(Y\mid X)/q(Y\mid X)\}$. Taking expectations and conditioning on $X$ gives

\begin{aligned} R(q)-R(p) &=E\!\left[\log\!\left\{\frac{p(Y\mid X)}{q(Y\mid X)}\right\}\right]\\ &=E_X\!\left[ \int p(y\mid X)\log\!\left\{\frac{p(y\mid X)}{q(y\mid X)}\right\}\,d\nu(y) \right]\\ &=E_X[D_{\mathrm{KL}}(P_X\|Q_X)]. \end{aligned}

The conditional expectation step is valid because the log terms are absolutely integrable by assumption. To see why KL divergence is nonnegative, fix $x$ with $P_x\ll Q_x$, and let $r=dP_x/dQ_x$. The convex function $\phi(t)=t\log t$, with $\phi(0)=0$, satisfies Jensen's inequality under $Q_x$:

D_{\mathrm{KL}}(P_x\|Q_x) =\int r\log r\,dQ_x \geq \left(\int r\,dQ_x\right)\log\!\left(\int r\,dQ_x\right) =1\log 1=0.

Here $\int r\,dQ_x=1$, and Jensen's inequality remains valid if the divergence is infinite because $t\log t$ is bounded below. Equality in the strict convexity inequality requires $r=1$ $Q_x$-almost surely, which means $P_x=Q_x$. If absolute continuity fails, the divergence is defined as infinity and cannot be zero. Averaging over $X$ proves the stated nonnegativity and equality condition for expected conditional KL divergence. $\square$

Worked Examples

Worked Example: Gaussian Conditional Distributions

Suppose $X$ takes values $0$ and $1$, each with probability $1/2$. Given $X=x$, let the true distribution be $Y\mid X=x\sim N(\mu(x),4)$, and let a candidate use $N(m(x),4)$. Set $m(0)-\mu(0)=1$ and $m(1)-\mu(1)=-2$. Find the excess conditional log-score risk.

For a normal density with variance $4$, the terms involving the log normalizing constant cancel in the log-density ratio. Thus, conditional on $X=x$,

\log\!\left\{\frac{p(Y\mid x)}{q(Y\mid x)}\right\} =\frac{(Y-m(x))^2-(Y-\mu(x))^2}{8}.

Under the true distribution, $E[(Y-\mu(x))^2\mid X=x]=4$, while $E[(Y-m(x))^2\mid X=x]=4+(\mu(x)-m(x))^2$. Therefore the conditional KL divergence is $(m(x)-\mu(x))^2/8$. It is $1/8$ at $x=0$ and $4/8$ at $x=1$. Averaging over the two covariate values gives

E_X[D_{\mathrm{KL}}(P_X\|Q_X)] =\frac12\left(\frac18+\frac48\right) =\frac{5}{16}=0.3125.

The candidate's expected log-score loss exceeds the true conditional model's risk by $0.3125$. This calculation also shows that the expected regret weights conditional discrepancies according to the distribution of $X$.

Worked Example: Bernoulli Conditional Probabilities

Suppose $P(X=0)=P(X=1)=1/2$. For a binary response, the true success probabilities are $p(1\mid 0)=0.8$ and $p(1\mid 1)=0.3$. A candidate uses $q(1\mid 0)=0.6$ and $q(1\mid 1)=0.3$. Compute its excess risk.

At $X=1$, candidate and truth agree, so the KL divergence is zero. At $X=0$, the divergence between Bernoulli distributions is

\begin{aligned} D_{\mathrm{KL}}(P_0\|Q_0) &=0.8\log\!\left(\frac{0.8}{0.6}\right) +0.2\log\!\left(\frac{0.2}{0.4}\right)\\ &=0.8\log(4/3)+0.2\log(1/2)\\ &\approx 0.8(0.287682)+0.2(-0.693147)\\ &\approx 0.230146-0.138629=0.091517. \end{aligned}

The expected conditional divergence, and hence the excess risk, is $0.5(0.091517)+0.5(0)\approx 0.045758$. The errors in predicted success probability contribute differently at each covariate value; the contribution from $X=1$ is zero because the distributions there match exactly.

Worked Example: A Candidate That Assigns Zero Probability

For a particular covariate value $x$, let $Y$ take values $A$ and $B$, with true probabilities $P_x(A)=0.7$ and $P_x(B)=0.3$. Suppose the candidate assigns $Q_x(A)=1$ and $Q_x(B)=0$. The true response is $B$ with probability $0.3$, and on that event the candidate loss is $-\log 0=+\infty$. Consequently, its conditional risk is infinite.

The absolute-continuity condition fails because $P_x(\{B\})=0.3>0$, while $Q_x(\{B\})=0$. By definition, $D_{\mathrm{KL}}(P_x\|Q_x)=+\infty$ as well. There is no finite regret calculation to make: this candidate is not merely inaccurate but assigns zero probability to a possible true outcome.

Conditional Likelihood Estimation

Suppose a parametric conditional model has densities $q_\theta(y\mid x)$, indexed by a parameter $\theta$. For independent observations $(X_i,Y_i)$, $i=1,\ldots,n$, the conditional log-likelihood and its negative average are

\ell_n(\theta)=\sum_{i=1}^n\log q_\theta(Y_i\mid X_i), \qquad -\frac{1}{n}\ell_n(\theta)=\frac{1}{n}\sum_{i=1}^n-\log q_\theta(Y_i\mid X_i).

Thus maximizing conditional likelihood is exactly minimizing the empirical average conditional log loss. At the population level, define $R(\theta)=E[-\log q_\theta(Y\mid X)]$. Whenever the theorem's finite-risk conditions hold,

R(\theta)-R(p)=E_X[D_{\mathrm{KL}}(P_X\|Q_{\theta,X})],

where $Q_{\theta,x}$ is the conditional distribution with density $q_\theta(\,\cdot\mid x)$. The true risk $R(p)$ does not depend on $\theta$. Consequently, a parameter minimizing population conditional log-score risk also minimizes the expected conditional KL divergence. If the model contains the truth, any parameter representing $P_x$ for $P_X$-almost every $x$ is a population minimizer. If it does not, population likelihood selects a model distribution with the smallest covariate-averaged KL divergence among the available candidates.

Worked Example: Conditional Bernoulli Likelihood

Consider a model with a single Bernoulli success probability $\theta$, where $0<\theta<1$, and observed binary responses $Y_1,\ldots,Y_n$. The conditional log-likelihood is $\ell_n(\theta)=\sum_{i=1}^n\{Y_i\log\theta+(1-Y_i)\log(1-\theta)\}$. For a sample with three successes and two failures,

\ell_5(\theta)=3\log\theta+2\log(1-\theta), \qquad -\frac{\ell_5(\theta)}5 =-\frac35\log\theta-\frac25\log(1-\theta).

The likelihood is maximized at $\hat\theta=3/5$: differentiating gives $\ell_5'(\theta)=3/\theta-2/(1-\theta)$, which is zero when $3(1-\theta)=2\theta$, or $\theta=3/5$. The second derivative is $-3/\theta^2-2/(1-\theta)^2<0$, confirming a maximum. This is the parameter value that minimizes the sample average log loss. The calculation is an empirical criterion; identifying its population target uses the risk–KL identity, not the finite-sample calculation alone.

Conditional likelihood treats the observed covariates as given for the purpose of modeling $Y\mid X$. If a joint likelihood for $(X,Y)$ is used and the marginal law of $X$ has no parameter in common with $\theta$, its marginal-likelihood term does not change which $\theta$ maximizes the likelihood. The connection above concerns the conditional distribution model; it does not assert that a finite-sample likelihood maximizer automatically equals the population minimizer.

Common Pitfall and Practical Meaning

A frequent error is to write $R(q)-R(p)=E_X[D_{\mathrm{KL}}(P_X\|Q_X)]$ without checking support or whether the risks are finite. If $q$ vanishes where $p$ is positive, both the conditional KL divergence and the candidate log loss are infinite. If both risks are infinite, subtracting them does not define a regret. Check absolute continuity and integrability before using the identity.

A second error is to interpret equality of conditional distributions pointwise for every possible $x$. Population risk averages over $P_X$, so it can only distinguish conditional laws on covariate values that occur with positive probability, up to $P_X$-null sets. The result is useful both as a theoretical target for conditional likelihood and as a reminder that the covariate distribution determines which conditional discrepancies matter most.

Check Your Understanding

Use the definitions and conditions above to answer these questions.

What does $P_x\ll Q_x$ mean, and why does its failure make the candidate's conditional log loss infinite?
Under what finite-risk conditions can excess conditional log-score risk be written as expected conditional KL divergence?
If two conditional models differ only at covariate values in a $P_X$-null set, can their population risks differ? Explain.
In the Gaussian example, why does the conditional KL divergence depend on the squared difference between the means?
What does conditional likelihood maximize in a finite sample, and what population quantity does its expected criterion minimize?