Predictive Scoring Rules For GLMs

Why Score Predictions?

A fitted GLM can produce more than a predicted mean. For a binary response it may give a predicted event probability; for a count response it may specify a complete predictive distribution, such as a Poisson distribution. Scoring rules turn predictions and subsequently observed outcomes into numerical assessments. With the convention used here, smaller scores are better.

The distinction between predicting a mean and predicting a distribution matters. A squared-error score can assess how close predicted means are to observed responses, but it cannot determine whether the model assigned sensible probabilities to different possible outcomes. A log score or a ranked probability score uses more of the predictive distribution. A score is a summary of predictive performance for specified outcomes; it does not, by itself, establish that a model is well calibrated or explain why one model scored better.

Log Scores for Binary and Count Outcomes

For a discrete response with predictive probability mass function \(q(y)\), the negative log score, also called the log loss, is

It rewards assigning substantial probability to the outcome that occurs and heavily penalizes assigning it a very small probability. If \(q(y)=0\), the loss is infinite. For a binomial observation \(Y=y\) from \(m\) trials with predicted success probability \(p\), the full predictive probability is

The combinatorial factor belongs in the probability of the observed count. For a Poisson prediction with mean \(\lambda\), the corresponding probability is \(q(y)=e^{-\lambda}\lambda^y/y!\). The log score evaluates these probabilities, not merely the predicted means \(mp\) or \(\lambda\).

Scores for Predicted Means and Binary Risks

For a binary response \(Y\in\{0,1\}\), let \(p\) be the predicted event probability. The Brier score is the squared error between the probability prediction and the observed outcome:

It is bounded between 0 and 1 and evaluates the probability prediction, which is also the predicted mean of a Bernoulli response. For grouped binomial data, one possible Brier score per trial is the squared difference between the observed event fraction and predicted probability, \((y/m-p)^2\). This is not the same quantity as the binomial log score: the latter uses the probability of the whole observed count.

For a count outcome, squared error can be applied directly to a predicted mean \(a\): \(S(a,y)=(a-y)^2\). This assesses the mean, not the distribution's shape. To compare count distributions rather than just their means, a useful alternative is the ranked probability score (RPS). For a predictive count distribution \(F\), define

where \(F(k)=P(Y\leq k)\), and \(\mathbf{1}(y\leq k)\) is 1 when \(y\leq k\) and 0 otherwise. Each term compares the predicted cumulative probability at a threshold with the corresponding indicator for the observed count. Thus the RPS assesses the predictive distribution across thresholds, rather than only its mean.

Why These Scores Target Different Things

Proof. Since \(Y=1\) with probability \(q\) and \(Y=0\) with probability \(1-q\),

The term \(q(1-q)\) does not depend on the reported \(p\), and the remaining squared term is nonnegative, equaling zero only at \(p=q\). This proves strict propriety for the binary probability. For a grouped binomial observation with \(m\) independent trials and score \((Y/m-p)^2\), the corresponding expectation is \(q(1-q)/m+(p-q)^2\), so it too targets the event probability. \(\square\)

Proof. Write \(Y-a=(Y-\mu)+(\mu-a)\), square, and take expectations:

because \(E[Y-\mu]=0\). The variance is fixed under the true distribution, so the expression is minimized when \(a=\mu\). Any two predictive distributions with the same mean receive the same reported prediction \(a=\mu\); their different variances and tail probabilities are not distinguished by this mean-only forecast. \(\square\)

The RPS gives a distributional alternative for counts. If the true count distribution has CDF \(G\) and finite mean, and the forecast has CDF \(F\) and finite mean, then the expected RPS is the sum of a term independent of \(F\) and \(\sum_{k\geq0}(F(k)-G(k))^2\). Indeed, for each \(k\), the indicator \(\mathbf{1}(Y\leq k)\) has success probability \(G(k)\), and the Brier-score identity just proved gives

Summing these nonnegative expected terms is valid by monotone convergence. The finite-mean condition ensures the RPS expectation is finite: for large \(k\), the relevant CDF tail errors are bounded by the corresponding upper-tail probabilities, whose sum is finite for a count with finite mean. The expected score is minimized when \(F(k)=G(k)\) for every \(k\), which determines the count distribution uniquely. Thus, unlike squared error for a mean, RPS can distinguish predictive count distributions with equal means.

Log-Score Propriety and Its Conditions

A log score is a strictly proper score for a full distribution when the expected losses being compared are finite. The qualification matters, particularly on an infinite outcome space such as the nonnegative integers. Without finiteness conditions, expected log losses can both be infinite, so subtracting them to claim a strict improvement is not justified.

where \(D_{\mathrm{KL}}(P\|Q)=\sum_{y:p(y)>0}p(y)\log\{p(y)/q(y)\}\), with value \(+\infty\) if \(q(y)=0\) at any point with \(p(y)>0\). Under the stated finiteness conditions, equality holds if and only if \(P=Q\).

Proof. The finite expected log losses allow their difference to be taken, giving

If \(q(y)=0\) for some \(y\) with \(p(y)>0\), the cross-entropy is infinite, contrary to the assumptions. Otherwise set \(R(y)=q(y)/p(y)\) on the support of \(P\). Then \(E_P[R]=\sum_{p(y)>0}q(y)\leq1\). The assumed finiteness makes \(\log R\) integrable under \(P\), so Jensen's inequality for the concave logarithm gives

Therefore \(D_{\mathrm{KL}}(P\|Q)=-E_P[\log R]\geq0\). Equality requires equality in Jensen's inequality, so \(R\) is constant \(P\)-almost surely, and also \(E_P[R]=1\). These conditions force \(R=1\) on the support of \(P\) and leave no probability mass for \(Q\) outside that support; hence \(P=Q\). Conversely, if \(P=Q\), the divergence is zero. \(\square\)

This result explains why log score is useful for comparing complete predictive distributions, including Poisson and negative-binomial predictions. It also explains why one should not assert strict propriety from a formal difference of expected log losses when those expectations may be infinite. For standard finite-parameter GLM predictive distributions, the relevant expected log losses are ordinarily finite under the same model families; when using broader distribution classes, the finiteness assumptions should be checked.

Using Scores Responsibly

Scores are most informative when their target and scale are explicit. Log loss and RPS evaluate full predictive distributions; Brier score evaluates binary probabilities; squared error for a count prediction evaluates the mean. These scores answer related, but not interchangeable, questions.

• Use the same observed outcomes. A score difference is interpretable as a comparison only when forecasts are evaluated on the same responses.
• Keep the score definition fixed. A grouped binomial log score is a loss for a count probability; a per-trial Brier score is a loss for an event probability. Their values should not be compared numerically as if they were on the same scale.
• Include the full probability when using log loss. For a binomial count, score the probability of the observed count, including its combinatorial factor.
• State what is being forecast. Two count distributions can share a mean yet differ in dispersion and tail probabilities. A mean-only score cannot reveal those differences.

A common mistake is to conclude that two models with the same predicted mean make equivalent predictions. For squared error on that mean, they may indeed receive the same score. But a Poisson distribution and a negative-binomial distribution with the same mean can assign different probabilities to the realized count, and therefore receive different log scores or RPS values. The correction is to choose a score that matches the predictive object of interest, then interpret the result as evidence about performance under that score—not as a universal ranking of the models.