Choice Prediction And Probability Calibration

From Choice Probabilities to Predictions

A multinomial logit model does not merely label a choice as “most likely.” It assigns a probability to every available alternative. Those probabilities can support different prediction tasks: selecting the most likely category, estimating how often a category will be chosen in a population, or assigning a score to the quality of a full probability forecast. These tasks are related, but they do not measure the same thing.

Suppose each observation \(i\) faces the same set of \(J\) mutually exclusive and exhaustive alternatives. The fitted multinomial logit model gives probabilities \(\hat p_{ij}\), where \(j\) indexes alternatives. For the standard multinomial logit specification, these probabilities are obtained from the alternatives’ systematic utilities \(V_{ij}\). For a new observation, the covariates and choice set must be available, and the fitted model must produce a valid probability vector: each probability is between zero and one and the probabilities sum to one.

A predicted category is commonly the alternative with the largest predicted probability. This rule is appropriate when every incorrect classification has the same cost. If mistakes have different costs, the best decision instead depends on those costs; the largest-probability category need not minimize expected loss.

Prediction quality should ordinarily be evaluated on observations not used to fit or select the model, with the same outcome definitions and a choice set appropriate to the prediction task. This is held-out evaluation. Performance measured on the fitting data can look overly favorable because the model was estimated using those outcomes. Held-out results describe performance for the population represented by the evaluation data; they do not guarantee performance under a different population or choice environment.

Category Predictions and Classification Accuracy

Classification accuracy is the proportion of observations for which the predicted category equals the observed choice. It is easy to interpret, but it uses only the largest-probability category and ignores how confident the model was. A forecast assigning probabilities \(0.51,0.49\) and one assigning \(0.99,0.01\) produce the same predicted category, even though they express very different uncertainty.

Here \(y_i\) is the observed category and \(I(\cdot)\) equals one when its argument is true and zero otherwise. A confusion table can further show which observed categories are predicted as which categories. Accuracy alone can be misleading when categories are unbalanced: a model that always predicts a very common alternative may achieve high accuracy while failing to identify less common choices. It is therefore useful to report category-specific performance or a probability-based score as well.

Calibration: Do the Probabilities Match Frequencies?

A model is calibrated when its predicted probabilities correspond to observed frequencies over repeated predictions in the population being considered. For alternative \(j\), class-specific calibration means that among observations assigned probability \(p\) to that alternative, the long-run proportion choosing it is \(p\). In a finite sample, exact equality is not expected, so calibration is assessed approximately.

A practical table groups observations by predicted probability for a particular alternative. Within each group, compare the mean predicted probability with the observed proportion choosing that alternative. Groups must be defined transparently; very wide groups can hide patterns, while very narrow groups can contain too few observations for stable proportions. Calibration tables are descriptive diagnostics, not proof that probabilities are calibrated.

Proper Scoring Rules for Probability Forecasts

A scoring rule evaluates a probability forecast against the outcome that occurs. Unlike accuracy, a probability score uses the entire forecast vector. A scoring rule is proper if, in expectation, a forecaster minimizes expected loss by reporting the true probabilities. It is strictly proper if those probabilities are the unique minimizer. Proper scores discourage forecasts that distort probabilities merely to improve the score.

Two standard choices for multinomial outcomes are the log loss and the multiclass Brier score. Both are losses, so smaller values indicate better performance on the same set of observations. The log loss penalizes assigning very small probability to the category that occurs. The Brier score measures squared differences between each predicted probability and the corresponding zero-or-one outcome indicator.

For a multiclass outcome, the Brier score ranges from zero to at most two. Zero occurs when every probability vector assigns probability one to the observed category. A perfect log score is also zero, but log loss is unbounded above if a forecast assigns probability approaching zero to the category that occurs. In theory, finite-utility multinomial logit probabilities are positive; numerical rounding or other prediction procedures can nevertheless produce zeros, which require careful handling rather than silently treating the log loss as finite.

For the log-score justification, nonnegativity follows from \(\log x\leq x-1\): applying this inequality to \(x=p_j/q_j\) for categories with \(q_j>0\) gives \(\sum_j q_j\log(p_j/q_j)\leq\sum_j(p_j-q_j)\leq 0\). Equivalently, \(\sum_jq_j\log(q_j/p_j)\geq 0\); equality requires the forecast to match the true distribution. The Brier derivation shows the same principle directly through squared distance from the true probability vector.

Prediction Is Not Structural Interpretation

Prediction evaluation asks how well a model forecasts outcomes for observations from a target setting. Structural interpretation asks what the model’s parameters and choice assumptions imply about behavior and substitution. A model can predict well in a particular evaluation sample without its structural assumptions being correct. Conversely, a theoretically motivated model can be useful for interpretation yet forecast imperfectly. Accuracy, calibration, and proper scores do not by themselves establish a causal effect, validate the model’s behavioral assumptions, or show that predicted probabilities will transport to a different choice environment.

The decision rule also matters. If a decision incurs zero loss for a correct category and equal loss for every incorrect one, choosing the largest probability minimizes expected classification error. With a general cost \(L(a,j)\) for taking action \(a\) when category \(j\) occurs, the appropriate action minimizes expected cost:

This distinction is important in applications where, for example, incorrectly predicting one alternative is more consequential than confusing two others. The model supplies probabilities; the decision-maker’s loss function determines how those probabilities should be converted into actions.

Common Mistakes and Practical Reporting

• Treating accuracy as a probability assessment. Accuracy records whether the most likely category was correct; it does not assess the quality of the full probability vector. Report a proper score or calibration assessment when probability quality matters.
• Calling a small calibration table proof of calibration. Grouped proportions fluctuate, and bin choices affect the display. Treat the table as a diagnostic summary, with group sizes and predicted means visible.
• Choosing the largest probability regardless of the decision. That rule is optimal only under equal misclassification costs. For unequal costs, minimize expected loss using the stated cost function.
• Interpreting predictive performance as structural validation. Good held-out predictions do not prove that behavioral assumptions or coefficient interpretations are correct.
• Evaluating on the fitting sample without qualification. In-sample performance can be optimistic. State how observations were held out and which population or setting the evaluation is intended to represent.