Nested Logit Diagnostics And Nest Selection

Why Nest Selection Requires More Than Better Fit

A nested-logit model groups alternatives that are expected to be closer substitutes for one another than for options in other groups. A tree is not just a convenient way to improve in-sample fit: it makes a behavioral claim about how substitution works. A useful diagnostic therefore asks whether that claim is plausible, whether the estimated parameters support it, and whether its predictions hold up beyond the data used to estimate it.

These checks answer different questions. Diversion patterns provide behavioral evidence; dissimilarity parameters and likelihood comparisons assess statistical support under a specified model; held-out predictions assess performance on new choices. No single diagnostic establishes that a tree is correct. In particular, a larger likelihood is expected when a model adds flexibility, and should not by itself decide which structure to use.

Model and Restrictions to Check

For a one-level tree, let \(j\) index alternatives and \(m\) index nests. Alternative \(j\) belongs to nest \(m(j)\). Let \(V_{ij}\) be the systematic utility of alternative \(j\) for decision maker \(i\), and let \(W_{im}\) be any systematic component at the nest level. The inclusive value for nest \(m\) is \(I_{im}=\log\sum_{k\in m}\exp(V_{ik}/\lambda_m)\). The nested-logit choice probability factors into a conditional probability within a nest and a probability of choosing that nest.

The scale normalization is the one used for the upper level of the tree. Under the standard random-utility-consistent specification, each dissimilarity parameter satisfies \(0<\lambda_m\leq 1\). A value below one permits stronger association in unobserved utility components within the nest; \(\lambda_m=1\) removes that additional within-nest dependence for that nest. If all relevant dissimilarity parameters equal one, the model reduces to multinomial logit under the corresponding utility specification. The restriction is a model condition, not an optional interpretation of an estimate.

Behavioral Similarity and Substitution

A nest should group alternatives that are plausibly close substitutes. One source of evidence is diversion: when an alternative becomes unavailable, where does its lost choice probability go? In the usual interpretation, a diversion ratio from alternative \(j\) to alternative \(k\) is the probability transferred to \(k\), divided by the probability lost from \(j\), under a specified removal or price-change calculation. Alternatives in the same nest should generally show relatively strong substitution with one another compared with alternatives in different nests.

This is a diagnostic pattern, not a rule that every within-nest diversion ratio must exceed every between-nest ratio. The pattern depends on the systematic utilities, the choice sets, and the particular change being considered. It is also important to use diversion estimates that are relevant to the setting of interest; an average pattern can hide substantial variation across decision makers or markets.

Parameter Restrictions and Likelihood Comparisons

After estimating a proposed structure, inspect the estimated dissimilarity parameters, their uncertainty, and the behavior of the optimization. Estimates well below one can indicate a meaningful within-nest dependence in the fitted model. Estimates close to one suggest little evidence that the corresponding nest needs its own dissimilarity parameter. An estimate above one conflicts with the standard random-utility-consistency restriction. A value close to zero, a boundary estimate, or a large standard error can indicate weak identification, a poorly supported structure, or numerical difficulty; it should prompt investigation rather than a mechanical interpretation.

Likelihood comparisons are useful when the models are genuinely nested: the restricted model must be obtainable from the more flexible model by imposing restrictions, and both must use the same observations and compatible utility specifications. For example, multinomial logit can be the restriction of a nested model in which the relevant dissimilarity parameters equal one. The likelihood-ratio statistic is \(LR=2(\ell_U-\ell_R)\), where \(\ell_U\) and \(\ell_R\) are the maximized log likelihoods under the unrestricted and restricted specifications. It cannot be negative when both are maximized over properly nested parameter spaces.

A caution matters here: \(\lambda_m=1\) is the upper boundary of the usual parameter space \(0<\lambda_m\leq1\). Consequently, a routine chi-square reference distribution for the likelihood-ratio statistic is not automatically justified. Boundary restrictions can produce a nonstandard null distribution. A parametric bootstrap under the restricted model is one practical way to calibrate the comparison: simulate choices under the fitted null, re-estimate both models on each simulated data set, and compare the observed statistic with the simulated null distribution. The precise reference distribution depends on the restrictions and regularity conditions.

Likelihood-ratio testing does not directly compare two different trees if neither is a restriction of the other. For example, a tree grouping rail with bus and a competing tree grouping rail with car are generally non-nested specifications. Their maximized likelihoods can be reported, and criteria such as AIC or BIC can summarize fit with a penalty for parameter count, but an ordinary nested-model likelihood-ratio test is not valid for that comparison. Predictive assessment on data not used to fit the models is especially useful in this situation.

Out-of-Sample Substitution and Prediction

A tree can fit the estimation sample yet give poor probability forecasts in new choice situations. Reserve observations or use cross-validation, ensuring that repeated observations from the same decision maker or market are kept together when dependence within those units would otherwise leak information across training and validation sets. For each held-out choice, evaluate the probability the model assigned to the alternative actually chosen. The mean log score is the average of the logarithms of those probabilities; higher values (closer to zero) are better. A model that assigns a very small probability to an observed choice receives a strong penalty.

Predictive fit alone is not enough either. Inspect predicted substitution when alternatives are removed or their attributes change, and compare those patterns with observed or substantively plausible behavior. A good validation score can coexist with implausible substitution in a policy-relevant scenario. Conversely, a small difference in average score may not justify a more complicated tree if the models’ predictions are practically indistinguishable.

Justification for the Probability Decomposition

The nested-logit factorization shows why both within-nest and between-nest behavior matter. Write \(D_{im}=\sum_{k\in m}\exp(V_{ik}/\lambda_m)\), so \(I_{im}=\log D_{im}\). Multiplying the conditional and nest probabilities for alternative \(j\) in nest \(m\) gives

The denominator \(D_{im}\) contributes to the probability of choosing within the nest, while the inclusive value contributes to the probability of choosing the nest. Changing the membership of a nest therefore changes the model’s substitution implications, not merely its labels. This is why a candidate tree should be evaluated for the full pattern of predicted choices and substitution, rather than selected solely by the largest in-sample likelihood.

Common Pitfalls and Practical Conclusions

• Choosing the tree with the largest likelihood. More flexible models often fit the estimation sample better. Compare nested models with an appropriate restriction test, and use predictive checks or complexity-adjusted criteria for competing trees.
• Treating any estimated dissimilarity parameter as admissible. Under the standard random-utility-consistent specification, it must lie between zero and one. Estimates outside that range, at a boundary, or with large uncertainty require scrutiny of constraints, identification, and numerical convergence.
• Using an ordinary likelihood-ratio test for any pair of models. The standard nested-model comparison requires a restriction relationship. Different trees are often non-nested, and tests involving a parameter at a boundary need suitable calibration.
• Assuming that a significant test validates a tree. A test addresses a specified restriction, not whether the proposed grouping is behaviorally meaningful or predicts well relative to alternatives.
• Reading one diversion estimate as a universal substitution law. Diversion can vary with the choice set and covariates. Check the pattern across relevant situations and against substantive knowledge.