Testing IIA And Diagnosing Multinomial Logit

Why Diagnose IIA?

A multinomial logit (MNL) model imposes a particular pattern of substitution across alternatives. That restriction can be useful, but it may be implausible when some options are much closer substitutes than others. For example, adding a second version of a product may draw disproportionately from the first version rather than from every existing option in proportion to its probability. Diagnosing this concern matters because a model can predict overall choice shares reasonably well while still giving misleading comparisons between alternatives.

No single diagnostic settles whether IIA is an adequate assumption for a given application. Knowledge of the choice context and observed probability patterns can reveal reasons for concern. Formal tests, including the Hausman–McFadden and Small–Hsiao tests, provide additional evidence, but their finite-sample behavior can be unstable. A failure to reject is not proof that IIA is true, and a rejection does not by itself identify a better choice model.

Definition: The independence of irrelevant alternatives (IIA) property in MNL means that, for any two available alternatives $j$ and $k$, the ratio of their choice probabilities does not depend on the attributes or availability of other alternatives. Under the standard MNL probability formula, $P(y=j\mid x)/P(y=k\mid x)=\exp(V_j-V_k)$, where $V_j$ and $V_k$ are their systematic utilities.

Probability Patterns That Imply IIA

For a choice set $C$, standard MNL assigns alternative $j\in C$ probability $\exp(V_j)/\sum_{l\in C}\exp(V_l)$. Dividing the probability for $j$ by the probability for $k$ cancels the denominator. This is the source of IIA: changing the other alternatives does not change the predicted odds between $j$ and $k$, provided their own utilities are unchanged.

\frac{P(y=j\mid x,C)}{P(y=k\mid x,C)} = \frac{\exp(V_j)}{\exp(V_k)} = \exp(V_j-V_k)

Worked Example: What Happens When a Third Alternative Is Added?

Suppose that, for a particular person and choice context, the MNL utilities of alternatives A, B, and C are $V_A=0$, $V_B=\log(2/3)$, and $V_C=0$. Calculate the probabilities with all three alternatives available, then compare A and B with C removed.

With all three alternatives available, their exponentiated utilities are $1$, $2/3$, and $1$. The denominator is $1+2/3+1=8/3$. Thus $P(A)=1/(8/3)=0.375$, $P(B)=(2/3)/(8/3)=0.250$, and $P(C)=1/(8/3)=0.375$. The odds of A versus B are $0.375/0.250=1.5$.

If C is removed, the denominator becomes $1+2/3=5/3$. The probabilities are $P(A)=1/(5/3)=0.600$ and $P(B)=(2/3)/(5/3)=0.400$. Their odds remain $0.600/0.400=1.5$. MNL therefore predicts that removing C reallocates probability between A and B without changing their relative odds.

This calculation is an implication of the model, not a claim that real people must substitute in this way. If A and C are nearly identical options while B is quite different, equal proportional reallocation may be substantively doubtful.

Choice-set knowledge is often the first diagnostic. Ask whether the alternatives are plausibly differentiated by attributes represented in the utilities, whether some are near-duplicates, and whether a change in availability would realistically affect all other options proportionally. These questions identify a modeling concern; they do not constitute a statistical test. Observed choice patterns can also be suggestive, but must be interpreted in light of differences in people, attributes, and choice contexts.

Worked Example: Comparing Observed Choice Odds Across Contexts

Imagine a hypothetical dataset with two groups of otherwise comparable people facing the same A and B attributes. In one group, C is available; among people choosing A or B, 45 choose A and 30 choose B. In a second group, C is unavailable; 30 choose A and 20 choose B. Compare the observed A-to-B odds.

In the first group, the odds are $45/30=1.50$. In the second, they are $30/20=1.50$. These counts show no difference in the observed A-to-B odds between the two contexts. They are consistent with the IIA pattern for this comparison, but do not establish it: the counts are finite, the groups might differ in unmeasured ways, and other covariates may matter.

If instead the second group had 36 choices of A and 14 choices of B, its odds would be $36/14\approx2.57$, compared with $1.50$ in the first group. That difference would raise a diagnostic question, but would not alone demonstrate an IIA violation. Sampling variation and group composition would need to be assessed.

The Hausman–McFadden Test

The Hausman–McFadden (HM) test compares estimates from the full MNL model with estimates from a model in which one alternative has been removed. Under IIA, conditioning on not choosing the removed alternative gives the MNL probabilities over the remaining alternatives. Consequently, the full-model estimates and the estimates from this reduced choice set should be consistent for the common parameters, assuming the model is correctly specified and the regularity conditions for the comparison hold.

A crucial feature of the standard HM comparison is how the two models are estimated. The unrestricted full-choice-set model is estimated using the full sample, including observations whose chosen alternative is the one being tested for removal. The reduced model is estimated after excluding those observations and removing that alternative from the choice sets of the retained observations. The full model must not be re-estimated as an ordinary MNL on only the retained observations while leaving the removed alternative available: that likelihood does not account for selection into the retained sample.

P(y=j\mid y\ne q,x) = \frac{P(y=j\mid x)}{P(y\ne q\mid x)} = \frac{\exp(V_j)}{\sum_{l\ne q}\exp(V_l)}, \qquad j\ne q

The equality follows by dividing the full-model probability for $j$ by the sum of the probabilities for all alternatives except $q$. The common denominator in the full probabilities cancels. Thus, under IIA, the conditional probabilities among retained alternatives have the reduced MNL form. This conditional argument justifies the reduced estimation; it does not turn the selected observations into an ordinary full-choice-set sample.

Test statistic: Let $\hat{\beta}_U$ be the unrestricted estimate of the parameters common to both models, estimated from the full sample and full choice set. Let $\hat{\beta}_R$ be the corresponding estimate from the reduced sample and reduced choice set. With covariance estimates $V_U$ and $V_R$, the Hausman statistic is $$ H=(\hat{\beta}_R-\hat{\beta}_U)'(V_R-V_U)^{-1}(\hat{\beta}_R-\hat{\beta}_U). $$ Under the null hypothesis of IIA and the test’s regularity conditions, $H$ is compared with a chi-square distribution with degrees of freedom equal to the rank of the tested covariance difference. A large statistic is evidence against the null.

Only parameters that are meaningfully common to both specifications should enter the comparison. Align the parameter vectors, their order, and the relevant covariance matrices before computing the statistic. If parameters are not identified in one model or the covariance difference is singular, the usual statistic is not directly available. The classical covariance-difference form also relies on the unrestricted estimator having the relevant efficiency property under the null.

Worked Example: Computing an HM Statistic

Suppose a hypothetical application removes alternative $q$. The unrestricted estimate from the full sample is $\hat{\beta}_U=(0.50,-0.20)'$, and the reduced estimate from the retained sample with $q$ removed is $\hat{\beta}_R=(0.80,-0.10)'$. Suppose the covariance estimates for the aligned common parameters are $V_U=\begin{pmatrix}0.05&0\\0&0.03\end{pmatrix}$ and $V_R=\begin{pmatrix}0.09&0\\0&0.04\end{pmatrix}$.

The estimate difference is $\hat{\beta}_R-\hat{\beta}_U=(0.30,0.10)'$. The covariance difference is $V_R-V_U=\begin{pmatrix}0.04&0\\0&0.01\end{pmatrix}$, whose inverse is $\begin{pmatrix}25&0\\0&100\end{pmatrix}$. Therefore,

H= \begin{pmatrix}0.30&0.10\end{pmatrix} \begin{pmatrix}25&0\\0&100\end{pmatrix} \begin{pmatrix}0.30\\0.10\end{pmatrix} =25(0.30)^2+100(0.10)^2 =3.25.

There are two tested parameters, so the reference distribution is chi-square with 2 degrees of freedom. The p-value is approximately $P(\chi^2_2\ge3.25)=0.197$. At the 5% significance level, this result does not reject the IIA null for this comparison. It does not show that IIA is true; it says that this test, with these data and assumptions, does not provide strong evidence against it.

Small–Hsiao and Finite-Sample Instability

The Small–Hsiao test is another likelihood-based test of IIA that uses sample splitting. In broad terms, the sample is divided into subsamples and likelihood evidence for the full and alternative-removed specifications is constructed using that split. Its purpose is to avoid relying on the same covariance-difference comparison used by the HM test. As with HM, the alternative-removal comparison is based on the implication that, under IIA, choice among the remaining alternatives follows the reduced MNL probabilities.

The precise implementation matters: the split, estimation samples, likelihood calculations, and degrees of freedom must follow the chosen procedure. It should not be replaced by an ordinary likelihood-ratio test that simply compares two likelihoods computed on different sets of observations. Such likelihoods are not automatically comparable. In reporting a Small–Hsiao test, state the alternative removed, the implementation and split used, the statistic, its reference distribution, and the p-value. Because the result depends on a random partition, different splits can produce different conclusions.

Both tests can be unstable in finite samples. The reduced sample may be much smaller when many people choose the removed alternative. Estimates can then be imprecise, and the covariance difference in HM may be nearly singular or fail to be positive definite. Sparse choices, influential observations, weak identification, and small changes in specification can also affect results. A Small–Hsiao result may vary with the split. These are not reasons to report only the most favorable test; they are reasons to treat a test statistic as one part of a broader diagnosis.

Worked Example: Reading Conflicting Diagnostic Evidence

Suppose an analyst examines hypothetical travel choices among a train, a bus, and a car. The HM test for removing the bus yields $H=7.8$ on 3 degrees of freedom, with a p-value of approximately $0.050$. Two Small–Hsiao runs, using different random splits, yield p-values of $0.18$ and $0.04$.

At a 5% significance level, the HM result is at the decision boundary, while the Small–Hsiao runs disagree. It would be inaccurate to conclude that the tests have established either IIA or its failure. The analyst should check the retained sample size and estimation stability, review whether train and bus are unusually close substitutes, examine probability predictions in relevant choice contexts, and report the sensitivity to the split. The evidence is mixed and warrants caution about relying on IIA as an unquestioned behavioral description.

Using Diagnostics Without Overclaiming

A useful diagnostic combines substantive reasoning with empirical evidence. Begin by asking whether the IIA substitution restriction makes sense for the alternatives and the decision process. Then examine whether predicted probabilities and observed choices show patterns that motivate closer scrutiny. Finally, use formal tests with correctly constructed samples and clearly stated assumptions. A p-value measures the extremeness of a test statistic under the null and the specified reference model; it is not the probability that IIA is true.

When a test rejects, first verify that the comparison is valid: the same common parameters must be aligned, the full model must use the full sample, and the reduced model must condition on not choosing the removed option. Consider whether the finding is sensitive to which alternative is removed, to sparse cells, or to the specification. When tests do not reject, do not infer that substitution patterns are proven to follow IIA. Limited information can make a test unable to detect meaningful departures.

A common mistake is to retain only people who chose alternatives other than $q$, estimate an ordinary full MNL while keeping $q$ available, and call that the unrestricted HM model. This treats a sample selected by $y\ne q$ as if it were an unselected full sample. Its likelihood uses $P(y=j\mid x)$, whereas the selected observations require probabilities conditional on $y\ne q$. For the standard HM comparison, estimate the unrestricted full-choice-set model on the full sample; estimate the reduced model on the retained observations with $q$ removed; then compare the aligned common parameters using the covariance-difference test when its conditions are satisfied.

Key takeaway: IIA is a restriction on relative choice probabilities, not merely a feature of a convenient estimation procedure. Choice-set knowledge, probability patterns, and formal tests can all inform its plausibility, but finite-sample instability means that no single diagnostic should be treated as decisive.

Check Your Understanding

Answer each question using the definitions and procedures in this tutorial.

Under IIA, what happens to the odds between two alternatives when a third alternative is removed from the choice set?
For a standard HM comparison removing alternative $q$, which sample is used to estimate the unrestricted model, and which observations and choice sets are used for the reduced model?
Why is an ordinary full-MNL likelihood on only the retained observations, with $q$ still available, not the correct unrestricted likelihood for the selected sample?
In an HM test with two common parameters, what reference distribution is used under the usual regularity conditions?
Give two reasons why HM or Small–Hsiao test results may be unstable in finite samples.
Why does failure to reject IIA not establish that the property is true?