IIA And Pairwise Odds Invariance

Pairwise Odds and the Multinomial Logit Model

In a multinomial response, it is natural to ask how the odds of one category compare with those of another. The baseline-category logit model has a particularly simple answer: the odds for a pair depend on the predictors for that pair, not on the predictors for the other categories. This cancellation is the source of the model’s independence of irrelevant alternatives (IIA) property.

The word “independence” needs care here. IIA does not say that category probabilities are unaffected by other categories. Those probabilities share a normalizing denominator, so adding an alternative generally changes them. The claim concerns the ratio of the probabilities of two specified categories, provided their predictors are held fixed.

Let the response \(Y\) take values in \(\{0,1,\ldots,J\}\), with category 0 as the baseline. For a fixed covariate value \(x\), write \(\eta_j(x)\) for the linear predictor of category \(j\), and set \(\eta_0(x)=0\). The softmax probabilities are

p_j(x)=P(Y=j\mid x) =\frac{\exp\{\eta_j(x)\}}{\sum_{r=0}^{J}\exp\{\eta_r(x)\}}, \qquad j=0,\ldots,J.

The denominator is positive, and the probabilities sum to one. The pairwise odds of category \(j\) against category \(k\) are \(p_j(x)/p_k(x)\), for any two distinct categories \(j\) and \(k\). This definition applies whether or not either category is the baseline.

Definition: The baseline-category logit model satisfies pairwise odds invariance if, for any two categories \(j\) and \(k\), their probability ratio at fixed covariates is unchanged by the predictors or availability of other categories, as long as the predictors for \(j\) and \(k\) themselves are held fixed. This model property is commonly called independence of irrelevant alternatives (IIA).

The Pairwise Odds Identity

The key result follows directly from the softmax formula. The same denominator appears in the probabilities of both categories, so it cancels in their ratio.

Theorem: For any distinct \(j,k\in\{0,\ldots,J\}\) and any fixed \(x\), \[ \frac{P(Y=j\mid x)}{P(Y=k\mid x)} =\exp\{\eta_j(x)-\eta_k(x)\}, \qquad \log\frac{P(Y=j\mid x)}{P(Y=k\mid x)} =\eta_j(x)-\eta_k(x). \] In particular, no predictor for a category other than \(j\) or \(k\) appears in this pairwise log-odds contrast.

Proof. Define \(D(x)=\sum_{r=0}^{J}\exp\{\eta_r(x)\}\). Since every exponential term is positive, \(D(x)>0\). By the softmax formula,

\frac{P(Y=j\mid x)}{P(Y=k\mid x)} = \frac{\exp\{\eta_j(x)\}/D(x)} {\exp\{\eta_k(x)\}/D(x)} = \frac{\exp\{\eta_j(x)\}}{\exp\{\eta_k(x)\}} = \exp\{\eta_j(x)-\eta_k(x)\}.

Both probabilities are positive, so taking logarithms is valid and gives the second identity. The expression contains only the two stated predictors; predictors for all other categories have disappeared with the common denominator. \(\square\)

If one of the categories is the baseline, its predictor is zero. Thus the log odds of category \(j\) against category 0 equal \(\eta_j(x)\). For two nonbaseline categories, their log odds are the difference of their baseline-category predictors. This is a contrast, not generally either predictor on its own.

Worked Example: Comparing Two Categories Directly

Suppose a response has categories 0, A, and B, and at a fixed \(x\) the predictors are \(\eta_0=0\), \(\eta_A=\log 2\), and \(\eta_B=\log 3\). Find all three probabilities and the odds of A against B.

The softmax denominator is \[ D=1+\exp(\log 2)+\exp(\log 3)=1+2+3=6. \] Therefore, \[ P(Y=0\mid x)=\frac{1}{6},\qquad P(Y=A\mid x)=\frac{2}{6}=\frac{1}{3},\qquad P(Y=B\mid x)=\frac{3}{6}=\frac{1}{2}. \]

The odds of A against B are \[ \frac{P(Y=A\mid x)}{P(Y=B\mid x)} =\frac{1/3}{1/2} =\frac{2}{3}. \] The predictor contrast gives the same result: \[ \exp(\eta_A-\eta_B) =\exp(\log 2-\log 3) =\frac{2}{3}. \] The baseline category affects the probabilities through the denominator, but it does not enter the A-versus-B odds.

What Changes When Alternatives Are Added or Removed?

A precise way to express IIA is to consider a choice set \(S\) of available categories. Suppose each category \(r\in S\) has a fixed predictor \(\eta_r(x)\), and its probability within that set is proportional to \(\exp\{\eta_r(x)\}\). For distinct \(j,k\in S\), the ratio of their probabilities is \(\exp\{\eta_j(x)-\eta_k(x)\}\), regardless of which other categories are in \(S\).

Theorem: Let \(S\) be a finite set of categories with probabilities \[ p_r^{(S)}(x)= \frac{\exp\{\eta_r(x)\}}{\sum_{s\in S}\exp\{\eta_s(x)\}}, \qquad r\in S. \] If \(j,k\in S\) and their predictors are unchanged when the choice set changes, then \[ \frac{p_j^{(S)}(x)}{p_k^{(S)}(x)} =\exp\{\eta_j(x)-\eta_k(x)\} \] is the same for every choice set containing \(j\) and \(k\).

Proof. Fix \(x\), and let \(S\) contain \(j\) and \(k\). The sum \(D_S(x)=\sum_{s\in S}\exp\{\eta_s(x)\}\) is positive. Hence

\frac{p_j^{(S)}(x)}{p_k^{(S)}(x)} = \frac{\exp\{\eta_j(x)\}/D_S(x)} {\exp\{\eta_k(x)\}/D_S(x)} = \exp\{\eta_j(x)-\eta_k(x)\}.

The final expression does not involve \(S\) except through the requirement that \(j\) and \(k\) are available. Therefore, changing the other members of \(S\) leaves the ratio unchanged when \(\eta_j(x)\) and \(\eta_k(x)\) stay fixed. \(\square\)

Although this ratio is invariant, the individual probabilities need not be. For example, adding a category increases the denominator while leaving the numerators for existing categories unchanged, so their probabilities typically decrease. Removing a category instead renormalizes the probabilities of those that remain.

Worked Example: Adding and Removing an Alternative

Return to categories 0, A, and B with weights \(1,2,3\), respectively, where a weight means \(\exp(\eta_r)\). Add category C with weight 4 while keeping the other weights fixed.

Before C is added, the total weight is \(1+2+3=6\), giving \(p_A=2/6=1/3\) and \(p_B=3/6=1/2\). After adding C, the total is \(1+2+3+4=10\), so \[ p_A=\frac{2}{10}=\frac{1}{5},\qquad p_B=\frac{3}{10}. \] Both probabilities change. But their odds remain \[ \frac{p_A}{p_B} =\frac{1/5}{3/10} =\frac{2}{3}, \] the same as before.

If C is then removed, the remaining weights again sum to 6, and the probabilities of A and B return to \(1/3\) and \(1/2\). More generally, restricting attention to a retained set and renormalizing gives \[ P(Y=j\mid Y\in R,x) =\frac{p_j(x)}{\sum_{r\in R}p_r(x)} \] for \(j\in R\), provided \(P(Y\in R\mid x)>0\). The common conditioning denominator cancels in the ratio of any two retained categories.

Predictor Changes Are Not “Irrelevant”

IIA does not imply that pairwise odds are constant across covariate values. The theorem says that, at a fixed \(x\), other category predictors cancel. If a covariate changes either member of the pair’s predictor, the odds can change. In a linear-predictor model, the pairwise log odds are the difference between those two linear predictors.

Worked Example: A Covariate Changes the Pairwise Odds

Consider categories 0, A, B, and C with \[ \eta_0(x)=0,\qquad \eta_A(x)=\log 2+(\log 2)x,\qquad \eta_B(x)=0,\qquad \eta_C(x)=\log 3-(\log 2)x. \] Find the odds of A against B at \(x=0\) and \(x=1\), and identify whether C’s predictor appears in the calculation.

At \(x=0\), \(\eta_A(0)=\log 2\) and \(\eta_B(0)=0\), so \[ \frac{P(Y=A\mid x=0)}{P(Y=B\mid x=0)} =\exp\{\log 2-0\}=2. \] At \(x=1\), \(\eta_A(1)=2\log 2=\log 4\), while \(\eta_B(1)=0\), giving \[ \frac{P(Y=A\mid x=1)}{P(Y=B\mid x=1)} =\exp\{\log 4-0\}=4. \] Thus the odds change because the predictor for A changes relative to that for B. The predictor for C does not appear in either pairwise calculation. Its value affects the individual category probabilities, but not the A-versus-B ratio at a given \(x\).

Why IIA Matters—and a Common Pitfall

Pairwise odds invariance is a structural implication of the baseline-category softmax model. It can be useful when interpreting pairwise comparisons: a third category does not enter the model-implied odds for two specified categories, conditional on their predictors. It also signals a modeling limitation. In some applications, two alternatives may be close substitutes, so adding a near-duplicate of one alternative may be expected to affect the odds between that alternative and a quite different one. The baseline-category logit model imposes invariance rather than representing that kind of substitution directly.

For example, suppose “car” and “bus” are initially represented by weights 1 and 1. Their probabilities are each \(1/2\), and the car-to-bus odds are 1. Now replace the single bus option with two identical options, “red bus” and “blue bus,” each assigned weight 1, while keeping the car’s weight at 1. The resulting probabilities are \(1/3\) for each of car, red bus, and blue bus. The odds of car against red bus remain 1, but the combined probability of a bus option is \(2/3\), not \(1/2\). This illustrates how duplicating a category can change aggregate shares even while pairwise odds among the listed alternatives obey IIA.

Worked Example: The Duplicate-Alternative Pitfall

With weights 1 for car and 1 for bus, the total is 2, so \(P(\text{car})=1/2\), \(P(\text{bus})=1/2\), and the car-to-bus odds are \(1\).

After bus is split into red bus and blue bus, each with weight 1, the total is \(1+1+1=3\). Therefore, \[ P(\text{car})=\frac{1}{3},\quad P(\text{red bus})=\frac{1}{3},\quad P(\text{blue bus})=\frac{1}{3}. \] The odds of car against either bus option are \( (1/3)/(1/3)=1\), while the combined bus probability is \(1/3+1/3=2/3\). The calculation does not show that real travelers must behave this way; it shows what this model predicts when the listed predictors and weights are held fixed.

Common mistake: “Because the odds of A against B do not depend on C, adding C cannot affect the probability of A.” This conclusion is false. For instance, weights 2 for A, 3 for B, and 1 for the baseline give \(p_A=1/3\); adding a category with weight 4 changes \(p_A\) to \(1/5\), even though \(p_A/p_B\) stays \(2/3\). The correction is to distinguish a ratio from its numerator and denominator separately: IIA preserves the specified pairwise ratio, not the individual probabilities.

The algebraic result also has a scope condition. Comparing choice sets under IIA means holding the predictors of the compared categories fixed. If changing the set changes those predictors, or if a model is re-estimated on different data and obtains different coefficient estimates, this cancellation alone does not imply the same numerical odds. The identity is a property of the model probabilities at the stated predictors; it is not a claim that all empirical comparisons must remain unchanged.

Key takeaway: In the baseline-category logit model, the log odds of category \(j\) against category \(k\) equal \(\eta_j(x)-\eta_k(x)\). Other category predictors cancel, but the denominator still makes individual probabilities sensitive to other alternatives.

Check Your Understanding

Use the softmax formula and the distinction between odds and probabilities to answer these questions.

For categories 0, 1, and 2 with predictors \(0\), \(\log 5\), and \(\log 2\), respectively, find the odds of category 1 against category 2.
Explain why adding a category can change \(P(Y=j\mid x)\) without changing the odds of category \(j\) against category \(k\).
State the condition on predictors needed to compare the same pairwise odds across two choice sets.
Does IIA imply that the odds of categories A and B stay constant as a covariate changes? Explain.
In the duplicate-bus example, why does the combined bus probability change even though the odds of car against each bus option are unchanged?