How Covariates Redistribute Category Probabilities

From Logit Slopes to Probability Changes

A multinomial logit model assigns a probability to each possible response category. Its coefficients describe changes in category log odds, but a probability is affected by the scores of all categories at once: increasing one category’s score changes its share of the total, while the probabilities must still sum to one. To understand the effect of a continuous predictor, we therefore need the derivative of each predicted probability, not just the slope of a logit.

Consider \(K\) response categories and a scalar continuous predictor \(x\), with other predictors held fixed. Write \(\eta_j(x)\) for category \(j\)’s linear predictor, using category \(K\) as the baseline, so that \(\eta_K(x)=0\). The softmax probabilities are

p_j(x)=\frac{\exp\{\eta_j(x)\}}{\sum_{r=1}^{K}\exp\{\eta_r(x)\}},\qquad j=1,\ldots,K.

The baseline convention is a way to represent the relative scores; it does not mean that the baseline category has no probability. Each \(p_j(x)\) is positive, and the probabilities sum to one. Suppose the functions \(\eta_j(x)\) are differentiable. Define \(g_j(x)=d\eta_j(x)/dx\), with \(g_K(x)=0\) under the baseline convention. In the common linear-predictor case, \(\eta_j(x)=\alpha_j+\beta_jx\) for \(j<K\), so \(g_j(x)=\beta_j\).

Definition: The probability-weighted average slope at predictor value \(x\) is \( \bar g(x)=\sum_{r=1}^{K}p_r(x)g_r(x) \). It averages the category-specific rates of change in the linear predictors, using the current predicted category probabilities as weights.

The Softmax Derivative

The central calculation expresses each probability’s rate of change as the difference between its category’s slope and the current weighted average slope.

Theorem: If each \(\eta_j(x)\) is differentiable, then \( \displaystyle \frac{dp_j(x)}{dx}=p_j(x)\{g_j(x)-\bar g(x)\} \) for every category \(j\). Thus the derivative has the sign of the category’s slope relative to the probability-weighted average slope.

Proof. Let \(S(x)=\sum_{r=1}^{K}\exp\{\eta_r(x)\}\). Then \(p_j(x)=\exp\{\eta_j(x)\}/S(x)\), and differentiation gives

\begin{aligned} \frac{dp_j}{dx} &=\frac{\exp\{\eta_j\}g_j S-\exp\{\eta_j\}S'}{S^2},\\ S'&=\sum_{r=1}^{K}\exp\{\eta_r\}g_r,\\ \frac{dp_j}{dx} &=p_j\left(g_j-\frac{\sum_{r=1}^{K}\exp\{\eta_r\}g_r}{S}\right) =p_j\left(g_j-\sum_{r=1}^{K}p_r g_r\right). \end{aligned}

The last equality follows from \(\exp\{\eta_r\}/S=p_r\), and the weighted sum is \(\bar g\). This proves the formula. \(\square\)

Worked Example: A Predictor Raises One Category’s Probability

There are three categories, \(A,B,C\), with \(C\) as baseline. Let \( \eta_A(x)=\log(2/3)+0.8x \), \( \eta_B(x)=\log(5/3)-0.2x \), and \( \eta_C(x)=0 \). At \(x=0\), the unnormalized category weights are \(2/3,5/3,1\). Their sum is \(10/3\), so the probabilities are \(p_A=0.2\), \(p_B=0.5\), and \(p_C=0.3\).

The slopes are \(g_A=0.8\), \(g_B=-0.2\), and \(g_C=0\). Therefore the weighted average slope at \(x=0\) is \( \bar g=0.2(0.8)+0.5(-0.2)+0.3(0)=0.16-0.10=0.06 \). The probability derivatives are

\begin{aligned} \frac{dp_A}{dx}&=0.2(0.8-0.06)=0.148,\\ \frac{dp_B}{dx}&=0.5(-0.2-0.06)=-0.130,\\ \frac{dp_C}{dx}&=0.3(0-0.06)=-0.018. \end{aligned}

At \(x=0\), a one-unit increase in \(x\) has an instantaneous effect of raising \(A\)’s probability at rate \(0.148\) per unit and lowering \(B\)’s and \(C\)’s probabilities at rates \(0.130\) and \(0.018\). These are local rates, not a promise that the same changes hold over a full one-unit increase.

Why the Probability Changes Balance

Probability is redistributed rather than created or lost: at every predictor value, the category probabilities sum to one. Their derivatives must consequently sum to zero. This also follows directly from the derivative formula.

Result: Under the differentiability conditions above, \( \sum_{j=1}^{K} dp_j(x)/dx=0 \). A category’s probability can rise only if the other category probabilities collectively fall at the same rate.

Proof. Substitute the derivative formula and use \(\sum_j p_j=1\):

\begin{aligned} \sum_{j=1}^{K}\frac{dp_j}{dx} &=\sum_{j=1}^{K}p_j(g_j-\bar g)\\ &=\sum_{j=1}^{K}p_jg_j-\bar g\sum_{j=1}^{K}p_j\\ &=\bar g-\bar g(1)=0. \end{aligned}

So the rates balance exactly, not merely approximately. \(\square\)

Worked Example: A Slope for Only One Category

Suppose the baseline is category \(C\), with \( \eta_A(x)=\log(0.4)+0.6x \), \( \eta_B(x)=\log(0.6) \), and \(\eta_C(x)=0\). At \(x=0\), the weights are \(0.4,0.6,1\), which sum to \(2\); hence the probabilities are \(0.2,0.3,0.5\). Only \(A\)’s logit has a nonzero slope: the slopes are \(0.6,0,0\).

The weighted average slope is \(\bar g=0.2(0.6)=0.12\). Thus

\frac{dp_A}{dx}=0.2(0.6-0.12)=0.096,\qquad \frac{dp_B}{dx}=0.3(0-0.12)=-0.036,\qquad \frac{dp_C}{dx}=0.5(0-0.12)=-0.060.

The derivatives add to \(0.096-0.036-0.060=0\). Even though only \(A\)’s linear predictor changes directly, all three probabilities change because each is a share of the sum of the category weights.

Reading the Sign and Size of a Derivative

Because \(p_j(x)>0\), the theorem gives a precise sign rule: category \(j\)’s probability is increasing at \(x\) if \(g_j(x)>\bar g(x)\), decreasing if \(g_j(x)<\bar g(x)\), and has zero derivative if the two are equal. The magnitude also depends on \(p_j(x)\). A category with a small probability can have a modest probability derivative even when its slope differs substantially from the average.

For linear predictors, the slopes are constant, but \(\bar g(x)\) usually is not: the probabilities used to weight the slopes vary with \(x\). A category’s probability derivative can therefore change as \(x\) changes. The sign rule describes the instantaneous change at the predictor value being considered, not necessarily the direction of change over the entire predictor range.

Worked Example: Checking a Derivative at a Second Predictor Value

Let the three category scores be \(u_A(x)=x\), \(u_B(x)=0\), and \(u_C(x)=-x\), with probabilities obtained by applying the softmax to these scores. Using \(C\) as baseline gives \(\eta_A(x)=2x\), \(\eta_B(x)=x\), and \(\eta_C(x)=0\); the relative slopes are therefore \(2,1,0\). At \(x=0\), all probabilities are \(1/3\), so \(\bar g=(2+1+0)/3=1\). The derivatives there are \(1/3,0,-1/3\).

At \(x=0.5\), it is convenient to use the equivalent scores \(u_A,u_B,u_C\). The unnormalized weights are \(e^{0.5},1,e^{-0.5}\), with sum approximately \(3.2553\). Thus the probabilities are approximately \(0.5065,0.3072,0.1863\). The slopes in this score representation are \(1,0,-1\), so their weighted average is approximately \(0.5065(1)+0.3072(0)+0.1863(-1)=0.3202\). The derivatives are consequently

\begin{aligned} \frac{dp_A}{dx}&\approx0.5065(1-0.3202)=0.3443,\\ \frac{dp_B}{dx}&\approx0.3072(0-0.3202)=-0.0984,\\ \frac{dp_C}{dx}&\approx0.1863(-1-0.3202)=-0.2460. \end{aligned}

The rounded derivatives sum to about \(-0.0001\), with the discrepancy due to rounding; the exact sum is zero. The weighted average slope differs from its value at \(x=0\), illustrating why a derivative calculated at one predictor value should not be treated as a constant rate across the range.

A Common Interpretation Mistake

A positive category-specific logit slope does not by itself imply that the category’s predicted probability rises. For instance, at a predictor value where the probabilities of \(A,B,C\) are \(0.5,0.3,0.2\), suppose their relative slopes are \(0.2,1,0\). The weighted average slope is \(0.5(0.2)+0.3(1)+0.2(0)=0.4\). Since \(0.2<0.4\), the derivative for \(A\) is \(0.5(0.2-0.4)=-0.1\): \(A\)’s probability is falling at that value even though its own logit slope is positive. Category \(B\)’s score is rising faster, and probability depends on all category scores together.

The correction is to distinguish the two interpretations. A category-specific slope describes how that category’s linear predictor changes with \(x\). To determine the change in its probability, compare that slope with the current probability-weighted average slope, then multiply their difference by the category’s current probability.

This distinction matters when reporting model results. A coefficient alone does not give the change in predicted probability. The derivative supplies a local rate at a specified predictor value and covariate setting; predicted probabilities at particular values of \(x\) answer a different, finite-change question.

Key takeaway: For a differentiable softmax model, \(dp_j/dx=p_j(g_j-\bar g)\). Each category’s probability changes according to how its own score slope compares with the probability-weighted average slope, and the changes across categories always sum to zero.

Check Your Understanding

Use the derivative formula to answer these questions.

For three categories with probabilities \(0.2,0.5,0.3\) and slopes \(0.8,-0.2,0\), calculate the weighted average slope and all three probability derivatives.
Why must the derivatives of all category probabilities sum to zero?
If a category’s score slope equals the probability-weighted average slope at a particular value of \(x\), what is its probability derivative there?
Can a category with a positive score slope have a decreasing probability? State the condition that determines the answer.
In a linear-predictor model, why might a category’s probability derivative vary with \(x\) even though its score slope is constant?