Comparing Adjusted Probabilities Across Groups

Why Compare Probabilities at the Same Profile?

Suppose a hypothetical insurer is studying whether a new renewal reminder is associated with policy renewal. Customers who received the reminder may differ from those who did not: for example, the groups may have different age distributions. Comparing their overall renewal proportions combines the reminder contrast with those differences in customer mix. Logistic regression offers a way to compare groups at a specified profile, such as customers of the same age band and risk category.

A model-based comparison is conditional: it asks what renewal probabilities the fitted model assigns to two groups when the other predictors are held fixed. It is not automatically the same as comparing observed proportions, nor does it by itself establish a causal effect. The interpretation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: In logistic regression, a model-based probability for a specified predictor profile is the probability assigned by the fitted model to an outcome, conditional on that profile. An adjusted group comparison contrasts these probabilities after setting the other predictors to the same values for both groups. A crude observed proportion is the number of observed outcomes in a group divided by the number of observations in that group, without adjustment for other predictors.

From the Linear Predictor to a Probability

For a binary response \(Y_i\), let \(p_i=P(Y_i=1\mid \mathbf{x}_i)\), where \(\mathbf{x}_i\) is the predictor profile for observation \(i\). Under a logistic regression model, the conditional response is Bernoulli with probability \(p_i\), and the logit of that probability is a linear predictor:

\log\left(\frac{p_i}{1-p_i}\right)=\eta_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_kx_{ik}.

This model specifies the conditional mean \(E[Y_i\mid\mathbf{x}_i]=p_i\). The usual likelihood-based analysis treats observations as conditionally independent given their predictors and assumes that the stated logit form is suitable for the conditional probabilities. The predictor coding matters: an indicator is 1 for the category it represents and 0 for its reference category.

To calculate a probability, first calculate \(\eta_i\), then convert it. If \(o_i=p_i/(1-p_i)\) is the odds, the model gives \(o_i=e^{\eta_i}\). Solving the odds equation for \(p_i\) gives the inverse-logit formula:

p_i=\frac{e^{\eta_i}}{1+e^{\eta_i}}=\frac{1}{1+e^{-\eta_i}}.

The conversion is necessary because the linear predictor is on the log-odds scale, not the probability scale. A coefficient changes \(\eta_i\); it does not directly add that amount to \(p_i\).

Useful result: If changing a group indicator multiplies the odds by \(R\), and the starting probability is \(p\), then the new probability is \(p'=\dfrac{Rp}{1-p+Rp}\). The probability difference depends on the starting probability as well as on \(R\).

To justify the result, write the starting odds as \(p/(1-p)\). Multiplying by \(R\) gives new odds \(Rp/(1-p)\). If the new probability is \(p'\), its odds are \(p'/(1-p')\). Equating these and solving,

\frac{p'}{1-p'}=\frac{Rp}{1-p} \quad\Longrightarrow\quad p'(1-p)=Rp(1-p') \quad\Longrightarrow\quad p'=\frac{Rp}{1-p+Rp}.

This also explains why a constant odds ratio need not produce a constant probability difference: the same \(R\) is applied to different starting probabilities \(p\).

Comparing Groups While Holding Predictors Fixed

A conditional comparison requires a clearly stated profile. To compare reminder groups at the same age band, set the age indicator to the same value in both calculations. To compare them at another age band, repeat the calculation using that profile. If an interaction is present, the group contrast can change across profiles, so the interaction term must be included in each linear predictor.

Worked Example: Comparing Reminder Groups at a Fixed Age Band

In a hypothetical renewal study, let \(M=1\) indicate receipt of a reminder and \(A=1\) indicate the older age band. Suppose the fitted model is \(\operatorname{logit}(p)=\log(1/4)+\log(2)M+\log(3)A\). Calculate and compare the fitted renewal probabilities for reminder and no-reminder customers in the older age band. Also calculate the probability difference.

Method. Hold age fixed at \(A=1\), calculate the two linear predictors, and convert each to a probability. This avoids treating the reminder coefficient as a probability increase.

No reminder. With \(M=0,A=1\), \(\eta_0=\log(1/4)+\log(3)=\log(3/4)\). The odds are \(e^{\eta_0}=3/4\), so \(p_0=(3/4)/(1+3/4)=3/7=0.4286\).

Reminder. With \(M=1,A=1\), \(\eta_1=\log(1/4)+\log(2)+\log(3)=\log(3/2)\). The odds are \(e^{\eta_1}=3/2\), so \(p_1=(3/2)/(1+3/2)=3/5=0.6000\).

Comparison and check. The model-based difference is \(3/5-3/7=6/35=0.1714\), or about 17.14 percentage points. The odds ratio for the reminder is \(e^{\log(2)}=2\), which checks against the odds \( (3/2)/(3/4)=2\). The probability difference is not 2 and is not the coefficient \(\log(2)\).

The example compares two predictions for the same age profile. It does not assert that every customer has those probabilities, and it is not an unadjusted comparison of all customers in the two reminder groups. A different fixed profile can give a different probability difference even without an interaction.

Worked Example: An Interaction Changes the Conditional Contrast

For a hypothetical claim-prevention program, let \(T=1\) indicate participation and \(H=1\) indicate a high-risk category. Suppose \(\operatorname{logit}(p)=\log(1/9)+\log(3)T+\log(3)H+\log(1/2)TH\). Calculate the participation probability difference separately for the low-risk and high-risk categories.

Method. Evaluate the full linear predictor for each of the four profiles. In particular, include the product \(TH\) when both indicators equal 1.

Low risk, \(H=0\). Without participation, \(T=0\), so the odds are \(1/9\) and the probability is \((1/9)/(1+1/9)=1/10=0.1000\). With participation, \(T=1\) and \(TH=0\), so the odds are \((1/9)(3)=1/3\). The probability is \((1/3)/(1+1/3)=1/4=0.2500\). The difference is \(1/4-1/10=3/20=0.1500\).

High risk, \(H=1\). Without participation, \(T=0\), so the odds are \((1/9)(3)=1/3\) and the probability is \(1/4=0.2500\). With participation, \(T=1\) and \(TH=1\), so the odds are \((1/9)(3)(3)(1/2)=1/2\). The probability is \((1/2)/(1+1/2)=1/3=0.3333\). The difference is \(1/3-1/4=1/12=0.0833\).

Comparison and check. The participation odds ratio is 3 in the low-risk category and \(3(1/2)=1.5\) in the high-risk category. The smaller high-risk probability difference is consistent with its different starting probability and its different odds ratio. Omitting the interaction would incorrectly use the same conditional odds ratio in both categories.

Why Adjusted Predictions Can Differ From Crude Proportions

A crude proportion combines outcomes across the predictor profiles actually present in a group. If the two groups have different mixes of profiles, their crude proportions can differ from a comparison made at a common profile. The difference is not an arithmetic error: the quantities answer different questions.

A conditional prediction answers, “What probabilities does the model assign to these groups for this specified profile?” A crude proportion answers, “What fraction of the observed members of each group had the outcome?” A weighted average of profile-specific predictions can also be calculated for a chosen common population mix, but the weights must be stated. That standardized average is distinct from both a single-profile conditional comparison and an unadjusted proportion.

Worked Example: A Crude Comparison Reverses the Within-Category Comparison

In a hypothetical prevention study, \(T=1\) denotes participation and \(H=1\) denotes a high-risk category. A fitted model is \(\operatorname{logit}(p)=\log(1/8)+\log(2)T+\log(8)H\). The observed counts are: low-risk participants, 9 outcomes among 45; high-risk participants, 2 among 3; low-risk nonparticipants, 1 among 9; and high-risk nonparticipants, 44 among 88. Calculate the crude outcome proportions, the model-based conditional probabilities, and the equal-weight average of the two risk-category predictions for each participation group.

Method. First pool counts within participation groups for the crude proportions. Separately, evaluate the model at each risk category. Finally, average the two predicted probabilities using weights of one-half each.

Crude proportions. Among participants, there are \(9+2=11\) outcomes among \(45+3=48\), giving \(11/48=0.2292\). Among nonparticipants, there are \(1+44=45\) outcomes among \(9+88=97\), giving \(45/97=0.4639\). The crude proportion is lower among participants.

Model predictions for low risk. For \(T=0,H=0\), the odds are \(1/8\), giving \(p=1/9=0.1111\). For \(T=1,H=0\), the odds are \((1/8)(2)=1/4\), giving \(p=1/5=0.2000\).

Model predictions for high risk. For \(T=0,H=1\), the odds are \((1/8)(8)=1\), giving \(p=1/2=0.5000\). For \(T=1,H=1\), the odds are \((1/8)(2)(8)=2\), giving \(p=2/3=0.6667\). Participation has a higher predicted probability in each risk category.

Equal-weight averages and check. For participants, the average is \((1/5+2/3)/2=13/30=0.4333\). For nonparticipants, it is \((1/9+1/2)/2=11/36=0.3056\). The difference is \(13/30-11/36=23/180=0.1278\), higher for participants. The reversal occurs because participants in these data are concentrated in the low-risk category, while nonparticipants are concentrated in the high-risk category. The equal weights define a chosen common mix; they are not the observed group weights.

Common Mistakes and a Reliable Calculation Process

The preceding examples illustrate why a group comparison should report both the profile and the scale being compared. In particular, a regression coefficient is not itself a probability difference, and an observed proportion is not automatically adjusted just because a logistic model is also available.

State the profile and comparison.
Name the group values being contrasted and specify the values of the other predictors that will be held fixed.

Evaluate the full linear predictor.
Substitute every predictor value, including any interaction products, separately for each group.

Convert each predictor to a probability.
Use \(p=e^\eta/(1+e^\eta)\), or calculate the odds first and convert them.

Compare the intended quantities.
Subtract probabilities for a probability difference, compare odds for an odds ratio, and label crude proportions as observed group summaries.

Trap: adding a log-odds coefficient to a probability. If an odds ratio is 2 and the starting probability is 0.20, doubling the odds gives \(p'=2(0.20)/(0.80+2(0.20))=0.3333\), not \(0.20+1\) and not a 100-percentage-point increase.
Trap: changing the other predictors between groups. That comparison combines the group contrast with a profile change. For an adjusted comparison, use the same specified values for all other predictors.
Trap: dropping an interaction term. When an interaction is in the model, include it in the predictor for each profile. The group contrast can depend on the other interacting variable.
Trap: calling a crude proportion adjusted. Pooling observed counts does not hold covariates fixed. Describe it as crude unless a model-based or explicitly standardized calculation was used.

Key takeaway: Calculate conditional probabilities by evaluating the model at clearly stated profiles and applying the inverse-logit transformation. Compare like with like: the same fixed predictors for an adjusted group contrast, and explicit weights for any averaged prediction. Crude observed proportions describe the groups as observed, including their different predictor mixes.

Check Your Understanding

For each problem, calculate the requested model-based quantities before comparing them.

A fitted model is \(\operatorname{logit}(p)=\log(1/3)+\log(2)X\), where \(X\) is a binary indicator. Find the probabilities for \(X=0\) and \(X=1\), then find their difference. (Answer: \(p_0=0.2500\), \(p_1=0.4000\), difference \(=0.1500\).)
A model is \(\operatorname{logit}(p)=\log(1/8)+\log(2)A+\log(4)B+\log(1/2)AB\). Compare the probability for \(A=1\) with \(A=0\) at each value of \(B\). (Answer: At \(B=0\), probabilities are \(0.2000\) and \(0.1111\), difference \(0.0889\); at \(B=1\), both are \(0.3333\), difference \(0\).)
In a hypothetical study, group R has 8 outcomes among 40 low-risk members and 12 among 20 high-risk members; group S has 2 among 20 low-risk members and 30 among 60 high-risk members. Find each crude proportion and compare the within-category proportions. (Answer: R crude \(=20/60=0.3333\); S crude \(=32/80=0.4000\); R is higher in the low-risk category, \(0.2000\) versus \(0.1000\), and lower in the high-risk category, \(0.6000\) versus \(0.5000\).)
A model assigns a starting probability of \(0.30\) to one profile. A group indicator has coefficient \(\log(3)\). Calculate the other group’s probability at the same profile and the probability difference. (Answer: \(p'=3(0.30)/(0.70+3(0.30))=0.5625\); difference \(=0.2625\).)
A fitted model predicts probabilities \(0.12\) and \(0.32\) for group U in low- and high-risk categories, and \(0.20\) and \(0.50\) for group V. Calculate the differences at each category and the equal-weight average prediction for each group. (Answer: U minus V differences are \(-0.0800\) in the low-risk category and \(-0.1800\) in the high-risk category; equal-weight averages are \(0.2200\) for U and \(0.3500\) for V.)

What You'll Learn