Tutorials › Exam SRM - Statistics for Risk Modeling › Logistic Regression With Multi-Level Factors

Home  /  Tutorials  /  Exam SRM / Logistic Regression

Exam SRM · Logistic Regression · Tutorial 307 of 926

Logistic Regression With Multi-Level Factors

Use indicator variables to model multi-level factors, interpret comparisons with a reference category, and turn fitted log-odds into probabilities.

Intermediate 10 min read

What You'll Learn

  • Build a \(K-1\) indicator-variable representation for a factor with \(K\) categories.
  • Explain why an intercept and indicators for every category create redundant coding.
  • Interpret each factor coefficient as a log-odds contrast with the reference category.
  • Calculate fitted odds and probabilities for specified factor levels.
  • Compare two non-reference categories using a difference of coefficients.
  • Identify common errors in reference-category coding and probability calculations.

Why Multi-Level Factors Need Indicator Variables

A predictor such as distribution channel, territory, or policy type may have three or more categories. Logistic regression can include such a factor, but its category labels cannot be treated as numerical measurements: assigning codes such as 1, 2, and 3 would impose a numerical ordering and spacing that may have no meaning. Instead, the model represents category membership with indicator variables.

Suppose a factor has \(K\) possible categories. Choose one category as the reference, and create an indicator for each of the other \(K-1\) categories. Each indicator equals 1 when an observation belongs to its named category and 0 otherwise. In a logistic regression with an intercept, this leaves the reference category represented by all-zero indicators.

Definition: For a categorical factor with \(K\) categories and a chosen reference category, reference-cell indicator coding uses \(K-1\) indicator variables. The model intercept describes the reference category when other predictors are zero, and each factor coefficient describes the difference in log-odds between its category and the reference, with other predictors held fixed.

For a binary response \(Y\), let \(p=P(Y=1\mid \text{predictors})\). Logistic regression models the logit of this conditional probability as a linear predictor:

$$ \log\left(\frac{p}{1-p}\right)=\eta. $$

With a factor having reference category 1 and other categories 2 through \(K\), a model containing only that factor is

$$ \eta=\beta_0+\beta_2 I_2+\beta_3 I_3+\cdots+\beta_K I_K, $$

where \(I_j=1\) for category \(j\) and \(I_j=0\) otherwise. The intercept \(\beta_0\) is the reference category's log-odds. For category \(j\), the log-odds are \(\beta_0+\beta_j\), so \(\beta_j\) is the category's log-odds difference from the reference.

Model Conditions and Coefficient Interpretation

The usual logistic regression setup assumes that, conditional on the predictors, each outcome is Bernoulli with probability \(p\), and observations are independent. The logit of that probability is modeled using the stated linear predictor. The factor categories must be defined so that each observation belongs to one category, and the chosen coding must be identifiable. These conditions do not guarantee that the model is a good fit; they specify the structure being used.

If other predictors are present, they are included in \(\eta\). For example, a continuous predictor \(x\) with coefficient \(\gamma\) gives

$$ \eta=\beta_0+\beta_2 I_2+\cdots+\beta_K I_K+\gamma x. $$

In this model, \(\beta_j\) compares category \(j\) with the reference at the same value of \(x\). Exponentiating the coefficient gives the corresponding odds ratio. This is a comparison of odds, not a fixed difference in probabilities.

Result: With reference category 1, the odds ratio for category \(j\) versus category 1 is \(\exp(\beta_j)\), holding the other predictors fixed. For two non-reference categories \(j\) and \(r\), the odds ratio for \(j\) versus \(r\) is \(\exp(\beta_j-\beta_r)\).

The second result follows because the two categories' linear predictors differ by \((\beta_0+\beta_j)-(\beta_0+\beta_r)=\beta_j-\beta_r\). Since the logit is the logarithm of the odds, exponentiating this difference gives the ratio of their odds. The intercept and any shared predictor terms cancel when those predictors are held fixed.

Worked Example: Coding a Four-Category Factor

A hypothetical insurer models whether an application is approved using distribution channel. The four channels are web, phone, agent, and branch. Web is the reference category. The fitted model has an intercept and channel coefficients \(\log(1.5)\) for phone, \(\log(0.8)\) for agent, and \(\log(2)\) for branch. There are no other predictors. Write the coding for the channels and interpret the coefficients.

Method. Use one indicator for each non-reference channel. Define \(I_{\text{phone}}\), \(I_{\text{agent}}\), and \(I_{\text{branch}}\) to equal 1 for the named channel and 0 otherwise. The fitted linear predictor is

$$ \eta=\beta_0+\log(1.5)I_{\text{phone}}+\log(0.8)I_{\text{agent}}+\log(2)I_{\text{branch}}. $$

Step 1: Record the indicator pattern. A web application has \((0,0,0)\); a phone application has \((1,0,0)\); an agent application has \((0,1,0)\); and a branch application has \((0,0,1)\). Exactly one channel is represented for each application.

Step 2: Interpret the phone coefficient. The phone-versus-web odds ratio is \(\exp(\log(1.5))=1.5\). At the same values of any other predictors, if present, phone applications have 1.5 times the odds of approval as web applications.

Step 3: Interpret the agent coefficient. The agent-versus-web odds ratio is \(\exp(\log(0.8))=0.8\). The modeled odds for agent applications are 0.8 times the web odds, or 20% lower.

Step 4: Interpret the branch coefficient. The branch-versus-web odds ratio is \(\exp(\log(2))=2\). The modeled odds are twice the web odds.

Check and common mistake avoided. There are four categories and three indicators, as required with an intercept. The coefficient \(\log(0.8)\) is a log-odds contrast; it is not a decrease of 0.8 in probability. The mistake to avoid is assigning the four channels codes 1, 2, 3, and 4 and treating the codes as a quantitative predictor.

Calculating Fitted Probabilities

Once the factor level is specified, substitute its indicator values into the linear predictor. Convert the resulting log-odds to odds by exponentiating, then convert the odds to a probability. If \(\eta\) is the fitted linear predictor, the probability is

$$ \widehat{p}=\frac{\exp(\eta)}{1+\exp(\eta)} =\frac{1}{1+\exp(-\eta)}. $$

This conversion applies to any fitted logistic linear predictor. A positive \(\eta\) gives fitted odds greater than 1 and probability greater than 0.5; a negative \(\eta\) gives fitted odds less than 1 and probability less than 0.5. The size of a coefficient alone is not a probability change: the fitted probability also depends on the intercept and the other predictor values.

Worked Example: Fitted Probabilities for Three Territories

A hypothetical model predicts whether a claim will require additional documentation. The factor territory has categories central, coastal, and mountain, with central as the reference. The fitted intercept is \(\log(1/9)\), and the coefficients are \(\log(2)\) for coastal and \(\log(4)\) for mountain. Calculate the fitted probability for each territory.

Method. Substitute the appropriate indicator values, calculate \(\eta\), exponentiate to get odds, and use \(p=\text{odds}/(1+\text{odds})\).

Step 1: Central territory. Both indicators are zero, so \(\eta=\log(1/9)\). The fitted odds are \(\exp(\eta)=1/9\). Thus

$$ \widehat{p}_{\text{central}}=\frac{1/9}{1+1/9}=\frac{1}{10}=0.1000. $$

Step 2: Coastal territory. The coastal indicator is 1 and the mountain indicator is 0. Therefore, \(\eta=\log(1/9)+\log(2)=\log(2/9)\), giving odds \(2/9\). Hence

$$ \widehat{p}_{\text{coastal}}=\frac{2/9}{1+2/9}=\frac{2}{11}=0.1818. $$

Step 3: Mountain territory. The mountain indicator is 1 and the coastal indicator is 0. Thus \(\eta=\log(1/9)+\log(4)=\log(4/9)\), giving odds \(4/9\). Hence

$$ \widehat{p}_{\text{mountain}}=\frac{4/9}{1+4/9}=\frac{4}{13}=0.3077. $$

Check and common mistake avoided. The fitted odds increase from \(1/9\) to \(2/9\) to \(4/9\), and the probabilities increase from 0.1000 to 0.1818 to 0.3077. The probabilities are not obtained by multiplying the reference probability by the odds ratios: odds and probabilities are different quantities. Probabilities are rounded to four decimal places.

Comparing Two Non-Reference Categories

The reference category is a convenient baseline, not the only comparison of interest. To compare two other levels, subtract their coefficients before exponentiating. The resulting odds ratio compares the categories directly while holding all other predictors fixed. A negative coefficient difference means the numerator category has lower odds; reversing the comparison changes the sign of the difference and takes the reciprocal of the odds ratio.

Worked Example: A Direct Contrast Between Two Levels

A hypothetical model predicts whether a commercial vehicle passes an inspection. The vehicle's service tier is basic, standard, or premium, with basic as the reference. The fitted coefficients are \(\beta_{\text{standard}}=\log(1.8)\) and \(\beta_{\text{premium}}=\log(3.6)\). Calculate the premium-versus-standard odds ratio. If a standard-tier vehicle has fitted odds of 0.25 at the specified predictor values, calculate the premium-tier fitted probability at those same values.

Method. Use a coefficient difference for the direct odds comparison, then use the given standard-tier odds to obtain premium-tier odds and probability.

Step 1: Form the coefficient contrast.

$$ \beta_{\text{premium}}-\beta_{\text{standard}} =\log(3.6)-\log(1.8) =\log(3.6/1.8)=\log(2). $$

Step 2: Exponentiate. The premium-versus-standard odds ratio is \(\exp(\log(2))=2\). Premium-tier odds are twice standard-tier odds when the other predictors are held fixed.

Step 3: Calculate premium-tier odds. The standard-tier odds are 0.25, so the premium-tier odds are \(2(0.25)=0.50\).

Step 4: Convert odds to probability.

$$ \widehat{p}_{\text{premium}}=\frac{0.50}{1+0.50}=\frac{1}{3}=0.3333. $$

Check and common mistake avoided. The direct contrast is the ratio \(3.6/1.8\), not \(3.6-1.8\). The given 0.25 is odds, not a probability; treating it as a probability and multiplying by 2 would not produce the fitted probability. The final probability is rounded to four decimal places.

Choosing a Reference and Avoiding Coding Errors

Changing the reference category changes the intercept and the displayed factor coefficients, but it does not change the fitted linear predictor for any observation when the model is represented equivalently. With \(K\) categories and an intercept, using all \(K\) category indicators would create exact redundancy: for every observation, the indicators sum to 1, which duplicates the intercept column. The model coefficients would not be uniquely determined. Use \(K-1\) indicators, or another identifiable parameterization, rather than an intercept together with all \(K\) indicators.

Reference selection is often guided by interpretability: a common, natural, or policy-relevant category may be a useful baseline. It does not make the reference category intrinsically better, and it does not determine which category has the highest fitted probability. Those conclusions depend on the fitted model and the other predictor values.

  • Count the indicators. For \(K\) factor levels in a model with an intercept, use \(K-1\) reference-cell indicators.
  • State the reference. A coefficient's meaning cannot be fully described without naming the category it is compared with.
  • Use the correct contrast. A coefficient compares its level with the reference; a comparison of two non-reference levels uses the difference between their coefficients.
  • Keep odds and probabilities distinct. Exponentiating a coefficient gives an odds ratio. Use the logistic conversion to calculate a probability.
  • Hold other predictors fixed. Category odds ratios describe the model comparison at the same values of any other included predictors.
Key takeaway: A multi-level factor is represented by indicators, not by arbitrary category numbers. With a stated reference category, each factor coefficient is a log-odds contrast; exponentiate it for an odds ratio, and apply the logistic conversion to obtain a fitted probability.

Check Your Understanding

For each question, identify the relevant indicator or contrast before calculating.

  1. A logistic model has an intercept and a factor with five categories. One category is the reference. How many indicators are needed, and what indicator pattern represents the reference category? (Answer: Four indicators; all four equal 0 for the reference.)
  2. A factor's reference category has fitted odds 0.25. The coefficient for the digital category is \(\log(3)\). Find the digital category's odds and fitted probability. (Answer: Odds \(0.75\); probability \(3/7=0.4286\).)
  3. A three-level factor has reference category A, intercept \(-1\), and coefficient \(-0.2\) for category C. Calculate the fitted probability for C, with no other predictors. (Answer: \(\eta=-1.2\), so \(\widehat{p}=1/(1+\exp(1.2))=0.2315\).)
  4. With category A as reference, the coefficients for B and C are \(\log(1.4)\) and \(\log(2.1)\). What is the odds ratio for C versus B? (Answer: \(\exp(\log(2.1)-\log(1.4))=1.5\).)
  5. A model has reference category A, intercept \(\log(0.25)\), coefficient \(\log(2)\) for category B, and continuous-predictor coefficient \(0.3\). For a B observation with predictor value \(x=2\), calculate the fitted probability. (Answer: \(\eta=\log(0.25)+\log(2)+0.6\), odds \(=0.5\exp(0.6)=0.9111\), and \(\widehat{p}=0.4767\).)

Previous

Next

Back to Tutorials
Clinical Biostats