Why Cluster Labels Need Careful Coding
A clustering procedure assigns each record to a group, but the resulting labels—such as 1, 2, and 3—are identifiers, not measurements. If those labels enter a regression as ordinary numbers, the model treats the difference between clusters 1 and 2 as the same kind of change as the difference between clusters 2 and 3. That ordering and spacing are generally arbitrary. To use cluster membership as a predictor, represent it as a categorical variable.
A categorical cluster predictor is commonly represented by reference-category coding. The model compares each non-reference cluster with one chosen reference cluster. This makes the coefficients interpretable while retaining the flexibility to estimate a separate effect for every cluster. The ideas apply to linear models and, with interpretation on the appropriate scale, to generalized linear models (GLMs). This material is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
What the Coefficients Mean
For a response with conditional mean \(\mu=E[Y\mid\text{predictors}]\), a GLM relates the mean to a linear predictor \(\eta\) through a link function \(g\), so that \(g(\mu)=\eta\). With cluster membership as the only predictor and cluster \(r\) as reference, the model is
For the reference cluster, \(\eta=\beta_0\). For a non-reference cluster \(j\), \(\eta=\beta_0+\beta_j\). Thus \(\beta_j\) is the difference in the linear predictor between cluster \(j\) and the reference cluster. With an identity link, this is a difference in conditional means. With a log link, it is a difference in log means, and exponentiating it gives a mean ratio. With a logit link, it is a difference in log odds, and exponentiating it gives an odds ratio.
The coefficient is not automatically a difference in predicted responses. For a nonlinear link, the response-scale difference depends on the starting value. For example, the same log-odds contrast can correspond to different probability differences at different baseline probabilities.
Worked Example: Reference Coding for Cluster Means
A hypothetical portfolio is divided into clusters A, B, and C using features that do not include the response. In a sample, the observed response values are A: 8, 12, 16; B: 18, 22, 26; and C: 6, 10, 14. Fit the cluster-only linear model using A as the reference, and interpret its coefficients.
Method. Use an identity-link model with an intercept and indicators for B and C. With a separate fitted mean for each cluster, the fitted value in each cluster is its sample mean.
Step 1: Calculate each cluster mean. For A, the sum is \(8+12+16=36\), so \(\bar{Y}_A=36/3=12\). For B, the sum is \(18+22+26=66\), so \(\bar{Y}_B=66/3=22\). For C, the sum is \(6+10+14=30\), so \(\bar{Y}_C=30/3=10\).
Step 2: Set the intercept. A is the reference cluster, so its indicators are both zero. Its fitted mean is therefore \(\beta_0=12\).
Step 3: Calculate the indicator coefficients. For B, \(\beta_B=\bar{Y}_B-\bar{Y}_A=22-12=10\). For C, \(\beta_C=\bar{Y}_C-\bar{Y}_A=10-12=-2\). The fitted model is \(\widehat{Y}=12+10I_B-2I_C\).
Step 4: Check the fitted values. In A, both indicators are zero and the prediction is 12. In B, the prediction is \(12+10=22\). In C, it is \(12-2=10\). These equal the three calculated sample means. The common mistake avoided here is treating the coefficients 10 and \(-2\) as cluster means; they are contrasts with A.
Why the Coding Gives These Contrasts
The coefficient interpretation follows directly from substituting the indicator values. In the reference cluster, every non-reference indicator equals zero, so its linear predictor is \(\beta_0\). In cluster \(j\), \(I_j=1\) and every other indicator is zero, so its linear predictor is \(\beta_0+\beta_j\). Subtracting the reference predictor from the cluster-\(j\) predictor gives \(\beta_j\).
For an identity-link model containing only cluster indicators, the fitted mean in each cluster is the sample mean. To see why, let \(\eta_k\) be the fitted value shared by observations in cluster \(k\). The sum of squared errors for that cluster is \(\sum_{i:C_i=k}(y_i-\eta_k)^2\). Differentiating with respect to \(\eta_k\) and setting the result to zero gives
Here \(n_k\) is the number of observations in cluster \(k\). Since the reference-coded model can assign a separate fitted value to each cluster, it reaches each cluster’s sample mean. This result assumes an identity-link least-squares model with an intercept, a complete cluster indicator structure, and no additional restrictions tying cluster effects together. In a GLM with a nonlinear link, the coefficient contrasts are still on the linear-predictor scale, but the identity-link sample-mean calculation does not apply.
Changing the Reference Cluster
The reference cluster determines the coefficient representation, not the underlying set of fitted values. In the first example, choosing C instead of A makes the intercept 10. The coefficient for A becomes \(12-10=2\), and the coefficient for B becomes \(22-10=12\). The predictions remain 12, 22, and 10 for A, B, and C, respectively.
More generally, if the cluster-specific linear predictors are \(\eta_1,\ldots,\eta_K\), then reference cluster \(r\) gives intercept \(\eta_r\) and coefficients \(\beta_j=\eta_j-\eta_r\). Choosing a different reference changes those contrasts, but reconstructs the same \(\eta_k\) for every cluster. Consequently, comparisons of coefficients across two differently coded fits should account for their different reference categories.
Worked Example: Interpreting a Logistic Cluster Effect
In a hypothetical study, a binary outcome occurs for 25 of 100 records in cluster A, 50 of 100 in cluster B, and 75 of 100 in cluster C. Fit a cluster-only logistic model using A as reference and interpret the coefficients. Assume independent Bernoulli outcomes and a logit link.
Step 1: Calculate the observed proportions and odds. The proportions are \(p_A=25/100=0.25\), \(p_B=50/100=0.50\), and \(p_C=75/100=0.75\). Their odds are \(0.25/0.75=1/3\), \(0.50/0.50=1\), and \(0.75/0.25=3\).
Step 2: Calculate the intercept. The reference-cluster log odds are \(\log(1/3)=-\log(3)\), so \(\beta_0=-\log(3)\), approximately \(-1.0986\).
Step 3: Calculate the contrasts. For B, \(\beta_B=\log(1)-\log(1/3)=\log(3)\), approximately \(1.0986\). For C, \(\beta_C=\log(3)-\log(1/3)=\log(9)\), approximately \(2.1972\). The odds ratios relative to A are therefore \(e^{\beta_B}=3\) and \(e^{\beta_C}=9\).
Step 4: Check the predictions. A has odds \(e^{-\log(3)}=1/3\), which gives probability \((1/3)/(1+1/3)=0.25\). B has linear predictor zero, giving probability \(1/(1+1)=0.50\). C has linear predictor \(-\log(3)+\log(9)=\log(3)\), giving odds 3 and probability \(3/(1+3)=0.75\). The common mistake avoided is interpreting a log-odds coefficient of \(\log(3)\) as a 1.0986 increase in probability; it represents a threefold change in odds relative to A.
Why Cluster IDs Are Not Quantitative Predictors
If cluster IDs are entered as a single quantitative variable, a linear model assumes a specific pattern: moving one unit upward in the code changes the linear predictor by a constant amount. A model using the numeric ID \(x\) has the form \(\eta=\alpha+\gamma x\). This is a one-dimensional trend model, not a general cluster-effect model. It can produce predictions that depend on arbitrary label choices and cannot represent every pattern of cluster means.
Worked Example: A Numeric Label Creates a False Trend
Suppose three clusters have numeric IDs 4, 8, and 12. A hypothetical response is 50 for each of two observations in cluster 4, 90 for each of two observations in cluster 8, and 30 for each of two observations in cluster 12. Compare the correct factor model with a least-squares regression on the numeric IDs.
Step 1: Calculate the overall means. The mean ID is \(\bar{x}=(2(4)+2(8)+2(12))/6=48/6=8\). The mean response is \(\bar{y}=(2(50)+2(90)+2(30))/6=340/6=170/3\), or approximately 56.6667.
Step 2: Calculate the numeric regression slope. The numerator \(\sum_i(x_i-\bar{x})(y_i-\bar{y})\), grouped by cluster, is
The denominator is \(\sum_i(x_i-\bar{x})^2=2(4-8)^2+2(8-8)^2+2(12-8)^2=32+0+32=64\). Thus the slope is \(-160/64=-2.5\). The intercept is \(\bar{y}-(-2.5)\bar{x}=170/3+20=230/3\), approximately 76.6667.
Step 3: Compare predictions. The numeric model predicts \(230/3-2.5(4)=200/3\), or approximately 66.6667, for ID 4; \(230/3-2.5(8)=170/3\), or approximately 56.6667, for ID 8; and \(230/3-2.5(12)=140/3\), or approximately 46.6667, for ID 12. By contrast, reference-coded cluster indicators with an intercept reproduce the cluster means 50, 90, and 30. The numeric model imposes a decreasing linear trend that does not describe those means.
Check. The numeric predictions average to \(56.6667\), the overall response mean, as an ordinary least-squares fit with an intercept must. Their slope calculation uses the centered cross-products, whose exact total is \(-160\), not \(-640\). The common mistake is to infer that numeric IDs have meaningful spacing or order; the codes could be permuted without changing cluster membership, but such a permutation can change the numeric regression.
Redundancy and Practical Coding Checks
With an intercept, including all \(K\) cluster indicators creates exact redundancy: for every record, the indicators sum to one. Thus the intercept column equals the sum of the \(K\) indicator columns. The model matrix is rank deficient, so the individual intercept and cluster coefficients cannot all be uniquely estimated. Use \(K-1\) indicators with a reference category, or use another valid contrast system that avoids this redundancy.
In R, store cluster membership as a factor rather than as a numeric vector of IDs, then check the factor’s levels and reference level before fitting. Treatment contrasts commonly use one level as the reference and estimate differences from it. The exact printed coefficients therefore depend on the level ordering and contrast settings. Verify both the reference category and the coding used; do not assume a desired reference level was selected automatically.
| Representation | Model implication | Appropriate interpretation |
|---|---|---|
| Factor with reference coding | One contrast for each non-reference cluster | Difference from the reference on the linear-predictor scale |
| Numeric cluster ID | One linear trend across the assigned numbers | Meaningful only if order and spacing have substantive justification |
| All cluster indicators plus intercept | Exact linear dependence among columns | Redundant parameterization; remove a column or change contrasts |
Common Exam Traps
- Calling a contrast a cluster mean. In a reference-coded identity-link model, the intercept is the reference mean and a coefficient is a difference from that mean.
- Interpreting a GLM coefficient on the response scale. First identify the link. Logit coefficients are log-odds contrasts; log-link coefficients are log-mean contrasts.
- Using numeric IDs because they look ordered. A cluster label is not a measured quantity. Numeric regression introduces a trend assumption that reference-coded factors do not require.
- Including all indicators with an intercept. The resulting columns are linearly dependent; use \(K-1\) indicators or a different full-rank contrast scheme.
- Comparing coefficients after changing the reference. The fitted cluster predictions can remain identical even though the intercept and contrasts change.
Check Your Understanding
For each problem, identify what the coding represents before calculating or interpreting its coefficients.
- A cluster-only identity-link model has observed means A = 14, B = 19, and C = 11. With B as reference, state the intercept and the coefficients for A and C. (Answer: Intercept \(19\); coefficient for A \(-5\); coefficient for C \(-8\).)
- A logistic model has fitted probabilities 0.25 in A and 0.50 in B, with A as reference. Calculate the B coefficient and its odds-ratio interpretation. (Answer: \(\log(3)\approx1.0986\); the odds in B are three times the odds in A.)
- A model has four mutually exclusive cluster levels and an intercept. How many reference-coded indicators are needed, and what happens if all four indicators are included with the intercept? (Answer: Three indicators; all four with the intercept create exact linear dependence because the indicators sum to one.)
- Three equally sized clusters have response means 20, 40, and 10 and numeric IDs 1, 2, and 3, respectively. For regression on the numeric ID, calculate the slope using the three cluster means with equal weights, and explain why it is not a general cluster-effect model. (Answer: Slope \(-5\); it imposes a constant linear trend across the numeric codes rather than estimating three unrestricted cluster means.)
- A factor-only model has cluster linear predictors \(\eta_A=-2\), \(\eta_B=1\), and \(\eta_C=4\). If C is the reference under reference coding, give the intercept and the A and B coefficients. (Answer: Intercept \(4\); coefficient for A \(-6\); coefficient for B \(-3\).)