Why Territory Categories Need Careful Harmonization
A territory field may look like a single categorical variable while containing several different data-quality and modeling problems. “North,” “NORTH,” and “N.” may describe the same territory; a factor may retain a territory that is no longer present in the data; and a genuinely distinct territory may have too few training records to support a stable estimate. These problems require different decisions. Standardizing spelling is not the same as pooling rare categories, and neither is the same as choosing a reference category for model encoding.
The objective is to create a consistent, auditable mapping from recorded labels to model categories. Apply that mapping using rules learned or specified without looking at scoring outcomes. Then preserve the model’s category definitions when preparing new records. This makes the encoded inputs meaningful and prevents a scoring row from silently being interpreted differently than a training row.
A Deliberate Sequence of Decisions
Use a fixed order. First define what the labels mean and standardize spelling, spacing, case, and known abbreviations. Next inspect the resulting category counts in the training data. Remove levels with no training observations; decide separately whether low-count, genuinely distinct territories should remain separate or be pooled. Finally, set the factor levels and reference category explicitly, and retain the mapping for scoring.
Use documented rules, such as trimming extra spaces and mapping approved abbreviations to a canonical territory. Do not merge labels just because their spellings look similar if they represent different places.
After harmonization, tabulate the categories using training data only. A frequency threshold must be specified before pooling, and the unit counted must match the modeling table.
Remove unused levels from the training factor. Pool rare levels only when the combined category is substantively defensible and supported by an explicit rule.
Specify the final factor levels, the reference level, the mapping to any pooled category, and what to do with an unrecognized label at scoring time.
In R, functions such as `trimws()`, `toupper()`, `factor()`, and `droplevels()` can support these operations. They do not decide which strings are substantively equivalent or whether a low-count territory should be pooled. Those choices must be made and documented as part of the data-preparation rules. After cleaning, verify the result with a frequency table; do not assume a transformation produced the intended categories.
Worked Example: Standardizing Labels Before Counting Rare Territories
A hypothetical insurer has 12 training policy records. Their recorded territory labels are “North” four times, “NORTH” twice, “N.” once, “South” twice, “S.” once, “Coastal” once, and “Rural” once. A documented business dictionary says that “N.” means North and “S.” means South. The modeling rule is to pool any harmonized territory with fewer than two training records into “Other.” Find the final counts and factor levels.
Method. Apply the documented label mapping first, then count, then apply the rare-level rule. The seven North spellings combine to \(4+2+1=7\) records. South has \(2+1=3\) records. Coastal and Rural each have one record.
The rare-level rule is “count fewer than two,” so the two one-record categories are pooled. The resulting counts are North: 7, South: 3, and Other: \(1+1=2\). They sum to \(7+3+2=12\), matching the original number of records.
For a model with North as the intended reference, define the ordered factor levels as North, South, Other. The order is set explicitly rather than left to an automatic alphabetical default. The pooled category is not a spelling correction: Coastal and Rural remain distinct in the source mapping but both map to Other under the stated modeling rule.
Common mistake avoided. Pooling “N.” into Other because it appears only once would confuse a spelling variant with a genuinely distinct territory. The dictionary establishes that “N.” means North, so it must be harmonized before the rare-category count is applied.
Unused Levels, Rare Levels, and Reference Categories
An unused factor level contributes no observations and therefore cannot supply information to estimate a category effect. In R, subsetting a factor does not necessarily remove its unused levels. Applying `droplevels()` after selecting the training records removes levels that are absent from those records. This is a cleanup step, not a rare-level policy: it does not combine two categories that each have observations.
A rare level has at least one observation but few enough that its separate estimate may be unstable or unsupported for the intended use. There is no universal cutoff. The choice depends on the modeling purpose, the number of records, the outcome, and whether categories can be combined sensibly. If pooling is appropriate, set the rule in advance and learn which levels are rare from the training portion. Use the saved mapping to process scoring records; do not recalculate rarity using their outcomes or their full-population distribution.
For a categorical predictor with \(K\) levels, treatment coding represents each non-reference level with an indicator. Under a model with an intercept, one level is omitted to avoid redundant columns. For example, with levels North, South, and Other and North as the reference, the model matrix columns for the factor are indicators for South and Other. North has zeros in both columns.
This representation follows directly from the indicator values. For a reference-category record, every indicator in the sum is zero, so \(\eta=\beta_0\). For a record in category \(j\), its own indicator is one and the others are zero, so \(\eta=\beta_0+\beta_j\). Thus, changing the reference changes coefficient values and interpretation, but not the fitted linear predictor when the same categories and model are represented correctly. In a generalized linear model, the coefficient contrast is on the link scale; it is not necessarily a direct difference in predicted dollars or probabilities.
Worked Example: Pooling Rare Training Levels for Scoring
A hypothetical training set contains 50 North records, 20 South records, two Coastal records, and one Island record. Before examining outcomes, the analyst sets a rule that any territory with fewer than three training records is pooled into Other. A separate scoring file contains three Coastal records, two Metro records, and one North record. Metro did not occur in training. Define the training and scoring encodings under a documented policy that sends unrecognized scoring labels to Other.
Method. Estimate the rare-level mapping using training counts only. North and South remain separate because their counts, 50 and 20, are at least three. Coastal and Island are pooled because their counts, two and one, are each below three. Therefore the training categories are North, South, and Other, with counts 50, 20, and \(2+1=3\). The total is \(50+20+3=73\), equal to the training total.
Save the mapping “Coastal to Other” and “Island to Other.” Apply it to scoring data: the three Coastal records become Other. Metro has no training mapping, so the stated scoring policy sends its two records to Other as well. North remains North. The scoring counts after mapping are Other: \(3+2=5\), North: 1, and South: 0. The absence of South in this scoring file does not change the trained model’s category definitions.
Set the factor levels for both data sets to North, South, Other, in that order, with North as the reference. Under treatment coding, a North record has indicator values (South = 0, Other = 0), a South record has (1, 0), and an Other record has (0, 1). The scoring data may contain no South observations, but South remains a valid trained level so that its model column and coefficient retain their intended meaning.
Common mistake avoided. Recomputing the rare-level rule in the scoring file would make the three Coastal scoring records appear common and could keep them separate, despite their having been pooled in training. The trained mapping, not the scoring-file count, determines the encoding.
Keep Training and Scoring Encodings Compatible
Compatibility means more than having columns with similar names. A scoring category must map to the same substantive model category as it did in training, and each encoded indicator must refer to the same category. Save the canonical labels, pooled-level mapping, factor-level order, reference level, and unknown-label policy alongside the model. Apply the same operations to new records and check the resulting labels and counts before prediction.
A useful validation is to inspect the model matrix structure on both data sets. With North, South, and Other levels and North as reference, the category columns should represent South and Other in that order. A missing South observation in a scoring batch is not a reason to change the model matrix definition. Conversely, a genuinely new label needs a defined disposition: map it to a substantively appropriate existing group, map it to a deliberately trained Other group, or flag the record for review rather than silently treating it as a known territory.
A scoring label mapped to Other is only supported by the model if Other was represented in training and its coefficient could be estimated. Creating an Other factor level only at scoring time does not create a trained effect. If there was no trained Other category, an unknown label cannot automatically receive an Other coefficient; the operational policy must provide a supported fallback or stop and request review. That decision should be made before deployment, not improvised when a prediction fails.
Worked Example: Changing the Reference Without Changing Predictions
Consider a hypothetical categorical-only model for annual claim cost, using an identity link and North as the reference. Suppose the fitted category means are North: $100, South: $130, and Other: $160. Show the treatment-coded coefficients, then recode the same fitted model with South as the reference and verify the predictions.
Method. In a categorical-only model with an intercept and identity link, each category’s fitted mean is its linear predictor. With North as reference, the intercept is the North mean, $100. The South coefficient is \(130-100=30\), and the Other coefficient is \(160-100=60\). The fitted means are therefore North: \(100\), South: \(100+30=130\), and Other: \(100+60=160\).
With South as reference, the intercept becomes the South mean, $130. The North coefficient is \(100-130=-30\), and the Other coefficient is \(160-130=30\). Predictions are now North: \(130+(-30)=100\), South: \(130\), and Other: \(130+30=160\). Each fitted mean matches its value under the original reference coding.
The calculation verifies the general reason that a reference change alters parameterization rather than fitted category values. It relies on keeping the same categories and fitted model structure; changing the pooling rule or category mapping would be a different model. The identity-link example also makes the coefficients dollar differences. With another link, coefficients describe contrasts on that link’s scale.
Common mistake avoided. A reference category is not automatically a “zero-risk” or “average-risk” group. Its indicators are zero by coding convention, and its fitted value is represented by the intercept in this categorical-only example. The category’s actual fitted value need not be zero.
Common Traps and a Practical Audit
- Using a global label replacement without checking meaning. Similar strings may refer to different territories. Apply only documented equivalences and inspect labels that remain unmapped.
- Confusing unused and rare levels. An unused level has zero training observations; a rare level has a positive count below a specified threshold. Removing unused levels does not decide whether to pool rare ones.
- Defining rarity on all records before splitting. This lets information from the scoring or validation portion influence feature construction. Set the mapping from training data and carry it forward.
- Relying on implicit factor order. Alphabetical order or an automatically selected baseline may not match the intended reference. Set the ordered levels and reference explicitly.
- Dropping a category just because it is absent in one scoring batch. The trained model’s levels should remain fixed; a batch with no examples of a category does not change the model.
- Assuming an unknown label is safely handled by a factor. An unrecognized level may cause a prediction error or be encoded incorrectly if preprocessing is inconsistent. Specify and test the unknown-label policy.
Before scoring, confirm that each recorded label has a documented disposition, that rare-level decisions came from training data, that the reference is explicit, and that the model’s trained categories are available in the scoring encoding. This audit is particularly important when the category mapping is reused across renewal periods or batches: a stable encoding supports comparison, while an undocumented change can make apparent model changes into data-preparation changes.
Check Your Understanding
For each problem, apply the stated mapping and counting rules in order. Answers use the specified categories and encoding conventions.
- A training file has “East” four times, “E.” once, “West” twice, and “W.” once. A dictionary maps E. to East and W. to West. No rare pooling is required. What are the harmonized counts? (Answer: East 5; West 3.)
- Training counts are Central: 8, Lake: 2, and Hill: 1. Pool categories with fewer than three training records into Other. What are the final training counts? (Answer: Central 8; Other 3.)
- A model has levels A, B, and Other, with A as reference. What indicator pair (B, Other) is assigned to a record in Other under treatment coding? (Answer: (0, 1).)
- Training counts are Harbor: 12, Plains: 4, and Ridge: 2. The pooling threshold is fewer than three records. A scoring file has Ridge: 5 and a new label, Valley: 2. Under the policy that unseen labels map to a trained Other category, what scoring counts result after mapping? (Answer: Harbor 0; Plains 0; Other 7; the five Ridge and two Valley records are mapped to Other.)
- A categorical-only identity-link model has fitted means A: $80, B: $110, and Other: $140. If B becomes the reference, what is the intercept and what are the coefficients for A and Other? (Answer: Intercept $110; A coefficient −$30; Other coefficient $30.)