Why New Territory Levels Need a Scoring Rule
A model may be fitted using policies from territories A, B, and C, then receive a policy from territory D at scoring time. The model has no fitted territory-D effect: no observations from D contributed information about that category. A scoring system still needs a defined response. Depending on the software and model, prediction may fail, produce a missing value, or use an encoding that was not intended.
The solution is not to let the scoring software decide implicitly. Specify how an unrecognized territory will be encoded, apply the same rule to every scoring batch, and inspect the resulting predictions. This is an operational modeling decision as well as a data-preparation decision. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
What a Fitted Model Knows About a Factor
Suppose a Poisson generalized linear model predicts claim counts \(N_j\) for policy \(j\). Assume that, conditional on the predictors, \(N_j\) has a Poisson distribution with mean \(\lambda_j\), and that the model uses a log link:
Here \(e_j\) is the policy’s exposure, \(T_j\) is its territory, and the indicator \(I(T_j=k)\) equals 1 when the policy is in territory \(k\), and 0 otherwise. The exposure term is an offset with coefficient fixed at 1. With treatment coding, one observed territory is the reference: its indicator is omitted, and its effect is included in the intercept. This model assumes that the specified log-linear mean structure is appropriate; the Poisson distribution also assumes that the conditional mean and variance of the count are equal.
A coefficient for a factor level is estimated from observations in that level, relative to the reference level. If territory D never appeared in fitting data, there is no data-based estimate of a D coefficient. In the displayed model, D also has no defined indicator column. Any prediction for D therefore requires an explicit convention, such as mapping D to the reference territory or to a trained pooled category called OTHER.
Two Reproducible Strategies
A direct fallback maps each previously unseen territory to a designated, observed reference level. For treatment-coded GLMs, the new policy then receives the reference-level contribution for territory. This is simple, but it assumes that the reference territory is a defensible stand-in for the unobserved category.
An alternative is to create an OTHER category during model development. This generally means grouping selected low-volume or similar territories into OTHER in the fitting data, then estimating its effect as a category. At scoring time, new territories are mapped to that trained category. OTHER is useful only if it was actually represented in training and its fitted effect is estimable. Creating an OTHER label after fitting does not supply the missing coefficient.
| Scoring rule | What the model uses | Main condition or limitation |
|---|---|---|
| Reference fallback | The reference category’s encoded contribution | State why this proxy is acceptable; it is not an estimate for the new territory. |
| Trained OTHER category | The fitted coefficient or model branch for OTHER | OTHER must have been included and represented during fitting. |
| Reject and investigate | No prediction is returned until the input is resolved | Appropriate when an unseen territory cannot safely be treated as a proxy. |
Whichever rule is selected, it belongs in the scoring specification alongside the fitted model. Preserve the training level list, the reference category, the mapping rule, and the order and names of encoded columns. Map unknown values before converting them to a factor with the training levels. In R, converting an unrecognized value directly to a factor restricted to known levels can turn it into a missing value. Some prediction functions also reject new factor levels. Check the behavior of the complete fitting and scoring pipeline rather than assuming it will silently choose a sensible fallback.
Record training territory levels, the reference level, encoding method, and any trained OTHER category.
For each incoming territory, retain a known level or apply the documented fallback. Do not assign a coefficient based on a category’s spelling or numeric code.
Apply the training contrasts and ensure that encoded columns match the fitted model’s required names and order.
Count mapped policies, confirm there are no unintended missing values, and compare predictions under the chosen rule with a stated alternative when the effect is material.
Why the Mapping Changes the Prediction
For the Poisson log-link model, let the fitted reference-territory rate per exposure unit be \(r_0=\exp(\beta_0)\). If a known territory has coefficient \(\beta_k\), its rate is multiplied by \(\exp(\beta_k)\) relative to the reference. This follows directly by exponentiating the model equation. For equal exposure \(e\), the two means are \(e\exp(\beta_0)\) and \(e\exp(\beta_0+\beta_k)\), so their ratio is:
The exposure cancels, leaving the territory’s rate multiplier. For an unseen level, there is no fitted \(\beta_k\) to insert in this calculation. Reference fallback assigns the reference contribution; mapping to OTHER assigns the OTHER contribution. Neither operation estimates a new effect from absent data.
Worked Example: Reference Fallback in a Frequency GLM
Method. Apply treatment coding and map an unseen territory to the stated reference level before computing the fitted Poisson mean. Suppose a hypothetical model has reference territory Cedar, fitted annual claim rate \(r_0=0.08\), and a known territory Pine with fitted coefficient \(\beta_{\text{Pine}}=\log(1.5)\). A policy from the previously unseen territory Ridge has exposure \(e=0.5\). Under the model, the conditional claim count is Poisson with mean equal to exposure times the fitted annual rate. The operational rule maps Ridge to Cedar.
Step 1: Determine the encoded territory contribution. Ridge is not Cedar or Pine and has no fitted coefficient. Under the specified rule it receives Cedar’s reference contribution, so no Pine multiplier is applied.
Step 2: Compute the reference mean. The model’s reference rate is \(0.08\) claims per exposure unit. Thus \(\lambda=0.5(0.08)=0.040\) expected claims.
Step 3: Check the resulting prediction. The output is \(0.040\) expected claims. If the policy had been in Pine, its rate would be \(0.08\exp(\log(1.5))=0.08(1.5)=0.12\), and its expected count at exposure \(0.5\) would be \(0.5(0.12)=0.060\). The fallback therefore uses the Cedar rate, not the Pine rate.
Common mistake avoided. Do not apply a known territory’s multiplier merely because it is the closest available code or happens to be adjacent in a list. The specified mapping—not the spelling or position of a level—determines the model input.
Worked Example: Mapping to an OTHER Category Trained in Advance
Method. Use an OTHER coefficient only when OTHER was a category in the fitted model. Suppose a hypothetical Poisson frequency model has reference territory Mesa with annual rate \(0.04\), and a trained OTHER category with coefficient \(\beta_{\text{OTHER}}=\log(1.5)\). OTHER was formed during fitting by grouping selected low-volume territories. A new territory, Summit, is assigned to OTHER by the documented scoring rule. For a Summit policy with exposure \(e=2\), the conditional count is Poisson with the model’s fitted mean.
Step 1: Find the OTHER rate multiplier. \(\exp(\beta_{\text{OTHER}})=\exp(\log(1.5))=1.5\).
Step 2: Find the fitted annual rate. The OTHER rate is \(0.04(1.5)=0.06\) expected claims per exposure unit.
Step 3: Apply exposure. The predicted Poisson mean is \(\lambda=2(0.06)=0.12\) expected claims.
Step 4: If needed, convert the mean to a probability. For a Poisson random variable with mean \(0.12\), the probability of at least one claim is \(1-P(N=0)=1-\exp(-0.12)\). Since \(\exp(-0.12)\approx0.886920\), the probability is \(1-0.886920=0.113080\), or 0.1131 to four decimal places. The expected count and the probability of at least one claim are different quantities.
Check. The mapped rate \(0.06\) exceeds the reference rate \(0.04\) by the fitted factor 1.5, and multiplying by exposure 2 gives the mean 0.12. The calculation uses the fitted OTHER coefficient; it would not be valid if OTHER were created only after fitting.
Worked Example: Checking the Encoded Inputs Before Scoring
Method. Inspect the model-matrix row produced by an explicit reference fallback. Suppose a hypothetical Poisson model was fitted with treatment coding, Cedar as reference, and Pine as the only nonreference territory. Its fitted equation is \(\log(\lambda)=\log(e)+\log(0.05)+\log(2)I(T=\text{Pine})\). A scoring policy has territory Ridge and exposure \(e=1.2\). Ridge is mapped to Cedar before factor conversion. Conditional on the inputs, the count is Poisson with mean \(\lambda\).
Step 1: Write the expected model-matrix values. For treatment coding, the intercept column is 1 and the Pine indicator is 0 for Cedar. After mapping Ridge to Cedar, the row is therefore \((1,0)\) for those two columns.
Step 2: Substitute the row and exposure. The Pine contribution is \(\log(2)(0)=0\). Thus \(\log(\lambda)=\log(1.2)+\log(0.05)\), and \(\lambda=\exp(\log(1.2)+\log(0.05))=1.2(0.05)=0.060\).
Step 3: Verify the expected score. A Pine policy at the same exposure would have mean \(1.2(0.05)(2)=0.120\). The Ridge score of \(0.060\) is consistent with the explicitly chosen Cedar fallback and the model-matrix row. This check also confirms that exposure is included once, through its offset.
Common mistake avoided. Do not rely on an input factor’s internal integer codes as model-matrix values. Factor codes identify labels; treatment contrasts determine the actual indicator columns. Rebuild those columns using the fitted schema.
Common Mistakes and Practical Checks
A frequent trap is to treat factor labels as numbers. Suppose the fitted reference rate is \(0.04\), and a known territory has a relative rate of \(1.5\). If a new territory happens to receive integer code 3, the incorrect calculation \(0.04(1.5)^3=0.135\) invents a third power of a fitted effect. The code 3 is a label index, not a model coefficient or risk multiplier. Under reference fallback the rate remains \(0.04\); under a trained OTHER rule it is the fitted OTHER rate. The correct value depends on the declared mapping, never on the arbitrary code.
Another mistake is to create an OTHER level only in the scoring data and assume the model will estimate its effect. A category with no training observations has no fitted coefficient. A safer scoring design also checks that:
- Every mapping rule is applied identically to individual and batch scoring.
- Unknown-level counts and resulting mapped categories are logged for review.
- The model matrix has the expected columns, with no unexpected missing values or all-zero rows caused by failed conversion.
- Prediction outputs are compared with a plausible fallback or alternative mapping, especially when many policies are affected.
- The saved model and its mapping recipe are versioned together, so a later change to factor levels cannot silently change predictions.
A large difference between fallback choices signals material dependence on an unobserved-category assumption; it does not identify which choice is correct. If no reasonable proxy exists, rejecting the score for investigation may be more defensible than issuing a seemingly precise prediction. The prediction report should state how new levels were handled and how often the rule was used.
Check Your Understanding
For each problem, state the scoring rule or show the calculation before checking the answer.
- A Poisson frequency GLM has a reference rate of \(0.06\) per exposure unit. A new territory is mapped to the reference, and the policy exposure is 3. What is the predicted mean claim count? (Answer: \(3(0.06)=0.18\).)
- A fitted Poisson model has reference rate \(0.05\) and a trained OTHER coefficient \(\log(1.4)\). An unseen territory maps to OTHER, and exposure is \(0.5\). Find the predicted mean count. (Answer: \(0.5(0.05)(1.4)=0.035\).)
- A treatment-coded model uses Cedar as reference and a Pine indicator. An unseen territory is mapped to Cedar. What are the intercept and Pine-indicator values in its model-matrix row? (Answer: \((1,0)\).)
- A new factor value is converted directly to a factor whose allowed levels are only those in the training data, and the result becomes missing. What should the scoring pipeline do before factor conversion? (Answer: Apply the documented unknown-level mapping first, then convert using the saved training levels and verify the result.)
- A fitted model has reference rate \(0.04\); a known territory has multiplier \(1.5\). A new territory receives integer factor code 3. A colleague reports \(0.04(1.5)^3=0.135\). What is wrong, and what rate follows under reference fallback? (Answer: The integer code is not a multiplier or coefficient; reference fallback gives rate \(0.04\).)