Why Missing and Unseen Values Need Separate Rules
A fitted rating model can encounter two situations that look similar in a data extract but are not the same. A predictor may be missing: no value was recorded for an observation. Or a categorical predictor may have a value that was not represented when the model was fitted: the value is observed, but its level is unseen by the fitted model. A missing value can be represented by a special category if that treatment is deliberately designed and supported by the training data. An unseen level has no directly estimated level effect.
The distinction matters operationally. A system that converts every unfamiliar value to a blank may apply a missing-value rule to an actual, newly introduced category. Conversely, a system that treats a blank as the reference category can assign a prediction that was never intended. The modeling specification should therefore say what happens in each case, how the choice is encoded, and whether a prediction is allowed when the case was not represented during fitting.
Missing Values in a Categorical Predictor
One possible treatment is to create an explicit “Missing” level. Under reference-level indicator coding, one level is the reference and has no separate coefficient; each other level has an indicator. If “Missing” is a non-reference level, its coefficient estimates the modeled difference between observations with missing values and observations in the reference level, conditional on the other predictors.
This treatment does not make missingness harmless or automatically informative. The missing group must occur in the fitting data for its effect to be estimated. Its estimated effect describes the observed missing group under the model; it does not establish why values were missing or guarantee that the relationship will persist. If the missing indicator is absent or constant during fitting, its coefficient cannot be identified from the data.
The relativity formula follows directly from the log link. Holding all other predictors fixed, the linear predictor for level \(B\) differs from that for \(A\) by \(\beta_B\). Exponentiating converts the difference on the log-rate scale into a ratio on the rate scale:
This comparison requires the same other predictor values and the same exposure basis. It is a fitted conditional rate ratio, not a claim that changing a value causes a policyholder’s risk to change.
Worked Example: Estimating a Missing-Level Relativity
A hypothetical insurer fits a Poisson claim-frequency model with a log link and an annual-exposure offset. The model uses territory A as the reference. The fitted annual frequency for A is \(0.080\), and the coefficient for the explicit Missing territory level is \(\log(0.75)\). A policy has 1.5 years of exposure and its territory is blank. Find its predicted frequency and expected claim count.
Assumptions and method. Conditional on the modeled predictors, claim counts are independent Poisson variables with mean exposure times frequency. The supplied coefficients are treated as fitted values. No interest or mortality assumption is involved; the response is claim count.
Step 1: Apply the missing-level indicator. The blank is represented by the Missing level, not by territory A. Its fitted relativity to A is \(\exp(\log(0.75))=0.75\).
Step 2: Calculate the frequency. The reference frequency is \(0.080\) claims per exposure-year, so the Missing-level frequency is \(0.080(0.75)=0.060\) claims per exposure-year.
Step 3: Apply exposure. The expected claim count is \(1.5(0.060)=0.090\). The predicted frequency is \(0.060\) per exposure-year, and the predicted count is \(0.090\).
Check and common mistake. The result is below the reference frequency because the estimated relativity is below 1. Treating the blank as territory A would instead give \(1.5(0.080)=0.120\) expected claims. That is the wrong calculation for this model because the data-processing rule assigns the blank to the fitted Missing level.
Unseen Levels and Fallback Rules
A level absent from fitting data has no separately estimated coefficient. In reference-level coding, the design row for a new level may accidentally be filled with zeros, producing the same numerical predictor contribution as the reference level. That is a software or implementation consequence, not evidence that the new level has the reference level’s risk. It is defensible only if the model specification explicitly adopts reference-level fallback for unseen categories.
Other possible policies include mapping a new level to a pre-defined broader group, pooling levels before fitting, or declining to produce a model prediction until the level is reviewed. A fallback should be selected using the rating purpose, available information, governance requirements, and validation evidence. It must be implemented consistently in training and prediction data processing, and monitored for changes in the mix of unseen values.
If a mapping is chosen, it is part of the rating rule. It should not be silently inferred from the numerical coding. In particular, the all-zero row for an unknown level does not have a universal interpretation across coding schemes: it may correspond to a reference category under treatment coding, while a different coding convention may assign different meanings to zero-valued contrasts.
Determine whether the value is blank, a valid fitted level, or a valid but unseen level. Do not collapse these cases before the rule is applied.
Use a fitted missing category, imputation rule, mapping, reference fallback, or prediction hold as specified. Do not let default software behavior choose the policy implicitly.
Check which indicators are active and confirm the resulting linear predictor and rate. Test the rule on representative known, missing, and unseen inputs.
Worked Example: Applying an Unseen-Level Fallback
A hypothetical insurer’s Poisson frequency model uses reference-level coding for region. The reference region, Central, has fitted frequency \(0.040\) claims per exposure-year. The fitted coefficient for Coastal is \(\log(1.30)\). A new region, Island, was not present in the fitting data. The approved operational rule maps unseen regions to Central. Find the predicted frequency and expected count for an Island policy with 0.75 years of exposure.
Assumptions and method. Claim counts are modeled as Poisson with mean equal to exposure times frequency, and the log link is used. The supplied fitted values are fixed. The written fallback rule, rather than an estimated Island coefficient, determines the prediction. Interest and mortality are not relevant.
Step 1: Apply the fallback. Island is unseen, so the specified rule maps it to Central. The Coastal coefficient does not apply, and there is no fitted Island coefficient.
Step 2: Determine the frequency. Central is the reference level, so its indicator contribution is zero. The frequency is \(0.040\) per exposure-year.
Step 3: Calculate expected count. The prediction is \(0.75(0.040)=0.030\) claims.
Check and common mistake. The result is \(0.030\) expected claims, not \(0.75(0.040)(1.30)=0.039\). The latter calculation incorrectly applies the Coastal relativity to Island merely because Coastal has a nonzero fitted coefficient. The example’s rule specifically maps Island to Central.
Imputation and a Missingness Indicator
For a continuous predictor, a common explicit approach is to replace missing values with a chosen constant and add a missingness indicator. The fitted model then uses the imputed value for every observation and uses the indicator to distinguish records that were imputed. The imputation value and indicator must be applied consistently when fitting and scoring. If the indicator was not variable in the training data, its effect cannot be estimated; adding it only at scoring time does not create an estimated coefficient.
This construction has a specific interpretation: among records with the same imputed or recorded predictor value and other predictors, the indicator multiplies the modeled rate by \(\exp(\gamma)\) when the value is missing. It is not the same as claiming the missing predictor truly equals the imputation constant. The indicator gives the model a separate adjustment for the missing-data group, subject to the model’s assumptions and the data available for estimating that adjustment.
Worked Example: Imputation With a Missingness Indicator
A hypothetical Poisson frequency model uses annual exposure and a log link. Its frequency model is \(\log(\lambda)=\log(0.020)+0.03(x-40)+\log(1.10)M\), where \(x\) is age in years after preprocessing and \(M=1\) if age was missing and \(M=0\) otherwise. Missing ages are imputed as 40. Find the predicted frequency and expected count for a policy with missing age and 2.5 years of exposure.
Assumptions and method. Conditional claim counts are Poisson with mean \(e\lambda\), the log link is used, and the stated model coefficients are fitted values. The indicator was present and estimable in the fitting data. No interest or mortality assumption is needed.
Step 1: Impute and set the indicator. For a missing age, \(x=40\) and \(M=1\). Thus \(x-40=0\), so the continuous-age contribution is \(0.03(0)=0\).
Step 2: Apply the missingness adjustment. The rate multiplier is \(\exp(\log(1.10))=1.10\). Therefore \(\lambda=0.020(1.10)=0.022\) claims per exposure-year.
Step 3: Apply exposure. The expected count is \(2.5(0.022)=0.055\) claims.
Check and common mistake. The frequency is \(0.022\) and the expected count is \(0.055\). Omitting the indicator would give \(2.5(0.020)=0.050\), which ignores the model’s fitted missingness adjustment. Applying the age slope to an arbitrary value such as zero would also be wrong: the specified imputation is 40.
Validation and Practical Controls
Missingness and unseen-level rules affect both statistical performance and the behavior of a deployed rating model. Evaluate them as explicit parts of the model specification rather than as incidental data cleaning. In particular:
- Use a consistent definition of blank, invalid, and unseen values across fitting, validation, and production scoring.
- Check whether each missing indicator and modeled category actually varies in the fitting data. A coefficient for an absent or constant indicator is not identified.
- Assess whether the missing-value treatment is supported by enough observations and whether its predictions are stable. Sparse missing groups may produce uncertain effects.
- Test each fallback path directly, including the resulting indicators, linear predictor, and predicted rate.
- Monitor the frequency and composition of missing and unseen inputs. A new level or a sudden rise in blanks can signal a data change that warrants investigation.
A model prediction for an unseen level is always conditional on a stated rule. If there is no defensible rule, “no prediction pending review” is more transparent than silently assigning a rate through accidental coding behavior.
Check Your Understanding
For each problem, identify the relevant coding or fallback rule before calculating.
- A Poisson frequency model has reference frequency \(0.030\), and a fitted Missing-category coefficient of \(\log(0.80)\). A blank predictor is assigned to that category. For 2 years of exposure, find the frequency and expected count. (Answer: frequency \(0.024\); expected count \(0.048\))
- A model’s reference region has frequency \(0.060\). A fitted non-reference level has coefficient \(\log(1.25)\). An unseen region is explicitly mapped to the reference, and its policy has 2.5 years of exposure. Find the expected count. (Answer: \(0.150\))
- A frequency model is \(\log(\lambda)=\log(0.012)+0.02(x-30)+\log(1.10)M\). Missing values are imputed as 30. Find the frequency and expected count for missing \(x\) with 4 years of exposure. (Answer: frequency \(0.0132\); expected count \(0.0528\))
- A categorical level did not occur at all in model fitting. Can its separate coefficient be estimated from that fit, and what must be specified before assigning it a prediction? (Answer: No; a documented fallback, mapping, pooling rule, or decision to withhold prediction is needed.)
- Under reference-level coding, a new category is encoded with all category indicators equal to zero. What does that row calculate, and why is that not automatically a valid risk assumption for the new category? (Answer: It calculates the reference-level contribution; the new category has no estimated effect, so using that contribution is valid only if reference-level fallback was explicitly chosen.)