Interpreting Multivariable Fairness Diagnostics

Why Multivariable Diagnostics Need Careful Interpretation

A multivariable rate model can help compare groups while accounting for other modeled risk characteristics. But an adjusted comparison is not a complete fairness conclusion: it depends on the response, variables, functional form, and population represented in the model. A residual pattern can signal model misfit or a group-level difference, but it does not by itself explain why the pattern occurred.

This tutorial focuses on interpreting these diagnostics. It assumes the analyst has already specified a model and has defined the groups and outcomes to be reviewed. The central discipline is to report what the model and data show, state the conditioning and comparison being used, and avoid presenting an association as a cause. The interpretation approach is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: An adjusted group comparison compares modeled outcomes for two groups while holding specified covariates fixed, or while applying the same covariate distribution to both groups. It is conditional on the model specification and the chosen comparison population. It is not automatically a causal effect or a complete fairness determination.

What a Group Coefficient Says

Consider a hypothetical claim-frequency model with a Poisson response and a log link. Let \(N_i\) be the claim count for policy \(i\), \(e_i\) its exposure, \(G_i\) an indicator for group membership, and \(Z_i\) a vector of other rating variables. Assume independent Poisson counts conditional on the predictors, a correctly specified conditional mean, and positive exposure. A standard model is

N_i \mid G_i,Z_i \sim \operatorname{Poisson}(e_i\lambda_i), \qquad \log(\lambda_i)=\beta_0+\beta_G G_i+\boldsymbol{\beta}_Z^\mathsf{T}Z_i.

The exposure enters as an offset, \(\log(e_i)\), in the count model. If \(G_i=1\) denotes the group under review, then, at the same values of \(Z_i\), the modeled rate ratio is

\frac{\lambda(G_i=1,Z_i)}{\lambda(G_i=0,Z_i)}=\exp(\beta_G).

This comparison is conditional on the modeled covariates. It does not say that group membership caused the difference. It also does not establish that the covariates are appropriate to adjust for, that the model is well specified, or that the comparison answers every relevant fairness question. The same caution applies to other response models: the coefficient’s interpretation follows the selected response distribution and link.

Worked Example: Reading an Adjusted Frequency Ratio

A hypothetical insurer fits a Poisson claim-frequency model with a log link. The response is claim count, exposure is measured in policy-years, and the model includes a group indicator \(G\) and a vehicle-use indicator \(Z\). The fitted mean rate is

\log(\lambda)=\log(0.06)+\log(1.20)G+\log(1.50)Z.

For a policy with \(Z=1\) and exposure \(e=2\) policy-years, calculate the fitted expected claim count for each value of \(G\), then interpret the adjusted group comparison. Assume the model’s conditional mean is correctly specified for this illustration.

Method. Calculate the rate for each group at the same \(Z\), multiply by the same exposure, and compare. This avoids the mistake of comparing unlike covariate profiles.

For \(G=0\), the rate is \(0.06 \times 1.50=0.09\) claims per policy-year. The fitted expected count is \(2 \times 0.09=0.18\) claims.

For \(G=1\), the rate is \(0.06 \times 1.20 \times 1.50=0.108\) claims per policy-year. The fitted expected count is \(2 \times 0.108=0.216\) claims.

The adjusted rate ratio is \(0.108/0.09=1.20\), equivalently \(\exp(\log(1.20))=1.20\). The expected count ratio is also \(0.216/0.18=1.20\), because exposure is equal in the comparison. Thus, the fitted rate is 20% higher for \(G=1\) at \(Z=1\), conditional on this model. The check is that the count ratio agrees with the rate ratio. This result is evidence of an adjusted association, not proof of its cause.

Why the Log-Link Ratio Cancels

The coefficient interpretation follows directly from the model. For a fixed covariate vector \(Z\), set \(G=1\) and \(G=0\) in the linear predictor. The intercept and all terms involving \(Z\) are identical, so they cancel when the rates are divided:

\frac{\exp(\beta_0+\beta_G+\boldsymbol{\beta}_Z^\mathsf{T}Z)} {\exp(\beta_0+\boldsymbol{\beta}_Z^\mathsf{T}Z)} =\exp(\beta_G).

This justification relies on the model having no interaction between \(G\) and \(Z\). If the model includes such an interaction, the group ratio depends on \(Z\). Further, the ratio is a model-based conditional comparison; the algebra does not establish that the model is correct or that its covariates are suitable for a fairness assessment.

Adjusted Comparisons Over a Common Population

A coefficient compares groups at fixed covariate values. Another useful diagnostic is to predict outcomes for both groups across the same set of covariate profiles, then average using the same weights. This is called standardization. It answers a population-specific question: how do the groups’ model-based averages compare if both are evaluated on this chosen covariate distribution?

For profiles \(j=1,\ldots,m\), let \(w_j\) be nonnegative weights summing to 1, and let \(\widehat{\lambda}(g,Z_j)\) be the model-predicted rate after setting group status to \(g\). The standardized rate for group \(g\) is

\widehat{\lambda}^{\,\mathrm{std}}_g =\sum_{j=1}^{m}w_j\widehat{\lambda}(g,Z_j).

Using the same weights for both groups separates the comparison from differences in their observed covariate mix. The result can still change with the chosen target population and model specification. It is a standardized association, not a causal estimate merely because the covariate distribution is held common.

Worked Example: Standardizing to a Common Mix

A hypothetical model predicts annual claim rates for two vehicle-use profiles. At profile \(Z_0\), predicted rates are 0.040 for group A and 0.044 for group B. At profile \(Z_1\), they are 0.080 for group A and 0.088 for group B. The selected common population is 75% profile \(Z_0\) and 25% profile \(Z_1\). Calculate the standardized rates and their ratio.

Method. Apply the same profile weights to both groups. This avoids comparing group A’s mix with group B’s different mix.

For group A, the standardized rate is \(0.75(0.040)+0.25(0.080)=0.030+0.020=0.050\).

For group B, it is \(0.75(0.044)+0.25(0.088)=0.033+0.022=0.055\).

The standardized ratio, B to A, is \(0.055/0.050=1.10\). As a check, the two profiles each have a 1.10 group ratio, so their weighted averages must also have that ratio. The interpretation is a 10% higher model-based standardized rate for B in this chosen population. It does not establish why the modeled rates differ.

Residuals and Observed-to-Expected Patterns

A group comparison based on fitted predictions is only one diagnostic. Comparing observed outcomes with model expectations can reveal where a model systematically underpredicts or overpredicts. For a Poisson count \(N_i\) with fitted mean \(\widehat{\mu}_i=e_i\widehat{\lambda}_i\), the Pearson residual is

r_i^{P}=\frac{N_i-\widehat{\mu}_i}{\sqrt{\widehat{\mu}_i}}.

A positive residual indicates an observed count above the fitted mean; a negative residual indicates one below it. For a collection of policies, an observed-to-expected ratio is \(O/E=\sum_iN_i/\sum_i\widehat{\mu}_i\). A ratio above 1 means the aggregate observed count exceeds the aggregate fitted count. It does not imply every policy in the group is underpredicted.

Residual summaries should be examined across relevant groups and covariate levels. A pattern can signal omitted structure, a poor functional form, an unsuitable link or distribution, data problems, or genuine differences not captured by the model. Residuals are diagnostics, not explanations. Because model parameters were estimated from data, a simple residual summary is not automatically a formal significance test; uncertainty and dependence must be addressed before making inferential claims.

Worked Example: Interpreting Group Observed-to-Expected Ratios

A hypothetical frequency model is applied to two groups. For group A, the observed counts in two risk cells are 36 and 52, while the corresponding fitted expected counts are 32 and 48. For group B, observed counts are 27 and 63, with expected counts 30 and 70. Calculate each group’s aggregate \(O/E\) and the simple aggregate difference divided by the square root of expected count.

Method. Sum observed and expected counts within each group before calculating \(O/E\). The aggregate standardized difference requested here is \((O-E)/\sqrt{E}\). It is a diagnostic summary, not a formal test.

For group A, \(O=36+52=88\) and \(E=32+48=80\). Thus \(O/E=88/80=1.10\). Also, \((O-E)/\sqrt{E}=(88-80)/\sqrt{80}=8/8.9443=0.8944\).

For group B, \(O=27+63=90\) and \(E=30+70=100\). Thus \(O/E=90/100=0.90\). Also, \((O-E)/\sqrt{E}=(90-100)/\sqrt{100}=-10/10=-1.0000\).

The checks are that group A has 8 more observed than expected claims and group B has 10 fewer. The model underpredicts in aggregate for A and overpredicts in aggregate for B in these data. The summaries do not establish a cause or, on their own, a statistically reliable disparity. A common mistake is to label 1.10 as a 10% causal effect; it is only the observed-to-expected relationship for this model and data.

Interactions Change the Comparison

When a model includes an interaction between group and another rating variable, the group coefficient alone is not the group comparison at every value of that variable. For a binary covariate \(Z\), a log-link model with interaction can be written

\log(\lambda)=\beta_0+\beta_GG+\beta_ZZ+\beta_{GZ}GZ.

At \(Z=0\), the group rate ratio is \(\exp(\beta_G)\). At \(Z=1\), it is \(\exp(\beta_G+\beta_{GZ})\). Therefore, the main group coefficient describes the comparison at the reference value \(Z=0\), not a universal difference. When an interaction is present, report comparisons at meaningful covariate values or standardize predictions over a clearly stated common population.

Worked Example: Interpreting a Group Interaction

A hypothetical model for annual rates has fitted predictions for group A of 0.040 at \(Z=0\) and 0.080 at \(Z=1\). The model’s group multipliers are 1.10 at \(Z=0\) and an additional interaction multiplier of 1.50 at \(Z=1\). A target population is 75% \(Z=0\) and 25% \(Z=1\). Calculate the group B rates and standardized group ratio.

Method. Apply the group multiplier at the reference level and multiply it by the interaction multiplier at \(Z=1\). Then average both groups over the same target mix. This avoids interpreting the main coefficient as applying at every \(Z\).

At \(Z=0\), group B’s rate is \(0.040(1.10)=0.044\). At \(Z=1\), the group B to A ratio is \(1.10(1.50)=1.65\), so its rate is \(0.080(1.65)=0.132\).

Group A’s standardized rate is \(0.75(0.040)+0.25(0.080)=0.050\). Group B’s is \(0.75(0.044)+0.25(0.132)=0.033+0.033=0.066\). The standardized ratio is \(0.066/0.050=1.32\). As a check, this lies between the two stratum-specific ratios, 1.10 and 1.65, because both target weights are positive. The interaction shows that the fitted association varies by \(Z\); it does not identify the reason for that variation.

Interpretation Traps

The purpose of a multivariable diagnostic is not to produce one number that settles every fairness question. State the comparison precisely, and distinguish the measured pattern from possible explanations. In particular:

Do not call adjustment causal. Holding measured covariates fixed is a model comparison. It does not show what would happen if group membership changed, or prove that the adjusted variables are appropriate.
Do not ignore the scale. A log-link coefficient becomes a rate ratio after exponentiation. It is not a dollar difference or a percentage-point difference.
Do not interpret an interaction-free coefficient as universal when an interaction is fitted. The reference level and relevant subgroup comparisons must be stated.
Do not confuse fit with fairness. Good aggregate calibration can coexist with group-level residual patterns; poor fit can also create misleading group comparisons.
Do not infer certainty from a residual sign. Residuals describe departures from fitted values. Their size, uncertainty, exposure, and dependence matter when assessing evidence.
Do not treat one adjusted metric as the entire assessment. A conclusion depends on the selected outcome, comparison population, model, and fairness objective.

A disciplined interpretation reports the model and response, identifies which variables are held fixed or standardized, gives the comparison scale, notes interactions and relevant uncertainty, and separates the observed evidence from hypotheses about causes. This approach makes it possible to describe a disparity without claiming more than the diagnostics support.

Check Your Understanding

For each item, calculate or interpret the diagnostic as requested. Treat all scenarios as hypothetical.

A log-link frequency model has a group coefficient of \(\log(1.08)\), with no group interaction. What is the adjusted rate ratio, and how should it be described? (Answer: \(1.08\); the modeled rate is 8% higher for the indicated group at the same modeled covariates, not necessarily because of group membership.)
A group has 126 observed claims and 120 fitted expected claims. Calculate its aggregate \(O/E\) and state what the value indicates. (Answer: \(126/120=1.05\); observed claims are 5% above expected in aggregate.)
A log-link model has group coefficient \(\log(0.92)\) and group-by-\(Z\) interaction coefficient \(\log(1.25)\), where \(Z\) is binary. What are the group rate ratios at \(Z=0\) and \(Z=1\)? (Answer: \(0.92\) at \(Z=0\), and \(0.92(1.25)=1.15\) at \(Z=1\).)
Under a common target mix, 60% of policies are in profile \(Z_0\) and 40% in \(Z_1\). Predicted rates are 0.030 and 0.060 for group A, and 0.036 and 0.072 for group B. Calculate both standardized rates and the B-to-A ratio. (Answer: A \(=0.60(0.030)+0.40(0.060)=0.042\); B \(=0.60(0.036)+0.40(0.072)=0.0504\); ratio \(=1.20\).)
A group’s aggregate residual summary is positive after fitting a multivariable rate model. Does this establish that the group was charged unfairly or identify the cause of the pattern? (Answer: No. It indicates observed outcomes exceeded fitted outcomes for the summarized data; model fit, uncertainty, data quality, covariates, and other explanations require assessment.)

What You'll Learn