Why Monitor Predictor Distributions?
A predictive model is applied to records that may differ from the data used to develop it. Monitoring predictor distributions is a structured way to notice such differences. For a numerical predictor, a shift might appear in its mean, spread, or range; for a categorical predictor, it might appear in the share of records in one or more levels. These comparisons can flag changes in the population, the data collection process, or the processing pipeline that deserve investigation.
A distribution shift is an investigation signal, not by itself evidence that prediction quality has worsened. The predictor summaries do not say whether outcomes changed or whether predictions are less accurate. Interpret them alongside the data definitions, collection context, and the purpose of the model. This monitoring approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A useful comparison first makes the populations and measurement rules comparable. State which records are included, the time periods, the predictor definitions, and the sample sizes. Use the same units and category labels in both sets. If definitions or processing have changed, document that fact rather than treating the resulting summaries as directly comparable. Report counts as well as proportions: the same proportion can represent very different amounts of data when sample sizes differ.
Summarizing Numerical Predictors
For a numerical predictor \(X\), common summaries include the number of records \(n\), mean \(\bar{x}\), sample standard deviation \(s\), median, and selected quantiles or range. The mean describes the center on the original scale, while the standard deviation and quantiles describe aspects of variability. No single summary fully describes a distribution. A mean can move because of a shift in the bulk of the data or because of a few unusually large or small values, so examine more than one summary when interpreting a difference.
One way to express a difference in means on a scale that accounts for within-window variability is the standardized mean difference. For two independent samples, let \(n_B,\bar{x}_B,s_B\) denote the baseline sample size, mean, and sample standard deviation, and let \(n_M,\bar{x}_M,s_M\) denote the corresponding monitoring values. When a common within-group variance is a reasonable assumption, the pooled sample standard deviation and standardized mean difference are:
The sign gives the direction of the change: a positive value means the monitoring mean is higher. The magnitude expresses the mean difference in pooled-standard-deviation units. This is a descriptive scale, not a universal pass/fail rule; there is no single cutoff that makes a shift consequential for every predictor or use case. If the groups are not independent, or their within-group variances differ substantially, the pooled scale may not be suitable. In that situation, report the separate spreads and choose a comparison appropriate to the data rather than implying a common variance.
Why the Pooled Standard Deviation Has This Form
The pooled variance combines the two groups’ within-group sums of squared deviations. A sample variance is the sum of squared deviations from its group mean divided by that group’s degrees of freedom, \(n-1\). Therefore the baseline sum of squares is \((n_B-1)s_B^2\), and the monitoring sum of squares is \((n_M-1)s_M^2\). Adding these sums and dividing by the total within-group degrees of freedom, \((n_B-1)+(n_M-1)=n_B+n_M-2\), gives the pooled variance. Taking its square root puts the pooled measure back in the predictor’s original units. Dividing the difference in means by this standard deviation then expresses that difference in standard-deviation units.
This justification also shows the condition behind the pooled summary: it treats the two within-group variances as estimates of a common variance. It is a useful descriptive standardization when that common scale is defensible, but it should not conceal markedly different spreads.
Worked Example: Standardizing a Shift in a Numerical Predictor
Suppose a hypothetical insurer compares the number of years of prior coverage in two independent record sets. The baseline values are \(2,4,4,6,9\), and the monitoring values are \(4,6,6,8,11\). Calculate each mean and sample standard deviation, then the pooled standard deviation and standardized mean difference. The method is the pooled descriptive comparison; the common mistake to avoid is dividing by a variance instead of a standard deviation.
Baseline mean. The values sum to \(2+4+4+6+9=25\), so \(\bar{x}_B=25/5=5\). The deviations from 5 are \(-3,-1,-1,1,4\); their squared values sum to \(9+1+1+1+16=28\). Thus the sample variance is \(28/(5-1)=7\), and \(s_B=\sqrt{7}\approx2.6458\).
Monitoring mean. The values sum to \(4+6+6+8+11=35\), so \(\bar{x}_M=35/5=7\). The deviations from 7 are \(-3,-1,-1,1,4\), again giving a sum of squares of 28. Thus the sample variance is \(28/4=7\), and \(s_M=\sqrt{7}\approx2.6458\).
Pooled scale and standardized difference. Both sample sizes are 5, so \(s_p=\sqrt{\{4(7)+4(7)\}/8}=\sqrt{7}\approx2.6458\). The mean difference is \(7-5=2\) years. Therefore \(\text{SMD}=2/\sqrt{7}\approx0.7559\), rounded to four decimal places.
Check and interpretation. Adding 2 to every baseline value produces the monitoring values, so the equal means’ difference of 2 and equal spreads are consistent with the data. The monitoring mean is about 0.7559 pooled standard deviations higher. This describes the predictor shift; it does not establish a change in predictive performance.
Comparing Categorical Predictors
For a categorical predictor with the same defined levels in both windows, calculate the proportion in each level using that window’s total as the denominator. If \(n_{Bj}\) and \(n_{Mj}\) are the counts in level \(j\), then \(\hat{p}_{Bj}=n_{Bj}/n_B\) and \(\hat{p}_{Mj}=n_{Mj}/n_M\). The difference \(\hat{p}_{Mj}-\hat{p}_{Bj}\) is a change in proportion; multiplying it by 100 expresses the result in percentage points. A percentage-point change is not the same as a relative percent change.
For categories that form a complete, mutually exclusive set, the proportions across all levels should sum to 1 in each window, apart from rounding. Compare the full distribution, not just the most common level. Check that the category definitions are consistent and that each record has been assigned appropriately; a changed label or coding rule can create an apparent shift. Retain counts and report rounding clearly.
Worked Example: Comparing Optional-Coverage Proportions
In a hypothetical portfolio, 120 of 400 baseline records have an optional coverage, while 90 of 200 monitoring records have it. Calculate the proportions with and without the coverage in each window, then the change in the share with coverage. The method is a level-by-level proportion comparison; the common mistake to avoid is reporting a proportion such as 0.30 as 30 percentage points.
Baseline proportions. The share with coverage is \(120/400=0.30\). The number without coverage is \(400-120=280\), so that share is \(280/400=0.70\). The two shares sum to \(0.30+0.70=1.00\).
Monitoring proportions. The share with coverage is \(90/200=0.45\). The number without coverage is \(200-90=110\), so that share is \(110/200=0.55\). These shares sum to \(0.45+0.55=1.00\).
Change. For the coverage level, the difference is \(0.45-0.30=0.15\), or an increase of 15 percentage points. It is also a 50% relative increase because \(0.15/0.30=0.50\), but that is a different measure and should not be substituted for the percentage-point change.
Check and interpretation. Both sets of level proportions sum to 1.00, and the monitoring share with coverage is higher. This observed difference is a prompt to check context and data handling; it is not proof of a performance loss.
Read Several Summaries Together
A shift in one statistic can have different explanations. Comparing the mean with the median and the spread can help describe what changed, without claiming to identify the cause. The next example uses a small hypothetical sample so every calculation is visible. For the standardized difference, the independent-sample pooled calculation is used as a descriptive scale; the very different variances are also reported rather than hidden.
Worked Example: A Mean Shift With an Unchanged Median
A hypothetical service team compares a numerical predictor measured in years. Baseline values are \(1,2,3,4,5\); monitoring values are \(1,2,3,4,10\). Compare the mean, median, range, sample standard deviation, and pooled standardized mean difference. The method is to calculate each summary directly; the common mistake to avoid is concluding that the entire distribution is unchanged just because the medians match.
Baseline summaries. The sum is \(1+2+3+4+5=15\), so the mean is \(15/5=3\). The middle value is the median, also 3. The range is from 1 to 5. Squared deviations from 3 sum to \(4+1+0+1+4=10\), so the sample variance is \(10/4=2.5\), and the sample standard deviation is \(\sqrt{2.5}\approx1.5811\).
Monitoring summaries. The sum is \(1+2+3+4+10=20\), so the mean is \(20/5=4\). The middle value remains 3, and the range is now from 1 to 10. Squared deviations from 4 sum to \(9+4+1+0+36=50\), so the sample variance is \(50/4=12.5\), and the sample standard deviation is \(\sqrt{12.5}\approx3.5355\).
Pooled scale. With four degrees of freedom in each sample, \(s_p=\sqrt{\{4(2.5)+4(12.5)\}/8}=\sqrt{60/8}=\sqrt{7.5}\approx2.7386\). The mean difference is \(4-3=1\), so the standardized mean difference is \(1/\sqrt{7.5}\approx0.3651\), rounded to four decimal places.
Check and interpretation. The median is unchanged, but the mean rises by 1, the range increases by 5, and the monitoring sample standard deviation is larger. The value 10 contributes substantially to the monitoring mean and spread. These summaries describe a change concentrated in the upper end of this small sample; they do not tell us whether it reflects a genuine population change, a data issue, or another cause.
A Practical Monitoring Sequence
A consistent process helps keep a descriptive comparison useful and appropriately cautious.
Record the baseline and monitoring periods or samples, the record counts, inclusion rules, and predictor definitions. Confirm that units and coding are comparable.
Compare means and standard deviations, and add medians or range summaries where useful. If reporting a standardized mean difference, state the scale and its assumptions.
Report the count and proportion for every level in both sets. Give differences in percentage points and check that the proportions for a complete set of levels sum to 1.
State what shifted, in which direction, and which summaries show it. Check relevant operational or data-definition context before drawing a conclusion about its meaning.
Comparisons should also be interpreted with sample size in mind. A proportion based on a small monitoring sample can vary substantially from one window to another, while a small difference in a very large sample may be precisely measured but operationally unimportant. Report the window sizes and avoid treating a numerical threshold as a universal rule. If a feature definition, data source, or pipeline changed, the resulting difference may reflect measurement or processing rather than a change in the underlying population.
Common Mistakes and Careful Interpretation
- Calling a distribution shift proof of degraded predictions. Predictor summaries contain no direct comparison of predictions with outcomes. They flag a difference to investigate, not a performance result.
- Confusing percentage points with relative change. A proportion rising from 0.30 to 0.45 increases by 0.15, or 15 percentage points; relative to 0.30, that is a 50% increase.
- Reporting only percentages. Include the counts and denominators. A proportion of 0.20 based on 10 records is not supported by the same amount of information as 0.20 based on 10,000 records.
- Overlooking spread or tails. Equal means do not imply equal distributions, and an unchanged median does not rule out changes elsewhere. Consider multiple summaries.
- Using a pooled standard deviation without qualification. The pooled scale assumes a common within-group variance for its usual justification. If spreads differ markedly, display them separately and explain the limitation.
- Comparing inconsistent definitions. Different units, category mappings, or record-inclusion rules can manufacture an apparent difference. Confirm comparability before interpreting it.
Check Your Understanding
For each problem, calculate the requested summaries and state what the predictor comparison does and does not establish.
- In a hypothetical portfolio, 15 of 100 baseline records and 27 of 120 monitoring records have a particular feature. Find both proportions and the change in percentage points. (Answer: baseline \(0.15\), monitoring \(0.225\), increase of 7.5 percentage points.)
- A numerical predictor has baseline mean 50 and sample standard deviation 10, based on 21 records. In a separate monitoring sample of 21 records, its mean is 54 and sample standard deviation is 12. Calculate the pooled standard deviation and standardized mean difference. (Answer: \(s_p=\sqrt{122}\approx11.0454\); SMD \(\approx0.3621\).)
- A categorical predictor has three mutually exclusive levels. Baseline counts are 50, 30, and 20 out of 100; monitoring counts are 40, 36, and 24 out of 100. Find each set of proportions and the percentage-point change for each level. (Answer: baseline \(0.50,0.30,0.20\); monitoring \(0.40,0.36,0.24\); changes \(-10,+6,+4\) percentage points.)
- Baseline values of a numerical predictor are \(2,3,3,4,4\), and monitoring values are \(2,3,3,4,8\). Find the two means and medians. What feature of the comparison is evident? (Answer: means \(3.2\) and \(4.0\); both medians are \(3\); the monitoring mean increases while the median is unchanged, consistent with a difference toward the upper end.)
- A summary shows that a categorical predictor’s share in one level increased from \(0.20\) to \(0.28\). Give the change in percentage points and explain why this alone cannot establish a loss of predictive performance. (Answer: increase of 8 percentage points; it describes a predictor-distribution difference, but contains no evidence about prediction errors or realized outcomes.)