Why Fairness Monitoring Must Continue After Deployment
A model can be evaluated carefully before use and still become less reliable or less fair later. The people who apply, purchase, or qualify for a product may change; outcome rates may shift; and the relationship between predictors and outcomes may no longer resemble the development data. A single fairness audit is therefore a snapshot, not a continuing assurance.
Monitoring should distinguish three questions. Has the composition of the population changed? Has the prevalence of the outcome changed within groups? Have the model’s errors changed within groups? These signals can move separately. A change in group representation alone does not establish that model errors are unequal, and similar error rates do not establish that the population mix is unchanged.
A useful monitoring plan states what is measured, for which cohort and groups, over what period, with what denominator, and what response follows a signal. The monitoring design should be set before results are reviewed, rather than adjusted after an unfavorable rate appears. The examples below use hypothetical insurance decision data; their alert thresholds are illustrative governance choices, not universal statistical or legal standards.
Specify the Cohort, Measures, and Comparison
Choose a consistent unit of analysis, such as one eligible application per decision month. Define the observation window, the model version, the decision threshold, and the point at which outcomes are considered mature enough to evaluate. If outcomes arrive after a delay, compare cohorts at the same development age or explicitly account for incomplete labels. Otherwise, recent cohorts may appear to have fewer adverse outcomes simply because they have had less time to emerge.
For each monitoring period, record group counts and the total number of eligible decisions. Let \(n_g\) be the number of observations in group \(g\), and \(N\) the total number. The group representation share is \(n_g/N\). If \(Y=1\) denotes the adverse outcome, the group-specific prevalence is the number with \(Y=1\) in group \(g\), divided by \(n_g\). For a binary prediction \(\hat{Y}\), common conditional error measures are:
The true positive rate (TPR) and false positive rate (FPR) require observed outcomes and denominators among people with the corresponding actual outcome. They are not proportions of all decisions in a group. Other measures may be appropriate to the decision, including false negative rate, precision, calibration, or an outcome-specific loss. The measure must be tied to the intended use and the harm being monitored; there is no single rate that fully describes fairness.
Compare current results both with a pre-deployment or approved baseline and with explicitly chosen tolerances. A gap between groups can be important, but so can a deterioration within one group even when the group gap narrows. Record counts beside rates: a rate based on a small denominator is uncertain, and a missing or delayed outcome can make a seemingly precise comparison misleading.
Worked Example: Separating Representation and Prevalence Drift
Worked Example: Separating Representation and Prevalence Drift
A hypothetical insurer monitors 1,000 eligible decisions in each of two periods. Group A and Group B are the monitored groups. In the baseline period, there are 600 people in A, of whom 72 experience the defined adverse outcome, and 400 in B, of whom 32 experience it. In the current period, there are 450 in A, with 81 outcomes, and 550 in B, with 44 outcomes. A pre-set plan flags a representation change above 10 percentage points or a within-group prevalence change above 3 percentage points. Identify the signals. The thresholds here are illustrative.
Method. Calculate each group share using all eligible decisions as the denominator, and calculate prevalence using the relevant group count as the denominator.
Baseline representation. Group A has \(600/1{,}000=0.60\), or 60%; Group B has \(400/1{,}000=0.40\), or 40%.
Current representation. Group A has \(450/1{,}000=0.45\), or 45%; Group B has \(550/1{,}000=0.55\), or 55%. A’s share changed by \(45\%-60\%=-15\) percentage points, and B’s by \(55\%-40\%=15\) percentage points. Both exceed the illustrative 10-point alert threshold in magnitude.
Baseline prevalence. In A, \(72/600=0.12\), or 12%. In B, \(32/400=0.08\), or 8%.
Current prevalence. In A, \(81/450=0.18\), or 18%. In B, \(44/550=0.08\), or 8%. A’s prevalence increased by 6 percentage points, exceeding the 3-point threshold. B’s change is 0 points.
Check and interpretation. The baseline total is \(72+32=104\) outcomes, so overall prevalence is \(104/1{,}000=10.4\%\). The current total is \(81+44=125\), so overall prevalence is \(125/1{,}000=12.5\%\). Representation drift and a prevalence increase in A both warrant investigation; neither calculation establishes that the model’s errors have changed. The method avoids treating a change in group share as if it were a change in within-group outcome prevalence.
Worked Example: Monitoring Conditional Error Rates
Worked Example: Monitoring Conditional Error Rates
A hypothetical screening model is evaluated on mature outcomes. In the baseline cohort, group A has 200 actual positive outcomes, with 180 predicted positive, and 800 actual negative outcomes, with 48 predicted positive. Group B has 200 actual positives, with 170 predicted positive, and 800 actual negatives, with 64 predicted positive. In the current cohort, A has 200 actual positives, with 160 predicted positive, and 800 actual negatives, with 80 predicted positive. B has 200 actual positives, with 164 predicted positive, and 800 actual negatives, with 72 predicted positive. A pre-set rule calls for investigation if a group’s TPR falls by more than 5 percentage points from baseline.
Method. Compute TPR among actual positives and FPR among actual negatives, separately by group and period.
Baseline rates. For A, \(\operatorname{TPR}_A=180/200=0.90\) and \(\operatorname{FPR}_A=48/800=0.06\). For B, \(\operatorname{TPR}_B=170/200=0.85\) and \(\operatorname{FPR}_B=64/800=0.08\).
Current rates. For A, \(\operatorname{TPR}_A=160/200=0.80\) and \(\operatorname{FPR}_A=80/800=0.10\). For B, \(\operatorname{TPR}_B=164/200=0.82\) and \(\operatorname{FPR}_B=72/800=0.09\).
Compare changes and gaps. A’s TPR fell by \(0.80-0.90=-0.10\), a 10-point decline, exceeding the 5-point trigger. B’s TPR fell by 3 points. The current TPR gap is \(0.80-0.82=-0.02\), only 2 points in magnitude; the smaller current gap does not erase A’s deterioration. The current FPR gap is \(0.10-0.09=0.01\), or 1 point.
Check uncertainty. As a rough standard-error check for the difference in A’s TPR, using independent cohorts and binomial sampling, the standard error is approximately \(\sqrt{0.90(0.10)/200+0.80(0.20)/200}=\sqrt{0.00125}=0.0354\). The observed 0.10 decline is about \(0.10/0.0354=2.82\) such standard errors. This is a signal worth examining, not proof of its cause; the monitoring team should also check label quality, cohort comparability, and repeated-testing procedures. The method avoids concluding that fairness improved merely because the between-group gap narrowed.
Why Prevalence Shifts Can Change Predictive Value
When outcome prevalence changes, the proportion of positive predictions that are correct can change even if conditional error rates remain stable. This follows from Bayes’ rule. Assume a binary outcome, a fixed decision rule, and stable TPR and FPR within a group. If the group’s prevalence is \(\pi=P(Y=1)\), then a positive prediction can arise from a true positive, with probability \(\operatorname{TPR}\pi\), or a false positive, with probability \(\operatorname{FPR}(1-\pi)\). Therefore:
The denominator is the total probability of a positive prediction, and the numerator is the probability of a true positive prediction. Thus, holding TPR and FPR fixed does not hold positive predictive value fixed when \(\pi\) changes. This result explains why monitoring only one fairness statistic can miss a consequential change. The assumptions matter: if the conditional score distributions or the decision process also change, stable TPR and FPR cannot be presumed.
Worked Example: Distinguishing a Prevalence Shift From Error-Rate Drift
A hypothetical model has baseline prevalence 10%, TPR 80%, and FPR 10%. In a current cohort, prevalence is 20%; investigation so far finds TPR and FPR still at 80% and 10%. For cohorts of 1,000, calculate the expected confusion counts and positive predictive value in each period. What changed?
Method. Apply each period’s prevalence to 1,000 observations, then apply TPR to the actual positives and FPR to the actual negatives.
Baseline counts. There are \(0.10(1{,}000)=100\) actual positives and 900 actual negatives. TPR of 0.80 gives \(0.80(100)=80\) true positives and \(100-80=20\) false negatives. FPR of 0.10 gives \(0.10(900)=90\) false positives and \(900-90=810\) true negatives. Positive predictive value is \(80/(80+90)=80/170=0.4706\), or 47.06% to two decimal places.
Current counts. There are \(0.20(1{,}000)=200\) actual positives and 800 actual negatives. TPR gives \(0.80(200)=160\) true positives and 40 false negatives. FPR gives \(0.10(800)=80\) false positives and 720 true negatives. Positive predictive value is \(160/(160+80)=160/240=0.6667\), or 66.67% to two decimal places.
Check and interpretation. Baseline correct classifications are \(80+810=890\), or 89.0%; current correct classifications are \(160+720=880\), or 88.0%. TPR and FPR stayed constant, while prevalence rose and positive predictive value increased from 47.06% to 66.67%. The change does not by itself show that conditional error performance improved. The calculation avoids treating a changed predictive value as direct evidence of a changed TPR or FPR.
Turn Signals Into Proportionate Actions
A monitoring plan needs an action ladder, not just a dashboard. An alert is a request to understand a signal; it is not automatically a finding of unfairness or a reason to change a model. A practical sequence separates detection, investigation, remedial action, and follow-up:
Confirm cohort definitions, group labels, model version, denominators, outcome maturity, and data pipelines. Check whether the alert is based on a rate with too few observations for a stable conclusion.
Compare representation, prevalence, and errors separately. Examine changes in eligibility, product mix, decisions, data collection, and operational practice. Assess who may be harmed and how severe or reversible that harm is.
If inputs or processes changed, correct them where appropriate. If predictions are systematically miscalibrated but the model remains suitable for ranking and use, consider recalibration and validate the result. If reliability or safety is in doubt, restrict use or suspend the affected decision until the issue is resolved.
Record the trigger, evidence, decision, responsible owner, and review date. After any intervention, monitor the same measures and relevant outcomes to check that the response worked and did not create a new problem.
Recalibration is not a remedy for every drift signal. It adjusts the mapping from model scores to probabilities; it does not necessarily repair a changed predictor-outcome relationship, unequal error patterns, poor data, or an unsuitable decision rule. Any recalibration should be assessed on fresh, representative data and by group as well as overall. Likewise, a stable aggregate metric cannot establish that every intersectional group is adequately served.
Suspension or restricted use is appropriate to consider when outcomes or inputs cannot be trusted, performance falls below a material safety or service standard, a serious group-specific harm is plausible, or a remedy cannot be validated in time. The response should be proportionate to the decision’s impact and should include a safe operational alternative where possible. A fixed number of alerts should not replace judgment about the consequences of an error.
Common Monitoring Traps
- Using the wrong denominator. Prevalence is calculated within a group; representation is calculated against the full eligible population; TPR and FPR use actual-positive and actual-negative denominators, respectively.
- Waiting for one overall metric to move. Aggregate performance can conceal deterioration in a group, while group composition changes can move aggregate rates even when within-group rates do not.
- Reacting to every fluctuation. Small groups and repeated monthly comparisons produce noisy rates and occasional alerts. Predefine review rules, include counts and uncertainty, and examine whether a signal persists or has a plausible operational cause.
- Comparing immature outcomes. A shorter observation period can make recent adverse outcomes look artificially rare. Align outcome maturity before comparing periods.
- Treating thresholds as universal. Alert limits depend on the decision, available data, potential harm, and governance requirements. Document why the chosen limits are suitable and who can authorize a change.
Check Your Understanding
For each problem, distinguish the measure being monitored and show the relevant denominator.
- A cohort has 2,000 eligible decisions, including 700 in Group C. In the next cohort, 640 of 2,000 decisions are in C. What is the change in C’s representation share in percentage points? (Answer: \(640/2{,}000-700/2{,}000=0.32-0.35=-0.03\), a decrease of 3 percentage points.)
- In Group D, 54 of 300 people have an adverse outcome in the baseline period; 72 of 360 do in the current period. What are the two prevalences and the change? (Answer: \(54/300=18\%\); \(72/360=20\%\); increase of 2 percentage points.)
- A model identifies 126 of 180 actual positive cases in a group. What is its TPR? (Answer: \(126/180=0.7000\), or 70.00%.)
- A group has prevalence 25%, TPR 80%, and FPR 20%. Using the stated fixed-rate assumptions, calculate positive predictive value. (Answer: \(0.80(0.25)/[0.80(0.25)+0.20(0.75)]=0.20/0.35=0.5714\), or 57.14%.)
- A group’s TPR declines from 92% to 84%; a monitoring rule flags declines greater than 5 percentage points. The current estimate uses 25 actual positive cases, and outcomes are mature. What follows immediately from the rule, and what should the analyst do next? (Answer: the 8-point decline triggers investigation; validate the data and denominator, assess uncertainty and cohort comparability, examine causes and potential harm, then decide whether a remedy or restriction is justified. The trigger alone does not prove the cause.)