Auditing Unequal Errors by Group

Why Audit Errors by Group?

A classification decision can be wrong in two different ways: it can flag a case whose actual outcome is negative, or it can fail to flag a case whose actual outcome is positive. These errors do not necessarily affect groups at the same rates, and their consequences may differ. A useful audit therefore does more than report one overall accuracy figure: it shows the groupwise counts, the denominator behind each rate, and the uncertainty in each estimate.

This tutorial develops a practical audit for a binary outcome and a binary decision. The observed outcome is treated as the reference label for the audit; that does not establish that the label is error-free. As the earlier material on historical bias in labels and measurement bias emphasizes, a rate computed against an observed label can inherit limitations in that label.

Definition: Let $Y=1$ denote an observed positive outcome and $Y=0$ an observed negative outcome. Let $D=1$ denote a positive model decision and $D=0$ a negative decision. For group $g$, the false-positive rate is the proportion of actual negatives assigned a positive decision, $P(D=1\mid Y=0,G=g)$. The false-negative rate is the proportion of actual positives assigned a negative decision, $P(D=0\mid Y=1,G=g)$.

The conditioning matters. False-positive rate uses actual negatives as its denominator; false-negative rate uses actual positives. In a group with $TN_g$ true negatives, $FP_g$ false positives, $TP_g$ true positives, and $FN_g$ false negatives, the estimated rates are

\widehat{\operatorname{FPR}}_g=\frac{FP_g}{FP_g+TN_g}, \qquad \widehat{\operatorname{FNR}}_g=\frac{FN_g}{FN_g+TP_g}.

The counts should be reported with the rates. A rate without its denominator hides how much information supports it. The positive predictive value, by contrast, uses predicted positives as its denominator; it answers a different question and cannot be substituted for either error rate.

Build the Audit From Counts

A clean audit starts with one confusion table per group. Here, “actual” means the outcome label used for evaluation, and “decision” means the model’s recorded classification at the specified operating threshold. State the evaluation population and time period as well: rates can change when either changes.

Group $g$	Actual positive, decision positive	Actual positive, decision negative	Actual negative, decision positive	Actual negative, decision negative	Actual positive denominator	Actual negative denominator
$g$	$TP_g$	$FN_g$	$FP_g$	$TN_g$	$TP_g+FN_g$	$FP_g+TN_g$

The table makes the rate calculations auditable and helps expose missing or inconsistent records. If a group has no observed actual positives, its false-negative rate cannot be calculated from that audit sample. If it has no actual negatives, its false-positive rate cannot be calculated. These are undefined rates, not zero rates.

Define the decision and evaluation set.
State the outcome label, what counts as a positive decision, the threshold or decision rule, the groups being compared, and the period covered.

Tabulate counts within each group.
Record $TP_g$, $FN_g$, $FP_g$, and $TN_g$, then show the positive and negative denominators.

Calculate error rates and uncertainty.
Compute false-positive and false-negative rates with their own denominators. Add uncertainty intervals when the sampling interpretation and sample size support them.

Interpret the differences in context.
Describe which errors occur, how often, and what they mean for affected people. Do not treat a numerical difference alone as a complete fairness conclusion.

Approximate Uncertainty for an Error Rate

When each evaluated case can reasonably be treated as an independent Bernoulli observation within the relevant group and outcome stratum, an estimated error rate is a sample proportion. For a rate $\hat p=x/n$, where $x$ is the number of errors and $n$ is the appropriate denominator, its estimated standard error is

\widehat{\operatorname{SE}}(\hat p)=\sqrt{\frac{\hat p(1-\hat p)}{n}}.

A common approximate 95% confidence interval is $\hat p\pm1.96\widehat{\operatorname{SE}}(\hat p)$. This normal approximation is most defensible when both $n\hat p$ and $n(1-\hat p)$ are reasonably large; a practical check is that each is at least about 10. The interval is an approximation, and it can extend outside the possible range from 0 to 1. Do not present it as reliable when the approximation conditions are poor, nor as a guarantee that the group’s underlying rate lies in the interval.

Why the standard-error formula holds: Under the stated independent Bernoulli model, each case has an error indicator $Z_i$ with $P(Z_i=1)=p$, so $\operatorname{Var}(Z_i)=p(1-p)$. The sample rate is $\hat p=(Z_1+\cdots+Z_n)/n$. Independence makes the variance of the sum $np(1-p)$, hence $\operatorname{Var}(\hat p)=p(1-p)/n$. Replacing the unknown $p$ with its estimate $\hat p$ gives the estimated standard error above.

An interval describes uncertainty in a rate estimate, not the consequences of an error. Further, seeing overlapping intervals does not prove that two groups have equal rates; seeing non-overlapping intervals is not, by itself, a complete analysis of the difference. The audit should report the estimates and their uncertainty plainly, then interpret them in the decision’s context.

Worked Example: Compare Error Rates With Their Denominators

In a hypothetical review of an automated referral rule, Group A has 500 cases: 100 actual positives and 400 actual negatives. Among the positives, 80 receive a positive decision; among the negatives, 40 receive a positive decision. Group B has 400 cases: 80 actual positives and 320 actual negatives. Among the positives, 56 receive a positive decision; among the negatives, 64 receive a positive decision. Calculate each group’s false-positive and false-negative rates, with approximate 95% intervals. Use four decimals for rates and interval endpoints.

Method. First obtain each denominator from the actual outcome, then calculate the rates and standard errors. This avoids confusing a rate conditioned on the outcome with one conditioned on the decision.

For Group A, $TP_A=80$, $FN_A=100-80=20$, $FP_A=40$, and $TN_A=400-40=360$. The false-positive denominator is $400$, so $\widehat{\operatorname{FPR}}_A=40/400=0.1000$. Its standard error is $\sqrt{0.1(0.9)/400}=\sqrt{0.000225}=0.0150$. The interval is $0.1000\pm1.96(0.0150)=0.1000\pm0.0294$, or $(0.0706,0.1294)$.

The false-negative denominator is $100$, so $\widehat{\operatorname{FNR}}_A=20/100=0.2000$. Its standard error is $\sqrt{0.2(0.8)/100}=\sqrt{0.0016}=0.0400$. The interval is $0.2000\pm1.96(0.0400)=0.2000\pm0.0784$, or $(0.1216,0.2784)$.

For Group B, $TP_B=56$, $FN_B=80-56=24$, $FP_B=64$, and $TN_B=320-64=256$. Thus $\widehat{\operatorname{FPR}}_B=64/320=0.2000$. Its standard error is $\sqrt{0.2(0.8)/320}=\sqrt{0.0005}=0.0224$ to four decimals. Using the unrounded standard error, the interval is $0.2000\pm1.96(0.0223607)=0.2000\pm0.0438$, or $(0.1562,0.2438)$.

The false-negative rate is $24/80=0.3000$. Its standard error is $\sqrt{0.3(0.7)/80}=\sqrt{0.002625}=0.0512$. The interval is $0.3000\pm1.96(0.0512348)=0.3000\pm0.1004$, or $(0.1996,0.4004)$.

Check. Group B has twice the estimated false-positive rate of Group A, $0.2000$ versus $0.1000$, and a higher estimated false-negative rate, $0.3000$ versus $0.2000$. The denominators are correctly the actual negatives for the first comparison and the actual positives for the second. These estimates do not, by themselves, explain why the differences arose or settle their practical significance.

Interpret Rates Alongside Consequences

Rates answer how frequently a type of error occurs within a specified outcome group. They do not directly say how many people experience that error in the full population, and they do not assign a cost or harm to it. A sound report therefore distinguishes three questions: what is the group-specific rate, how many cases does it represent, and what is the consequence of the decision error in this setting?

Worked Example: Equal Error Rates, Different Error Burdens

Consider a hypothetical eligibility screen with the same estimated error rates in two groups. In Group C, 1,000 people include 100 actual positives and 900 actual negatives. The screen correctly flags 80 positives and incorrectly flags 90 negatives. In Group D, 1,000 people include 300 actual positives and 700 actual negatives. It correctly flags 240 positives and incorrectly flags 70 negatives. Suppose, solely for this example, that each false positive has an assigned consequence of \$500 and each false negative has an assigned consequence of \$3,000. Calculate the rates and the total assigned consequence in each group.

Method. Derive the remaining confusion-table counts, calculate each rate with its outcome denominator, and then multiply error counts by the stated hypothetical consequence amounts.

In Group C, $TP_C=80$, $FN_C=100-80=20$, $FP_C=90$, and $TN_C=900-90=810$. Therefore $\widehat{\operatorname{FPR}}_C=90/900=0.1000$, and $\widehat{\operatorname{FNR}}_C=20/100=0.2000$. The assigned consequence is $90(\$500)+20(\$3{,}000)=\$45{,}000+\$60{,}000=\$105{,}000$.

In Group D, $TP_D=240$, $FN_D=300-240=60$, $FP_D=70$, and $TN_D=700-70=630$. Thus $\widehat{\operatorname{FPR}}_D=70/700=0.1000$, and $\widehat{\operatorname{FNR}}_D=60/300=0.2000$. The assigned consequence is $70(\$500)+60(\$3{,}000)=\$35{,}000+\$180{,}000=\$215{,}000$.

Check. Both groups have a 10% false-positive rate and a 20% false-negative rate, but Group D has more false negatives because it has more actual positives. Its total assigned consequence is higher under the hypothetical cost assumptions. The calculation is not a claim that these dollar amounts are real or that all affected cases have identical consequences; it illustrates why rates, counts, and context belong together.

Pooling Can Hide the Comparison

A pooled rate combines error counts and denominators across groups. It is not generally the same as the unweighted average of group rates. The pooled rate gives more influence to groups with larger relevant denominators. Both calculations can be useful for different questions, but they must be labelled rather than substituted for one another.

Worked Example: Report Group Rates Before a Pooled Rate

A hypothetical quality review records the following counts. Group E has 400 actual negatives, of whom 40 receive a positive decision, and 100 actual positives, of whom 10 receive a negative decision. Group F has 100 actual negatives, of whom 20 receive a positive decision, and 100 actual positives, of whom 30 receive a negative decision. Calculate each group’s two error rates and approximate 95% intervals, then calculate the pooled false-positive rate. Use four decimals.

Method. Calculate groupwise rates first. Pool false positives over all actual negatives only after preserving the group results; do not average group percentages and label that value “pooled.”

For Group E, the false-positive rate is $40/400=0.1000$. Its standard error is $\sqrt{0.1(0.9)/400}=0.0150$, giving the interval $0.1000\pm1.96(0.0150)=(0.0706,0.1294)$. Its false-negative rate is $10/100=0.1000$. The standard error is $\sqrt{0.1(0.9)/100}=0.0300$, giving $(0.0412,0.1588)$.

For Group F, the false-positive rate is $20/100=0.2000$. Its standard error is $\sqrt{0.2(0.8)/100}=0.0400$, giving $(0.1216,0.2784)$. Its false-negative rate is $30/100=0.3000$. The standard error is $\sqrt{0.3(0.7)/100}=\sqrt{0.0021}=0.0458$. Using the unrounded value, the interval is $0.3000\pm1.96(0.0458258)=(0.2102,0.3898)$.

Across the groups, the pooled false-positive rate is $(40+20)/(400+100)=60/500=0.1200$. The unweighted average of the two group false-positive rates is $(0.1000+0.2000)/2=0.1500$, which is a different quantity. The pooled value is closer to Group E’s rate because Group E contributes four times as many actual negatives. This audit should retain both group rates rather than letting the pooled number stand in for them.

Check. The pooled numerator, 60 false positives, is divided by the pooled actual-negative denominator, 500. The interval calculations use the group-specific denominators. Keeping those operations separate prevents a weighted average from obscuring the group comparison.

Common Audit Traps

Using predicted positives as the false-positive denominator. The incorrect calculation $FP/(TP+FP)$ is positive predictive value, not false-positive rate. Correct it to $FP/(FP+TN)$, the number of errors among actual negatives.
Reporting rates without denominators. “The false-negative rate is 20%” is incomplete for an audit. Include the number of false negatives and the number of actual positives used to calculate the rate.
Treating a missing rate as zero. If there are no actual positives in a group, $FN/(TP+FN)$ has a zero denominator and is undefined.
Interpreting a pooled rate as a typical group rate. Pooling weights each group by its denominator. Show the group-specific estimates, and explicitly label any pooled summary.
Turning an interval into a fairness verdict. A confidence interval describes sampling uncertainty under assumptions. It does not establish that a decision process is fair, that labels are valid, or that the consequences are acceptable.

Key takeaway: A useful groupwise error audit reports counts, correct outcome-specific denominators, false-positive and false-negative rates, and appropriate uncertainty. Interpret those results alongside the number of people affected and the consequences of each error; no single rate or pooled summary provides the whole assessment.

Check Your Understanding

For each problem, identify the relevant denominator before calculating. Probabilities and interval endpoints should be reported to four decimals where applicable.

A hypothetical screening group has 240 actual negatives, including 36 false positives, and 80 actual positives, including 12 false negatives. Calculate both error rates. (Answer: FPR $=36/240=0.1500$; FNR $=12/80=0.1500$.)
In a group of 600 cases, 120 are actual positives and 480 are actual negatives. The decision correctly flags 90 actual positives and incorrectly flags 48 actual negatives. Calculate the false-positive rate, false-negative rate, and overall error rate. (Answer: FPR $=48/480=0.1000$; FNR $=(120-90)/120=0.2500$; overall error rate $=(48+30)/600=0.1300$.)
A group has 400 actual negatives and 80 false positives. Calculate the false-positive rate, its estimated standard error, and its approximate 95% interval. (Answer: rate $=0.2000$; standard error $=\sqrt{0.2(0.8)/400}=0.0200$; interval $(0.1608,0.2392)$.)
Group A has 200 actual negatives with 20 false positives and 100 actual positives with 10 false negatives. Group B has 300 actual negatives with 60 false positives and 150 actual positives with 30 false negatives. Find each group’s FPR and FNR, then calculate the pooled FPR and the unweighted average of the group FPRs. (Answer: A has FPR $0.1000$, FNR $0.1000$; B has FPR $0.2000$, FNR $0.2000$; pooled FPR $=(20+60)/(200+300)=0.1600$; unweighted average FPR $=(0.1000+0.2000)/2=0.1500$.)
In a hypothetical review, Group R has 50 false positives and 20 false negatives; Group S has 30 false positives and 40 false negatives. If the assigned consequence is \$400 per false positive and \$2,000 per false negative, calculate the total assigned consequence for each group. (Answer: Group R: $50(\$400)+20(\$2{,}000)=\$60{,}000$; Group S: $30(\$400)+40(\$2{,}000)=\$92{,}000$.)

What You'll Learn