Why Audit Intersections?
A model can appear to treat age groups similarly and to treat sex groups similarly, while a particular combination of age and sex experiences a very different outcome. Marginal summaries combine people across one attribute at a time; that aggregation can conceal disparities within the joint groups. An intersectional diagnostic makes those joint comparisons visible.
The purpose is not to assume in advance that every combination has a distinct problem. It is to check whether the groups relevant to the decision show patterns that an overall or one-attribute-at-a-time summary would miss. This is especially important when a model is used in a hypothetical insurance application, claim-review, or service-access decision. Results should be interpreted in light of how the data were collected, how outcomes were measured, and which people are represented.
Define the Metric Before Comparing Groups
An intersectional comparison is meaningful only when the metric and its denominator are clear. Let \(G=g\) identify membership in a group, let \(D=1\) mean the model makes a positive decision, and let \(Y=1\) mean the observed outcome is positive. The empirical selection rate is the proportion of group members receiving a positive decision. It is not, by itself, an error rate: it does not account for whether the observed outcome was positive.
When observed outcomes are available and suitable for the question, calculate conditional error metrics as well. The true positive rate (TPR) is the proportion of people with \(Y=1\) who receive \(D=1\). The false positive rate (FPR) is the proportion of people with \(Y=0\) who receive \(D=1\). Their denominators differ, so neither can be calculated by dividing the number of positive decisions by the total group size.
These are descriptive estimates from the observed data. Their interpretation depends on the meaning and quality of \(Y\), the decision process that produced \(D\), and who enters the data. Earlier material on measurement bias, selection bias, and group error rates is relevant when deciding whether the observed metrics answer the intended audit question. A difference in a metric identifies a pattern to investigate; by itself, it does not establish why that pattern occurred.
Worked Example: Marginal Parity Hides an Intersectional Gap
In a hypothetical automated application-screening audit, each of four age-by-sex groups contains 100 applicants. The recorded positive-decision counts are 30 for younger women, 70 for younger men, 70 for older women, and 30 for older men. Calculate the four intersectional selection rates and the age and sex marginal rates. State what the comparisons show. Method: calculate each rate using its own group count, then aggregate counts before calculating each marginal rate. This avoids assuming that marginal parity guarantees parity at the intersections.
For younger women, the selection rate is \(30/100=30\%\). For younger men, it is \(70/100=70\%\). For older women, it is \(70/100=70\%\). For older men, it is \(30/100=30\%\).
For younger applicants, there are \(30+70=100\) positive decisions among \(100+100=200\) applicants, so the rate is \(100/200=50\%\). For older applicants, the calculation is \((70+30)/(100+100)=100/200=50\%\). For women, it is \((30+70)/(100+100)=100/200=50\%\); for men, it is \((70+30)/(100+100)=100/200=50\%\).
Thus both age and sex marginal comparisons show equal selection rates, while the intersectional rates range from 30% to 70%, a 40-percentage-point gap. The overall rate is \((30+70+70+30)/400=200/400=50\%\), which also conceals the differences. These synthetic counts demonstrate why marginal parity does not rule out intersectional disparities. They do not identify a cause or, on their own, establish that any specific process is unfair.
Aggregation Requires the Correct Weights
A marginal rate is not generally the unweighted average of the rates for its component groups. It is a weighted average, where each subgroup's weight is its share of the marginal population. This follows directly from the definition: the total number of positive decisions is the sum of subgroup positive counts, and the total denominator is the sum of subgroup denominators.
Here \(n_j\) is the number of people in component group \(j\), and \(\widehat{r}_j\) is its selection rate. Since the number of positive decisions in that group is \(n_j\widehat{r}_j\), adding positive counts gives the numerator \(\sum_j n_j\widehat{r}_j\). Adding group sizes gives the denominator \(\sum_j n_j\). Dividing those totals proves the formula. The same reasoning applies to other rates only when the relevant denominators are combined correctly; for example, a marginal TPR must pool actual positive outcomes, not all people in each group.
Worked Example: Pooling Rates With Unequal Group Sizes
A hypothetical review contains 20 applicants in intersectional group A, of whom 12 receive positive decisions, and 80 in group B, of whom 32 receive positive decisions. Calculate both group rates and the pooled rate. Explain why the unweighted average is wrong. Method: sum the positive-decision counts and group sizes. This avoids giving a small group the same weight as a much larger group.
Group A's rate is \(12/20=60\%\). Group B's rate is \(32/80=40\%\). The pooled rate is \((12+32)/(20+80)=44/100=44\%\).
The unweighted average of the subgroup rates is \((60\%+40\%)/2=50\%\), but it does not represent the 100 applicants: it gives the 20-person group the same influence as the 80-person group. Using the required weights gives \((20(0.60)+80(0.40))/(20+80)=(12+32)/100=44\%\). The result is checked against the direct count, \(44/100=44\%\).
Check Error Rates at the Intersections
Selection-rate comparisons answer whether positive decisions occur at different frequencies. They do not answer whether errors differ. When the observed outcome is appropriate for evaluation, calculate TPR and FPR separately for each intersectional group. Then compare both the intersectional rates and the corresponding marginal rates. The following example uses equal denominators to make the aggregation transparent; real audits should use the actual denominators in each group.
Worked Example: Equal Marginal Error Rates, Different Intersectional TPRs
In a hypothetical model audit, each age-by-sex group has 20 people with \(Y=1\) and 30 with \(Y=0\). The true positive counts for younger women, younger men, older women, and older men are 8, 18, 18, and 8, respectively. Each group has 3 false positives among its 30 people with \(Y=0\). Calculate the intersectional TPRs and FPRs, then calculate the age and sex marginal TPRs. Method: keep actual positives and actual negatives as separate denominators. This avoids treating a selection rate as an error rate.
The intersectional TPRs are \(8/20=40\%\) for younger women, \(18/20=90\%\) for younger men, \(18/20=90\%\) for older women, and \(8/20=40\%\) for older men. Each intersectional FPR is \(3/30=10\%\).
For younger applicants, the pooled TPR is \((8+18)/(20+20)=26/40=65\%\). For older applicants, it is \((18+8)/(20+20)=26/40=65\%\). For women, it is \((8+18)/40=65\%\); for men, it is \((18+8)/40=65\%\). Each age and sex marginal FPR is \((3+3)/(30+30)=6/60=10\%\).
The marginal TPRs and FPRs match across the displayed age and sex categories, yet the intersectional TPRs range from 40% to 90%. This is an important diagnostic result: summaries at one attribute at a time miss a 50-percentage-point difference between intersections. The FPRs are equal in this constructed example, but that does not remove the TPR pattern or explain its cause. Each calculation is checked using its outcome-specific denominator.
Sparse Intersections Limit What the Data Can Show
Adding attributes creates more intersectional groups. That can reveal meaningful variation, but it can also leave some groups with very few observations or very few relevant outcomes. A rate based on a small denominator can change sharply when only one or two records change. Therefore, report the numerator and denominator alongside the rate, and distinguish an observed difference from a well-supported conclusion about the underlying process.
For error-rate comparisons, sparsity must be assessed using the metric's denominator. A group may have a reasonable total sample size but very few observed positives, making its TPR unstable; the same group may have few observed negatives, making its FPR unstable. A rate without its denominator can hide this limitation. Earlier material on uncertainty in small-group estimates provides the broader uncertainty framework; here the key intersectional point is that cross-classifying attributes can make the relevant denominators especially small.
Worked Example: Interpreting a Rate From a Sparse Intersection
A hypothetical audit reports positive decisions for 4 of 5 people in intersectional group A, 12 of 20 in group B, 60 of 100 in group C, and 120 of 200 in group D. Calculate all four rates and the pooled rate. Then assess whether the 20-percentage-point difference between A and the other groups is strong evidence of a stable underlying disparity. Method: show each count with its denominator, pool counts for the overall rate, and treat the small-cell rate as uncertain rather than as a precise group characteristic. This avoids interpreting a striking percentage as conclusive when it is based on very few records.
The rates are: group A, \(4/5=80\%\); group B, \(12/20=60\%\); group C, \(60/100=60\%\); and group D, \(120/200=60\%\). The pooled rate is \((4+12+60+120)/(5+20+100+200)=196/325\), which is approximately 60.31% to two decimal places.
The observed difference between A and each other group is \(80\%-60\%=20\) percentage points. But A's rate is based on only five people: changing one outcome changes its rate by \(1/5=20\) percentage points. That sensitivity means the observed rate alone is not strong evidence that the underlying rate for group A is stably higher. A fuller analysis would quantify uncertainty with an appropriate method and consider the data collection and outcome limitations. The point estimate is a reason to examine the pattern, not a basis for certainty about its cause or persistence.
A Practical Diagnostic Sequence
A clear intersectional audit makes its choices explicit and keeps descriptive results separate from explanations. The following sequence is suitable for a first diagnostic pass and is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Define what counts as a positive decision and, if evaluating errors, what observed outcome is used. State any limitations in how that outcome was recorded.
List the categories being cross-classified and the resulting intersections. Check that the categories are clear and that records can be assigned consistently.
Report counts and rates for intersections and relevant marginal groups. Use total counts to form marginal rates rather than averaging subgroup percentages without weights.
Identify small denominators, particularly for outcome-conditioned error rates. Consider how data representation, measurement, and selection into the observed sample affect interpretation.
Describe the measured pattern and its uncertainty. Do not treat a rate difference as proof of its cause, and do not treat a small or noisy difference as proof that no disparity exists.
Common Exam Traps and Reporting
The central trap is to treat a reassuring marginal comparison as proof that every intersection is similar. Another is to compare percentages without checking their denominators. A selection rate uses all group members; a TPR uses observed positives; an FPR uses observed negatives. Mixing these quantities creates a comparison that answers no well-defined question.
- Do not average percentages blindly. Aggregate counts and denominators, or use the corresponding denominator weights.
- Do not omit cell sizes. A percentage from a small intersection can be highly sensitive to a few records.
- Do not claim causation from a disparity. The diagnostic describes observed outcomes; explanation requires additional evidence about the data and decision process.
- Do not interpret absence of a visible gap as proof of equivalence. Sparse or noisy data may not reveal a real difference.
A useful report states the joint-group definitions, the metric, each numerator and denominator, the observed disparities, and the uncertainty or data limitations. It also distinguishes the descriptive finding from any proposed explanation. This gives a non-technical reader a fair account of both what the audit reveals and what the available data cannot establish.
Check Your Understanding
For each problem, calculate the requested rates from the stated counts and check whether the marginal summary conceals an intersectional pattern.
- A hypothetical screening system has four equal-sized intersections with selection rates of 40%, 60%, 60%, and 40%, arranged so that each age category and each sex category contains one 40% group and one 60% group. What are the age and sex marginal rates, and what is the range of intersectional rates? (Answer: each marginal rate is 50%; intersectional rates range from 40% to 60%, a 20-percentage-point gap.)
- Group A has 18 positive decisions among 30 people; group B has 14 among 70. Calculate both rates and the pooled rate. Why is the unweighted average not the pooled rate? (Answer: A is 60%, B is 20%, pooled is \(32/100=32\%\); the unweighted average is 40% because it gives the 30-person and 70-person groups equal weight.)
- For a particular intersection, 9 of 15 people with \(Y=1\) receive \(D=1\), and 4 of 25 people with \(Y=0\) receive \(D=1\). Calculate the TPR and FPR. (Answer: TPR \(=9/15=60\%\); FPR \(=4/25=16\%\).)
- Two intersections each have 40 observed positives. The first has 24 true positives; the second has 32. Calculate their TPRs and the difference in percentage points. (Answer: 60% and 80%, respectively; the second is 20 percentage points higher.)
- A small intersection has 3 positive decisions among 4 people; three larger intersections each have 30 among 50. Calculate the small-group rate and pooled rate, then state one reason not to treat the small-group difference as conclusive. (Answer: small-group rate \(=75\%\); pooled rate \(=(3+90)/(4+150)=93/154\), approximately 60.39% to two decimal places; the small rate is based on only four people, so one changed outcome shifts it by 25 percentage points.)