Why Group Definitions Matter
A fairness audit can only compare groups that have been defined clearly. The same person might be classified by language at application, residence at the time of a decision, or a later updated record. Each choice can produce a different group count and a different interpretation. A group definition is therefore not a neutral formatting decision: it determines who is included in each comparison and what the comparison can reasonably say.
The audit should start with the decision and the people affected by it. For example, an audit of eligibility decisions should identify the applications subject to that decision, not a broader population that was never eligible to apply. It should also specify which recorded attributes are relevant to the question. A group definition that is easy to calculate but disconnected from the decision can produce a precise answer to the wrong question.
For example, “rural applicants” is incomplete unless the audit says how rural status is determined, which address is used, and whether status is measured at application or at decision time. Likewise, “older customers” needs a boundary and an age-measurement date. Explicit rules let another analyst reproduce the groups and let decision-makers see what the results do—and do not—represent.
Choose Groups in the Context of the Decision
A useful group attribute has a defensible connection to the decision, the potential harm being examined, or the way people experience the process. The attribute need not be a direct input to a model to matter to an audit. Conversely, a convenient field in the data is not automatically a suitable basis for comparison. Earlier material on protected attributes, representativeness, measurement bias, and proxy variables provides relevant context for assessing these choices.
Before calculating any group summaries, document the following:
- Decision and population: Which decision is being audited, and which cases were eligible to receive it?
- Attribute and rule: Which attribute defines the groups, and how are its values categorized?
- Measurement time: Is the attribute measured at application, decision, or another specified time?
- Missing and ambiguous values: Are they retained as a separate category, investigated, or excluded for a stated reason?
- Reference group: If one group is used as a comparator, why is that comparison useful for this decision?
- Small categories: Are they kept distinct, combined under a meaningful rule, or handled differently in public reporting?
A reference group is a selected comparator, not a declaration that its members are typical, correct, or entitled to better treatment. It may be operationally useful to compare groups with the same service pathway, or to use a historical baseline when the audit question is about change. In either case, explain the reason and retain enough context to avoid presenting the comparator as a universal standard. Where useful, show more than one comparison rather than allowing an arbitrary reference choice to shape the entire interpretation.
Worked Example: Defining Language Groups for an Application Audit
A hypothetical public-benefit office wants to audit a model-assisted appointment-priority decision. There are 240 eligible applications in the quarter. Language preference recorded at application is English for 144 cases, Spanish for 48, Arabic for 24, and another listed language for 12; 12 applications have no recorded language. Define defensible groups and identify a suitable reference choice for describing the audit population.
Step 1: Fix the population. The audit population is all 240 eligible applications that received the decision during the quarter. It is not only the applications with a known language field: excluding the 12 missing records would silently change the population.
Step 2: State the measurement rule. Use language preference as recorded at application, before the priority decision. This is appropriate if the audit question concerns the application-stage service experience. The rule does not claim that the field captures every language a person uses.
Step 3: Check the counts. The recorded categories sum to \(144+48+24+12=228\). Adding the 12 missing records gives \(228+12=240\), which matches the audit population. The known-language share is \(228/240=0.95\), or 95%; the missing share is \(12/240=0.05\), or 5%.
Step 4: Preserve distinctions in the audit definition. English, Spanish, Arabic, other listed language, and missing are five distinct categories. Combining Spanish, Arabic, and other languages merely to make a simpler table would erase potentially relevant differences in access. The audit team can examine whether the small “other” category supports a meaningful separate analysis before deciding how to display results.
Step 5: Choose and describe a comparator. English may be a practical reference if the office’s standard application workflow is delivered in English. That choice must be explained as an operational comparison, not as a claim that English speakers are the norm. The audit record should also show the other group definitions and the missing count.
Check. All five categories account for exactly 240 applications, and the reference choice has a stated connection to the process. The common mistake avoided is dropping missing records and then describing 228 cases as if they were the full eligible population.
Changing Membership and Time-Specific Rules
Some attributes change. A person can move, change a recorded preference, or have a status updated between application and decision. For a given audit, group membership should follow a rule tied to the decision under review. If the relevant question concerns the information available when the decision was made, using a value recorded only afterward can misrepresent the decision context. If the question concerns a later service experience, a later measurement may be relevant instead.
When membership can change, preserve the measurement date and, where feasible, the earlier and later values. This makes it possible to distinguish a change in group composition from a change in how cases were treated. A stable rule does not require people’s attributes to remain unchanged; it requires the audit to say which time-specific value determines each assignment.
Worked Example: Residence Changes Before a Decision
A hypothetical housing-support program reviews 100 applications. At submission, 40 applicants are classified as rural and 60 as urban. Before the decision date, 8 rural applicants move to urban areas and 4 urban applicants move to rural areas. The audit concerns access to the decision process as experienced at decision time. Find the group counts under both possible timing rules and recommend a definition.
Method. Apply the recorded movements to the submission counts, counting each move once. Under the decision-date rule, rural membership loses 8 and gains 4: \(40-8+4=36\). Urban membership loses 4 and gains 8: \(60-4+8=64\).
Check the total. The decision-date counts sum to \(36+64=100\), the full audit population. Under the submission-date rule the counts remain 40 rural and 60 urban, also totaling 100.
Because the stated audit question concerns experience at decision time, decision-date residence is a defensible primary definition, provided that the program reliably records that address by the decision date. The report should state the date rule and note that a submission-date definition would instead produce counts of 40 and 60. If the decision itself used the application-time address, that earlier value may be more relevant to auditing the decision inputs. The appropriate timing rule follows the audit question, not a preference for whichever counts look simpler.
Check. Both definitions cover the same 100 applicants, but they assign 4 net applicants differently between groups. The common mistake avoided is presenting one set of counts without saying when residence was measured.
A Count Check That Clarifies the Comparison
When groups form a complete partition of the audit population, every case belongs to exactly one group. If a descriptive event is recorded for each case, the overall event proportion is a weighted average of the group-specific proportions. This identity is useful for checking denominators and understanding how group composition contributes to an overall summary. It is a bookkeeping result, not by itself a fairness standard.
To justify the result, the total number of event cases is \(\sum_g y_g\), because each case belongs to one and only one group. The overall proportion is therefore \(\sum_g y_g/n\). Since \(y_g=n_g r_g\), substitution gives \(\sum_g n_g r_g/n=\sum_g(n_g/n)r_g\). The weights \(n_g/n\) sum to one because the groups cover the population without overlap.
If categories overlap, some cases can be counted more than once and this derivation no longer applies. If cases are omitted—for example, because missing attributes were dropped—the calculation describes a restricted population, not necessarily the original audit population. State the population and group assignment rule before using any such summary.
Worked Example: Checking Group Counts and an Overall Summary
A hypothetical equipment-inspection service audits 120 completed inspections. By a documented site classification, 72 are at urban sites and 48 are at rural sites. A recorded follow-up event occurred in 18 urban cases and 18 rural cases. Calculate the event proportions by group and overall, then verify the weighted-average identity.
Urban group: \(18/72=0.25\), or 25%.
Rural group: \(18/48=0.375\), or 37.5%.
Overall: there are \(18+18=36\) events among \(72+48=120\) inspections, so the overall proportion is \(36/120=0.30\), or 30%.
The group weights are \(72/120=0.60\) and \(48/120=0.40\). Their weighted average is \(0.60(0.25)+0.40(0.375)=0.15+0.15=0.30\), matching the direct overall calculation. The counts also check: \(72+48=120\), and the event count \(18+18=36\).
The method is a denominator and partition check. It avoids treating the unweighted average \((25\%+37.5\%)/2=31.25\%\) as the overall proportion. That average gives equal weight to groups of different sizes, so it does not represent the 120 inspections.
Small Categories: Preserve Meaning, Protect People
A small category creates two distinct concerns. First, its cases may be substantively important even if there are few of them; automatically merging them into a larger category can hide a relevant difference. Second, detailed reporting may reveal information about individuals, particularly when categories are rare or combined with other attributes. These concerns should be handled explicitly rather than by quietly changing group definitions after seeing results.
There is no universal sample-size rule that decides whether a category should be combined. A defensible process distinguishes the internal audit definition from the public presentation. The audit can retain a meaningful small group for review while limiting the detail disclosed externally. If pooling is needed, the combined categories should share a reason relevant to the decision—for example, a genuinely common service pathway—and the pooled label should describe what it contains. Do not imply that unlike groups are interchangeable simply because their counts are small.
If a reporting policy sets a minimum cell size, apply it consistently and check whether other published totals would allow a suppressed count to be inferred. Document the rule, the categories affected, and whether any pooling is only for display. Keep the underlying group definitions available to authorized reviewers where appropriate.
Worked Example: Handling a Rare Geographic Category
A hypothetical claims-support audit includes 400 cases from four regions: North, 182; South, 151; Coast, 58; and Island, 9. The audit team’s written public-reporting policy says not to publish a category count below 10. The Island region has a distinct claims-routing process. Decide how to define the audit groups and what to do with the public count.
Step 1: Verify the population count. The regions sum to \(182+151+58+9=400\), so they account for the stated audit population.
Step 2: Apply the reporting rule. Island has 9 cases, and \(9<10\), so its count does not meet the stated public-reporting threshold. North, South, and Coast have counts of at least 10.
Step 3: Keep the audit definition meaningful. Because Island has a distinct routing process, combining it with Coast merely because both are coastal would blur a process difference. Retain Island as its own group in the controlled audit analysis. Do not describe the public display decision as evidence that Island is substantively equivalent to another region.
Step 4: Protect the public presentation. Withhold or aggregate the Island count in a way consistent with the policy, and check whether the remaining displayed totals would let a reader calculate it by subtraction. If they would, additional suppression or a broader presentation is needed. The policy governs disclosure, not the definition used by the audit team.
Check. The four internal groups still sum to 400, while the public count for Island is not released. The common mistake avoided is merging a distinct group into another category solely to make its count publishable.
Common Traps and a Defensible Audit Record
- Changing the population without saying so. Excluding cases with missing group attributes changes the population being described. Report the missing count and explain any exclusion.
- Choosing a comparator by habit. A majority group is not automatically the most relevant reference. Explain the operational or historical reason for the choice.
- Ignoring measurement time. “Residence” or “language” is not enough; specify when it was recorded and which value determines membership.
- Pooling unlike categories for convenience. A larger count does not make a combined group meaningful. State a substantive rationale and retain the component definitions.
- Confusing audit groups with disclosure groups. A privacy-driven public aggregation need not replace the finer internal audit definition.
- Reading group counts as a complete fairness conclusion. Counts describe who is represented in a comparison; they do not alone establish whether a decision process is fair.
A defensible audit record states the decision, population, group attribute, category boundaries, measurement date, missing-value treatment, reference-group rationale, and handling of small categories. It also checks that group counts reconcile to the intended population and explains any difference between internal analysis and public reporting. These steps make later interpretation more transparent without treating the group definition itself as a fairness verdict.
Check Your Understanding
For each problem, identify the group-definition issue and give the requested calculation or recommendation.
- A hypothetical permit office audits 180 decisions: 90 applications were submitted online, 66 in person, and 18 have no recorded submission channel. What fraction of the audit population has a missing channel, and what should the audit state about those cases? (Answer: \(18/180=0.10\), or 10%; retain and report the missing category or justify any exclusion rather than silently dropping it.)
- At application, a service has 55 rural and 45 urban applicants. Before decisions, 6 rural applicants move to urban areas and 3 urban applicants move to rural areas. What are the decision-date counts, and what check should be applied? (Answer: rural \(55-6+3=52\), urban \(45-3+6=48\); check \(52+48=100\), and state which date defines membership.)
- A hypothetical audit has three mutually exclusive groups with sizes 50, 30, and 20. Their recorded event proportions are 0.20, 0.10, and 0.40. Find the overall event proportion using the group weights. (Answer: \((50/100)(0.20)+(30/100)(0.10)+(20/100)(0.40)=0.10+0.03+0.08=0.21\), or 21%.)
- A public report has a written minimum display count of 12. A meaningful group has 10 cases and a distinct review pathway; the only other group has 40 cases and a different pathway. What is a defensible approach? (Answer: retain the 10-case group in the appropriately controlled audit; withhold or aggregate its public detail under the stated policy, check whether other totals reveal its count, and do not merge it with the unlike pathway merely for convenience.)
- An audit compares a reference group of 120 cases with another group of 30. The reference group is selected because it uses the standard service pathway. What must the report avoid implying, and what should it explain? (Answer: avoid implying the reference group is inherently normal or fair; explain the pathway-based reason for choosing it and specify the group definitions, population, and measurement timing.)