Individual Fairness and Similarity Choices

Why Similarity Is Central to Individual Fairness

Group fairness measures compare outcomes or errors across groups. Individual fairness asks a different question: does a model treat individuals who are alike in ways relevant to the task alike in its predictions? This principle is intuitive, but it cannot be applied until “alike” and “alike predictions” are made precise. Those choices require a similarity measure for individuals and a measure of difference between model outputs.

The central difficulty is not usually evaluating a formula once those measures are chosen. It is justifying the measures themselves. A distance based on task-relevant, accurately measured features can support a useful audit. A distance that omits important differences or gives substantial weight to irrelevant differences can make the same model appear fair or unfair for reasons that do not fit the decision.

Definition: Let $x$ and $x'$ be two individuals’ represented profiles, $f(x)$ the model output for profile $x$, $d(x,x')$ a distance between profiles, and $D(f(x),f(x'))$ a distance between outputs. A Lipschitz form of individual fairness with constant $L \geq 0$ requires $$ D(f(x),f(x')) \leq Ld(x,x') $$ for every pair being assessed. The constant $L$ sets the maximum permitted change in output per unit of input distance. The condition does not determine what distance or value of $L$ is appropriate; those need a defensible, task-specific rationale.

What Makes a Distance a Metric?

A metric is a function $d$ that assigns a nonnegative distance to every pair of profiles. For all profiles $x,y,z$, it must satisfy nonnegativity, symmetry, identity of indiscernibles, and the triangle inequality. In particular, $d(x,y)=0$ if and only if $x=y$. A function that is merely nonnegative and symmetric is not necessarily a metric.

d(x,y) \geq 0,\qquad d(x,y)=d(y,x),\qquad d(x,y)=0 \text{ if and only if } x=y,\qquad d(x,z) \leq d(x,y)+d(y,z)

A metric defined on feature vectors may not distinguish every pair of actual people. Two people can have the same recorded values even if their unrecorded circumstances differ. In that situation, the formula is a metric on distinct feature vectors but acts as a pseudometric on people: distinct individuals can have distance zero. This is one reason an audit should describe exactly what profiles are compared, rather than implying that a data representation captures every relevant difference between people.

A common construction for $p$ numeric features is a weighted $L_1$ distance:

d(x,x')=\sum_{j=1}^{p}w_j|x_j-x'_j|,\qquad w_j>0

The positive weights set the importance and scale of each feature. If all weights are positive, this is a metric on the represented vectors: absolute differences are nonnegative and symmetric, and the sum is zero exactly when every coordinate agrees. The triangle inequality follows from $|x_j-z_j|\leq |x_j-y_j|+|y_j-z_j|$: multiplying each coordinate inequality by $w_j>0$ and summing gives $d(x,z)\leq d(x,y)+d(y,z)$. If a weight is zero, differences in that coordinate are ignored, so distinct vectors can have distance zero; the construction is then generally a pseudometric on the full vector space.

Units matter. A one-unit change in income, age, or claim amount does not have the same practical meaning. Standardizing features or choosing explicit weights can help, but neither step makes the weights automatically appropriate. The task and the consequences of the decision should determine which differences matter and how much.

Applying the Individual-Fairness Bound

For a numerical score, a natural output distance is the absolute difference between scores, $D(f(x),f(x'))=|f(x)-f(x')|$. The audit then checks whether each pair’s score difference is at most $L$ times its input distance. This is a pairwise condition: passing a test for a few selected pairs does not establish that the condition holds for every relevant pair.

The bound has a direct interpretation. If two profiles are within $\epsilon$ units under the chosen distance, then

d(x,x')\leq \epsilon \quad\Longrightarrow\quad D(f(x),f(x'))\leq L\epsilon.

This follows by substituting $d(x,x')\leq\epsilon$ into the individual-fairness condition and multiplying by $L\geq0$. Thus, the metric determines what “close” means, while $L$ determines how much prediction variation is allowed among close profiles.

Worked Example: Auditing Referral Scores for Claims

Method: calculate a task-specific weighted distance, then compare the score gap with its permitted bound. In a hypothetical claims review system, a profile uses estimated covered damage $z_1$, measured in thousands of dollars, and a count of verified documentation gaps $z_2$. The proposed distance and fairness constant are $$ d(x,x')=\frac{|z_1-z'_1|}{10}+\frac{|z_2-z'_2|}{2},\qquad L=0.20. $$ Two claims have profiles $A=(40,2)$ and $B=(45,3)$, and referral scores $f(A)=0.42$ and $f(B)=0.55$. Assume for this example that these features and weights have been justified for the review task.

Step 1: calculate the profile distance. The damage estimates differ by $45-40=5$ thousand dollars, and the documentation-gap counts differ by $3-2=1$. Therefore, $$ d(A,B)=\frac{5}{10}+\frac{1}{2}=0.5+0.5=1. $$

Step 2: calculate the permitted score difference. $$ Ld(A,B)=0.20(1)=0.20. $$

Step 3: compare the actual and permitted differences. The score gap is $|0.55-0.42|=0.13$, and $0.13\leq0.20$. This pair satisfies the proposed bound.

Check: The damage component contributes $0.5$ distance units and the documentation component contributes $0.5$, so their sum is $1$, not $0.5$. The common mistake is to check the score difference without first applying the stated feature scales and weights.

Similarity Must Match the Decision Task

The same two individuals can be close under one metric and far apart under another. Consider a hypothetical underwriting review in which the relevant prediction is a risk score. Suppose the task team has documented that, for the product and decision under review, predicted annual claim cost and years of verified exposure are appropriate dimensions for comparing profiles. A distance using only these features may be more informative than one that also gives substantial weight to geographical separation when location has already been accounted for in the risk assessment.

That conclusion is not a universal rule about location. In another task, location could be important. Rather, the example shows why the feature set and weights need to be justified in context. A metric can be mathematically valid but substantively poor. Conversely, a carefully chosen task-relevant metric does not by itself establish that the model or decision process is fair in every other sense.

Worked Example: How a Similarity Choice Changes the Audit

Method: evaluate the same pair under two explicit metrics and compare each resulting bound. In a hypothetical review, profile $P$ has predicted annual claim cost of $12$ thousand dollars and $4$ years of verified exposure. Profile $Q$ has cost of $13$ thousand dollars and $4$ years of exposure. Their model scores are $0.31$ and $0.43$. Let $L=0.50$. A task-focused distance is $$ d_{\text{task}}(x,x')=\frac{|c-c'|}{5}+\frac{|e-e'|}{4}, $$ where $c$ is cost in thousands of dollars and $e$ is exposure in years.

Step 1: apply the task-focused distance. $$ d_{\text{task}}(P,Q)=\frac{|12-13|}{5}+\frac{|4-4|}{4} =\frac{1}{5}+0=0.20. $$ The allowance is $Ld_{\text{task}}(P,Q)=0.50(0.20)=0.10$. The score gap is $|0.43-0.31|=0.12$, and $0.12>0.10$. The pair does not satisfy the bound under this metric.

Step 2: consider a different, explicitly defined distance. Suppose an alternative metric adds geographical separation, measured in miles, divided by $5$. The profiles are $20$ miles apart, so $$ d_{\text{alt}}(P,Q)=d_{\text{task}}(P,Q)+\frac{20}{5}=0.20+4=4.20. $$ The alternative allowance is $0.50(4.20)=2.10$. Since $0.12\leq2.10$, this pair satisfies the bound under the alternative metric.

Interpretation and check: The score gap is unchanged; only the similarity judgment changed. The geographical term makes these profiles appear far apart and therefore permits a larger score difference. If geographical separation is not relevant to this particular review task, that term weakens the audit rather than improving it. The common mistake is to report that a model “passes” without stating the metric that made the comparison possible.

Scores and Decisions Are Different Outputs

Individual fairness depends on what counts as the model output. A score may change gradually with the inputs, while a threshold converts that score into a discrete decision. The thresholded decision can change abruptly for two profiles whose scores are nearly identical. An audit of scores therefore does not automatically establish that the resulting decisions satisfy the same fairness bound.

Worked Example: A Threshold Can Separate Nearly Equal Scores

Method: check the score bound and the decision bound separately. A hypothetical outreach model produces a score $f(z)=z$ from a one-dimensional validated risk index $z$. For two profiles, $z_A=0.499$ and $z_B=0.501$. Use input distance $d(A,B)=|z_A-z_B|$, score-output distance equal to absolute score difference, $L=1$, and a decision rule that selects a person when the score is at least $0.500$.

Step 1: check the score condition. The input distance is $|0.501-0.499|=0.002$. The score difference is also $0.002$, so $$ |f(A)-f(B)|=0.002\leq 1(0.002)=0.002. $$ The pair satisfies the score-level bound, exactly at its limit.

Step 2: check the binary decision condition. Profile $A$ is below $0.500$, so its decision is $0$; profile $B$ is at least $0.500$, so its decision is $1$. With absolute difference as the binary-output distance, the decision gap is $|1-0|=1$. The permitted difference remains $Ld(A,B)=0.002$, and $1>0.002$. The pair does not satisfy that same bound when the output is the thresholded decision.

Check: The scores differ by two thousandths, but the decisions differ by one full binary unit. The common mistake is to conclude that a score-level fairness check also certifies a decision-level check. The output being audited must be stated explicitly.

Using the Principle Responsibly

A practical individual-fairness assessment should make its choices visible. It should specify which profiles are compared, which features enter the distance, how features are scaled, the output being audited, and the constant $L$. It should also explain why the features and weights fit the task, and examine whether measurement problems could distort the distances. Earlier material on measurement bias and proxy variables is relevant when deciding whether recorded features represent the intended differences reliably.

The principle is a standard for relating predictions to a chosen notion of similarity; it does not create a uniquely correct notion of similarity. Nor does satisfying a pairwise bound prove that a system has no other fairness problems. For example, if the distance omits a difference that matters to the task, profiles may be treated as similar when they should not be. If the distance emphasizes an irrelevant difference, it can excuse large prediction gaps. Any conclusion should therefore be reported with the metric and assumptions that produced it.

Individual fairness and counterfactual fairness answer related but distinct questions. Counterfactual fairness asks how a prediction for a person would compare under a specified counterfactual change. Individual fairness compares predictions across profiles using a specified distance. Neither comparison can be interpreted without the assumptions that define what is being held fixed and what differences matter.

Define the task and output.
State what prediction or decision is being assessed and identify the population of profiles to compare.

Specify and justify similarity.
Name the features, units, scaling, and weights; explain why they represent task-relevant differences.

Set the comparison standard.
State $L$ and the distance used for outputs. Explain what prediction difference is permitted for a given input distance.

Audit pairs and report limitations.
Calculate the bound for relevant comparisons, distinguish score results from decision results, and describe what the metric cannot capture.

Common Exam Traps

Calling an incomplete function a metric. Nonnegativity, symmetry, and the triangle inequality are not enough by themselves. A metric also requires $d(x,y)=0$ if and only if $x=y$, including $d(x,x)=0$.
Treating a valid metric as automatically fair. Mathematical properties do not establish that the features or weights represent task-relevant similarity.
Reversing the inequality. Individual fairness requires the output difference to be no greater than $Ld(x,x')$. A larger actual difference fails the stated bound.
Ignoring units or weights. A raw difference of one unit can mean very different things for different features; compute the stated scaled distance before comparing scores.
Confusing score fairness with decision fairness. Thresholding can turn a small score difference into different decisions, so identify which output the fairness condition concerns.

Key takeaway: Individual fairness is a pairwise prediction bound relative to a chosen notion of similarity. The calculation is only as useful as the task relevance of that similarity measure, the justification for $L$, and the clarity about whether scores or decisions are being compared.

Check Your Understanding

For each problem, use the specified distance and output. A pair satisfies the condition only when the actual output difference is no greater than $Ld(x,x')$.

A hypothetical model uses $d(x,x')=|a-a'|/5+|b-b'|/2$, with $L=0.30$. Two profiles differ by $2$ units in $a$ and $1$ unit in $b$. Their scores are $0.40$ and $0.58$. Does the pair satisfy the score bound? (Answer: Yes; $d=2/5+1/2=0.9$, the allowance is $0.30(0.9)=0.27$, and the score gap is $0.18$.)
For a function to be a metric, is it enough that it is nonnegative, symmetric, and satisfies the triangle inequality? State the missing requirement. (Answer: No; it must also satisfy $d(x,y)=0$ if and only if $x=y$, including $d(x,x)=0$.)
A score model has $L=0.40$. For a pair at distance $0.20$, the scores are $0.32$ and $0.39$. Does the pair satisfy the bound? (Answer: Yes; the score gap is $0.07$ and the allowance is $0.40(0.20)=0.08$.)
A score model has $L=0.50$. Two profiles are at distance $0.04$, with scores $0.48$ and $0.52$. A decision is positive when the score is at least $0.50$. Does the pair satisfy the score bound, and do the binary decisions satisfy the same bound using absolute difference? (Answer: The scores do not satisfy it: the gap is $0.04$, greater than the allowance $0.50(0.04)=0.02$. The decisions also do not satisfy it: their gap is $1$, also greater than $0.02$.)
Two profiles have distance $1.5$, score difference $0.10$, and $L=0.08$. Does the pair satisfy the condition? How would the conclusion change if the distance were $1.0$, with all else fixed? (Answer: At distance $1.5$, yes: the allowance is $0.08(1.5)=0.12$, which exceeds $0.10$. At distance $1.0$, no: the allowance is $0.08$, below $0.10$.)

What You'll Learn