Misclassification and Differential Reporting Error

Why Observed Categories Can Mislead

Economic data often turn complex situations into categories: employed or not employed, insured or uninsured, received a benefit or did not. A recorded category is not automatically the true category. A respondent may misunderstand a question, a survey rule may assign a label imperfectly, or people in different circumstances may report the same underlying fact differently. The resulting observed pattern can differ from the pattern in the underlying concept.

This tutorial distinguishes misclassification, in which a categorical value is recorded in the wrong category, from the broader idea of reporting error, in which a reported value differs from the underlying value. It then shows why it matters whether such errors vary across groups or circumstances. These are measurement concerns: by themselves, they do not establish whether a group difference is causal.

Definition: Let $T$ denote a true or target category and $W$ its observed, recorded category. Misclassification occurs when $W\ne T$. For a binary category, call one category positive and the other negative. Sensitivity is $P(W=1\mid T=1)$, the probability of recording positive among those truly positive. Specificity is $P(W=0\mid T=0)$, the probability of recording negative among those truly negative. Error is differential across a group or circumstance when the distribution of recorded values conditional on the target value differs across that group or circumstance.

Confusion Tables for Binary Categories

A confusion table compares the target category with the recorded category. The target value may be known from a validation measure, an administrative record, or a carefully defined benchmark. Without such information, a table of self-reports alone does not reveal which reports are misclassified.

	Observed positive $W=1$	Observed negative $W=0$	Total
True positive $T=1$	True positive: $TP$	False negative: $FN$	$TP+FN$
True negative $T=0$	False positive: $FP$	True negative: $TN$	$FP+TN$
Total	$TP+FP$	$FN+TN$	$n$

The sensitivity is $TP/(TP+FN)$, and the specificity is $TN/(TN+FP)$. The false-negative rate is $1-\text{sensitivity}$; the false-positive rate is $1-\text{specificity}$. These rates have different denominators. Sensitivity asks how often true positives are recognized, while the false-positive rate asks how often true negatives are recorded as positive.

Worked Example: Self-Reported Union Membership

Suppose a hypothetical validation exercise compares workers’ self-reports of union membership with a reliable membership record. Among 1,000 workers, the record identifies 200 members and 800 nonmembers. The comparison gives 180 true positives, 20 false negatives, 40 false positives, and 760 true negatives. Calculate the sensitivity, specificity, and observed membership proportion.

The table entries are internally consistent: $180+20=200$ true members, $40+760=800$ true nonmembers, and $180+20+40+760=1{,}000$ workers. Sensitivity is

\frac{TP}{TP+FN}=\frac{180}{180+20}=0.900.

Thus, 90.0% of true members were reported as members. Specificity is

\frac{TN}{TN+FP}=\frac{760}{760+40}=0.950.

Thus, 95.0% of true nonmembers were reported as nonmembers. There are $180+40=220$ observed positive reports, so the observed membership proportion is $220/1{,}000=0.220$, or 22.0%. The target proportion is $200/1{,}000=0.200$, or 20.0%. In this hypothetical sample, false positives more than offset false negatives, making the observed proportion 2.0 percentage points higher than the target proportion.

How Misclassification Changes a Proportion

Let $p=P(T=1)$ be the target prevalence, and let $p_W=P(W=1)$ be the observed proportion recorded positive. If sensitivity and specificity apply to the population being described, then an observed positive can arise either from a true positive recorded correctly or a true negative recorded incorrectly. Therefore,

p_W=P(W=1\mid T=1)P(T=1)+P(W=1\mid T=0)P(T=0) =\text{sensitivity}\cdot p+(1-\text{specificity})(1-p).

This identity makes the role of both error types explicit. A high sensitivity does not by itself guarantee a small upward error: if the target category is uncommon, even a modest false-positive rate can contribute many observed positives. Conversely, missed true positives can lower the observed proportion.

If sensitivity and specificity are known, constant for the target population, and satisfy $\text{sensitivity}+\text{specificity}>1$, this equation can be rearranged to recover $p$:

p=\frac{p_W+\text{specificity}-1}{\text{sensitivity}+\text{specificity}-1}.

This is an algebraic correction, not a guarantee of accurate measurement. It is useful only if the error rates are credible for the population and measurement process in question. If the rates are estimated imprecisely or vary across groups, treating them as fixed common rates can itself mislead.

Worked Example: Correcting an Observed Proportion

Return to the hypothetical membership data. The observed proportion is $p_W=0.220$, sensitivity is 0.900, and specificity is 0.950. Using the correction formula gives

p=\frac{0.220+0.950-1}{0.900+0.950-1} =\frac{0.170}{0.850} =0.200.

The correction recovers a target membership proportion of 20.0%, matching the 200 recorded members among 1,000 workers in the validation data. The calculation relies on the stated sensitivity and specificity being applicable to this same population and measurement procedure. It would not justify applying these rates to a different survey or population without evidence that the error process is comparable.

Differential Error Across Groups

An error process is differential when reporting or classification accuracy varies with a relevant group or circumstance. For example, respondents with different eligibility experiences may interpret a benefits question differently. A classification rule may also perform differently for groups whose cases are harder to categorize. Differential error is not synonymous with intentional misreporting: differences can arise from question wording, recall, access to records, or the measurement procedure.

For a binary target with group-specific error rates, the observed positive proportion in group $g$ is

p_{W,g}=\text{sensitivity}_g p_g+(1-\text{specificity}_g)(1-p_g).

Consequently, a difference in observed proportions can reflect a difference in target proportions, a difference in error rates, or both. A useful special case also shows why non-differential misclassification is not a universal remedy. If both groups share the same sensitivity and specificity, subtracting the two observed proportions gives

p_{W,1}-p_{W,0} =\bigl(\text{sensitivity}+\text{specificity}-1\bigr)(p_1-p_0).

This follows by expanding the group formula as $p_{W,g}=(1-\text{specificity})+(\text{sensitivity}+\text{specificity}-1)p_g$, then subtracting the expression for group 0 from the expression for group 1. If the common rates are better than random classification, so that their sum exceeds 1, the observed difference in this particular binary-prevalence comparison is scaled toward zero. That result depends on common error rates and this setup; it should not be generalized automatically to every misclassified variable or regression coefficient.

Worked Example: Equal Target Rates, Different Observed Rates

Consider a hypothetical survey of 2,000 workers, with 1,000 in each of two groups. In both groups, 400 workers truly received a specified benefit and 600 did not, so the target rate is 40.0% in each group. Suppose group A has sensitivity 0.950 and specificity 0.980, while group B has sensitivity 0.700 and specificity 0.900.

For group A, the expected classification counts are 380 true positives and 20 false negatives among the 400 true recipients; among the 600 nonrecipients, 12 are false positives and 588 are true negatives. The observed positive count is $380+12=392$, giving $392/1{,}000=0.392$, or 39.2%.

For group B, there are 280 true positives and 120 false negatives among recipients, plus 60 false positives and 540 true negatives among nonrecipients. The observed positive count is $280+60=340$, giving $340/1{,}000=0.340$, or 34.0%. The observed group difference, A minus B, is $39.2\%-34.0\%=5.2$ percentage points, even though the target rates are equal. Here the group-specific reporting and classification accuracy creates an apparent difference. The observed gap alone cannot establish that the underlying benefit receipt differs.

Reporting Error Is Broader Than Misclassification

Misclassification concerns categories. Reporting error also includes inaccurate numerical reports, such as a rounded, forgotten, or otherwise incorrect amount. A numerical report can be written as $W=T+U$, where $T$ is the target value and $U$ is the reporting error. If the distribution or average of $U$ differs across groups or circumstances, group comparisons of $W$ can reflect both differences in $T$ and differences in reporting.

Worked Example: Group Differences in Reported Earnings

Suppose a hypothetical data check finds that two groups each have a target mean monthly earnings amount of \$3,000. In group A, reports have an average error of \$0. In group B, reports have an average error of $-\$200$, perhaps because some respondents omit irregular earnings. Since the reported value is $W=T+U$, the group mean of $W$ is the group mean of $T$ plus the group mean of $U$.

For group A, the reported mean is $\$3{,}000+\$0=\$3{,}000$. For group B, it is $\$3{,}000-\$200=\$2{,}800$. Thus the observed reported-earnings difference is $\$2{,}800-\$3{,}000=-\$200$ per month, while the target mean difference is zero. The arithmetic illustrates how group-varying reporting error can produce a misleading comparison; it does not show that this specific error occurs in any actual survey.

What the Evidence Can and Cannot Establish

A confusion table is informative when the target category can be checked for at least a validation sample. It describes how recorded categories align with that benchmark in the population or sample where the comparison is made. It does not, by itself, establish why errors occurred, whether the benchmark is perfect, or whether the rates apply elsewhere. When there is no validation information, the observed distribution alone generally cannot separate the target prevalence from sensitivity and specificity: several combinations of those quantities may produce the same observed proportion.

For an empirical comparison, ask whether the measurement process plausibly operates similarly across the groups being compared. Relevant evidence may include validation samples, repeated measurement, administrative records, or changes in question wording and collection mode. The goal is not to assume that every difference in reports is error, but to identify what the data can support and what remains uncertain.

Common Mistakes

Using the wrong denominator. Sensitivity is calculated among true positives, while specificity is calculated among true negatives. Dividing both by the total sample does not give either rate.
Assuming high accuracy means no bias in a group comparison. Accuracy can differ across groups, and even common errors can alter observed rates. State which error rates are assumed to apply.
Calling any reporting error differential. Differential error requires that the reporting or classification process varies across a group or circumstance conditional on the target value, not merely that observed reports differ.
Treating an observed gap as a target gap. A difference in recorded categories can arise from different underlying rates, different measurement accuracy, or both.
Applying a correction without validation. A correction formula needs credible sensitivity and specificity for the relevant population and process. Values from another setting may not transfer.

Key takeaway: Misclassification changes categorical records; reporting error also includes inaccuracies in noncategorical values. When error rates vary across groups or circumstances, observed patterns can be created, hidden, or distorted. Interpretation requires information about the measurement process, not just a comparison of recorded values.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

A validation table contains 72 true positives and 8 false negatives. What is the sensitivity, and what is its denominator?
In your own words, distinguish misclassification from reporting error in a numerical variable.
Two groups have the same true positive rate, but one has a lower sensitivity. Explain how their observed positive proportions could differ.
Why can a modest false-positive rate matter when the true category is uncommon?
What information is needed to use sensitivity and specificity to correct an observed prevalence, and why might rates from another population be unsuitable?

	Observed positive \(W=1\)	Observed negative \(W=0\)	Total
True positive \(T=1\)	True positive: \(TP\)	False negative: \(FN\)	\(TP+FN\)
True negative \(T=0\)	False positive: \(FP\)	True negative: \(TN\)	\(FP+TN\)
Total	\(TP+FP\)	\(FN+TN\)	\(n\)