Detecting Near-Duplicate Predictors

Why Detect Near-Duplicate Predictors?

A data set may contain two columns that record almost the same characteristic: a measurement in two units, two differently named versions of the same category, or a variable copied with a small number of errors. Finding these cases matters because duplicate information can make a model harder to interpret, create avoidable data-quality problems, and conceal inconsistent coding. A high association is a reason to investigate, not by itself a reason to delete or combine predictors.

The practical question is whether two predictors carry nearly the same information for the intended analysis and across the relevant records. Pairwise correlation is useful for numeric columns, but does not establish agreement. Cross-tabulations make categorical agreement visible. Direct comparisons show the size and pattern of actual discrepancies. In R, these checks should use the same records and should preserve the original variables until a defensible decision is made.

Definition: Two predictors are exact duplicates on a data set if their recorded values agree for every relevant record, using an appropriate comparison for their types. They are near-duplicates when they encode essentially the same information but have limited, explainable discrepancies, such as a few inconsistent labels or a known unit conversion. Whether discrepancies are limited and explainable depends on the measurement, data source, and intended use; there is no universal cutoff that makes two predictors duplicates.

Start With a Pairwise Screen, Then Inspect the Data

For numeric predictors, the Pearson correlation summarizes the strength and direction of a linear relationship. If one nonconstant predictor is exactly an affine transformation of another, $Y=a+bX$ for constants $a$ and $b\ne 0$, their correlation is $1$ when $b>0$ and $-1$ when $b<0$. This makes a correlation matrix a useful first screen for possible duplicate measurements or rescalings.

However, correlation measures association, not equality. It is unchanged by adding a constant or multiplying by a positive constant. Thus, two columns may have correlation 1 while having entirely different units and values. A high correlation can also coexist with consequential discrepancies for a small number of records. After a correlation screen, check the values, units, ranges, and record alignment.

A simple R screen for two numeric columns is cor(data$measure_a, data$measure_b, use = "complete.obs"). This computes the correlation for records where both columns are observed. The complete-case choice should be deliberate: report how many records were compared, and check whether missingness differs between columns. Do not compare two vectors after independently removing their missing values, because their remaining positions may no longer refer to the same records.

Justification: If $Y=a+bX$, then $\operatorname{Cov}(X,Y)=\operatorname{Cov}(X,a+bX)=b\operatorname{Var}(X)$, because adding a constant does not change covariance. Also, $\operatorname{Var}(Y)=b^2\operatorname{Var}(X)$. Provided $\operatorname{Var}(X)>0$, the correlation is $$ \operatorname{Corr}(X,Y) = \frac{b\operatorname{Var}(X)} {\sqrt{\operatorname{Var}(X)b^2\operatorname{Var}(X)}} = \frac{b}{|b|}. $$ It is therefore $1$ or $-1$, even when $Y$ is not numerically equal to $X$. The result does not apply if either variable is constant, because its correlation is undefined.

For categorical predictors, compare their recorded categories with a cross-tabulation. The table shows which values of one variable occur with each value of the other. A near-diagonal table can suggest a consistent mapping, but the categories must also have an understood relationship. Two variables can have high agreement simply because one category is very common, so inspect the counts within each category as well as the overall proportion of matches.

In R, table(data$category_a, data$category_b, useNA = "ifany") displays the joint counts, including missing values. If category names differ but are intended to correspond, write down the proposed mapping explicitly before measuring agreement. Do not silently rename categories to make the table appear diagonal.

Worked Comparisons

Worked Example: The Same Measurement in Different Units

A hypothetical data extract records five account opening ages in months and years. The month values are 12, 24, 36, 48, and 60; the year values are 1, 2, 3, 4, and 5. Determine whether these predictors are duplicates, and show what the correlation does and does not tell you.

Method. Check the proposed unit conversion record by record, then interpret the pairwise correlation. The conversion from months to years is to divide by 12.

Step 1: Apply the conversion. The month values divided by 12 are 12/12 = 1, 24/12 = 2, 36/12 = 3, 48/12 = 4, and 60/12 = 5 years. Each converted value agrees with the recorded years value, so the columns encode exactly the same measurement on these five records.

Step 2: Check the correlation. The values satisfy $Y=X/12$, where $X$ is age in months and $Y$ is age in years. This is an affine relationship with positive slope, so the correlation is 1. In R, cor(data$age_months, data$age_years) returns 1 for these complete, nonconstant columns.

Step 3: Decide what is defensible. These are not identical recorded numbers, but they are exact unit-converted versions of the same information. If the intended predictor is age, retaining a single clearly labelled unit is defensible after checking the conversion on all relevant records. Replacing one column with the other is a unit choice, not evidence that the underlying age should be discarded.

Check. Multiplying the five year values by 12 gives 12, 24, 36, 48, and 60 months, matching the original month column. The conclusion is consistent with both the direct comparison and the correlation. The common mistake avoided here is treating correlation 1 as proof that recorded values are equal without checking their units.

Worked Example: Comparing Two Categorical Encodings

A hypothetical service team records a two-level intake category in two systems. System A records “standard” or “priority”; System B records “basic” or “plus.” For 100 shared records, the cross-tabulation is:

System A	System B: basic	System B: plus	Total
Standard	58	2	60
Priority	4	36	40
Total	62	38	100

Assess whether the systems are near-duplicates under the proposed mapping standard to basic and priority to plus.

Method. Use the cross-tabulation to count agreements and disagreements, then inspect discrepancies within each System A category. In R, the relevant table can be produced with table(data$system_a, data$system_b).

Step 1: Count matches under the mapping. The mapped agreement is the sum of the standard/basic and priority/plus cells: 58 + 36 = 94 records. The observed agreement proportion is 94/100 = 0.94, or 94%.

Step 2: Count and locate mismatches. There are 2 standard/plus records and 4 priority/basic records, for 2 + 4 = 6 mismatches. Within the 60 standard records, 2/60 = 0.0333, or 3.33%, disagree with the proposed mapping. Within the 40 priority records, 4/40 = 0.10, or 10%, disagree.

Step 3: Interpret in context. The table supports a strong correspondence, but not exact duplication. The larger mismatch rate within priority records is worth investigating: it could reflect errors, a real difference in definitions, or a process that changed over time. Recoding one system to the other is defensible only if a subject-matter review confirms that the labels are intended to represent the same classification and explains how the six mismatches should be handled.

Check. The diagonal under the proposed mapping totals 94, and the off-diagonal cells total 6; together they account for all 100 records. The common mistake avoided is reporting 94% agreement as proof that the variables are interchangeable without examining what the disagreements mean.

Worked Example: A High Correlation With a Consequential Difference

A hypothetical sensor feed has five paired readings. Predictor $X$ is 1, 2, 3, 4, 5, and predictor $Z$ is 2, 4, 6, 8, 14. Decide whether the correlation alone justifies treating $Z$ as a duplicate of twice $X$.

Method. Calculate the Pearson correlation and then compare each observed $Z$ with the proposed duplicate $2X$. The direct comparison checks whether the relationship holds for every record.

Step 1: Calculate the correlation. The means are $\bar X=(1+2+3+4+5)/5=3$ and $\bar Z=(2+4+6+8+14)/5=34/5=6.8$. The deviations from the means are $X-\bar X=(-2,-1,0,1,2)$ and $Z-\bar Z=(-4.8,-2.8,-0.8,1.2,7.2)$. The sum of cross-products is

$$ (-2)(-4.8)+(-1)(-2.8)+(0)(-0.8)+(1)(1.2)+(2)(7.2)=28. $$

The sums of squared deviations are $\sum(X-\bar X)^2=4+1+0+1+4=10$ and $\sum(Z-\bar Z)^2=23.04+7.84+0.64+1.44+51.84=84.8$. Therefore,

r=\frac{28}{\sqrt{10(84.8)}}=\frac{28}{\sqrt{848}}\approx 0.9615.

Step 2: Compare the proposed duplicate values. Twice $X$ is 2, 4, 6, 8, 10. Subtracting these values from $Z$ gives differences 0, 0, 0, 0, 4. The first four records agree exactly with $Z=2X$, but the fifth does not: $14-2(5)=4$.

Step 3: Decide what the evidence supports. The correlation is high, but the direct comparison reveals a discrepancy of 4 in the fifth record. Whether that is negligible depends on the sensor’s units, precision, and use. For example, if a difference of 4 is operationally important, the predictors should not be treated as interchangeable without resolving the record. In R, inspect both cor(data$x, data$z) and data$z - 2 * data$x; a single summary correlation would conceal the location and size of the mismatch.

Check. The squared-deviation sums and cross-product sum give $28/\sqrt{848}$, approximately 0.9615, while the final direct difference is 4. Both calculations support the same conclusion: strong linear association is not exact agreement. The common mistake avoided is using a high correlation as a substitute for record-level comparison.

A Defensible R Comparison Workflow

A comparison is only meaningful when the columns refer to the same records. Before calculating a correlation, cross-tabulation, or difference, verify that rows are aligned by the relevant identifier and that filtering has been applied consistently. If one column has missing values, report how many paired observations remain. A result based on complete pairs describes those pairs; it does not automatically establish agreement for records excluded because of missingness.

Confirm the variables and their meaning.
Check variable definitions, units, category labels, observation dates, and record identifiers. Similar column names do not guarantee similar definitions.

Choose a type-appropriate screen.
For numeric predictors, inspect pairwise correlations. For categorical predictors, use a cross-tabulation. Do not treat a numeric code as a quantitative measurement without confirming its meaning.

Compare records directly.
For numeric columns, inspect differences or deviations from a proposed conversion. For categorical columns, count mapped agreements and mismatches, including missing values where relevant.

Investigate discrepancies before changing data.
Determine whether differences are data errors, expected rounding, changed definitions, or meaningful distinctions. Keep the original values available while resolving the issue.

Document the decision.
State which variable is retained or how recoding is performed, the mapping or conversion used, the records affected, and why the choice is appropriate for the intended analysis.

For numeric variables, a useful direct diagnostic might be data$measure_b - data$measure_a when the values should agree, or data$measure_b - 12 * data$measure_a when a particular unit conversion is proposed. A summary such as the minimum and maximum difference can help identify the size of discrepancies, but inspect the records themselves when any difference may be consequential. For categories, use the cross-tabulation to find which combinations disagree; a single overall percentage can hide concentrated problems in a less common category.

Common Traps and Practical Decisions

Equating correlation with equality. Correlation 1 can occur when one column is a rescaled or shifted version of another. Check units and values directly.
Ignoring a small number of records. A few disagreements can be important if they occur in a critical group or reflect a genuine difference in meaning. Locate and investigate them.
Trusting overall categorical agreement alone. Report the cross-tabulation and inspect mismatch counts within categories, especially when category sizes differ.
Comparing unaligned observations. A correlation or difference computed from rows that do not represent the same records cannot diagnose duplication.
Recoding based only on similar labels. Similar names or a near-diagonal table do not prove that definitions match. Confirm the mapping with the data dictionary or relevant subject-matter context.
Deleting a predictor before resolving the issue. Preserve the original columns and document any correction or recoding so the decision can be reviewed.

The goal is not to remove every pair of strongly associated predictors. It is to determine whether two columns represent the same underlying information, whether their differences can be explained, and whether a recoding or a single retained representation is defensible for the analysis. These comparison habits are aligned with the SOA Exam PA (Predictive Analytics) syllabus because they connect screening results to data quality and modeling judgment.

Check Your Understanding

For each problem, decide what the comparison establishes and state the key check that supports your conclusion.

A file contains a nonconstant distance measure in meters and a second column with values equal to the first column divided by 1,000. What correlation should you expect, and what direct check establishes whether the columns are exact unit conversions? (Answer: Correlation 1; verify record by record that the second value equals the first divided by 1,000.)
Two mapped categories have counts 72 and 18 on the diagonal of a cross-tabulation, and 7 and 3 off the diagonal. What is the overall mapped agreement among the records shown? (Answer: 90/100 = 90%.)
For five paired numeric observations, the correlation is 0.99, but one proposed conversion has a difference of 25 units on a record where differences above 10 matter. Is it defensible to call the columns interchangeable without further review? (Answer: No; investigate the 25-unit discrepancy because it exceeds the stated meaningful-difference threshold.)
In R, one predictor has missing values in rows 2 and 6, while another has missing values in rows 3 and 6. Why is it unsafe to remove missing values separately before comparing the resulting vectors? (Answer: The remaining positions may refer to different records; use aligned rows with both values observed and report the paired sample.)
A categorical cross-tabulation has 98% overall agreement, but all disagreements occur in a small category whose meaning is important to the analysis. What should be checked before recoding? (Answer: Confirm the category definitions and investigate the mismatches; overall agreement alone does not establish that the recoding is appropriate.)

What You'll Learn