Duplicate Observations and Zero-Distance Neighbors

Why Repeated Predictors Need Care

In nearest-neighbor learning, an observation’s predictor vector determines its distance from a query point. If several observations share the same predictor vector, they have the same distance to every query. At an exact match, that distance is zero. This creates two related questions: how should a distance-weighted prediction handle a zero distance, and should repeated observations count separately?

These are not merely computational details. In regression, repeated predictors can have different responses. In classification, they can carry conflicting labels. A prediction procedure must say how it uses those records; silently dropping duplicates or applying an undefined inverse-distance weight can change the result.

Definition: For a query predictor vector \(x_0\), an observation is an exact match if its distance \(d_i=d(x_i,x_0)\) is zero. A set of observations with identical predictor vectors consists of duplicate predictor vectors. Their responses or class labels need not be identical.

Distance Weights and the Zero-Distance Case

For a common distance-weighted KNN rule, the weight assigned to observation \(i\) is \(w_i=d_i^{-p}\), where \(p>0\) is a chosen power. This rule is defined only when \(d_i>0\). At \(d_i=0\), the weight is undefined; it is not a finite number that can be substituted directly into the usual weighted-average formula.

A practical exact-match convention is to use only the zero-distance observations whenever at least one exists, giving those observations equal weight. This is an explicit modeling rule, not a consequence of evaluating \(d_i^{-p}\) at zero. It avoids letting arbitrary positive-distance observations affect a prediction when exact matches are available, and handles conflicting exact-match outcomes by averaging or counting them.

Convention used here: If the query has \(m\geq1\) zero-distance observations, use all \(m\), each with weight 1, regardless of the requested \(k\). For regression, predict their mean response. For classification, use their class proportions and then apply the stated class tie rule if needed. If there are no zero-distance observations, use the selected \(k\) neighbors with the specified positive-distance weights.

This convention can make the effective number of observations exceed \(k\) when more than \(k\) records match exactly. An implementation could instead impose a deterministic selection rule among exact matches, but that rule may discard relevant records and should be stated. In particular, selecting an arbitrary subset can produce a different prediction when exact matches have conflicting outcomes.

When no exact match is present, inverse-power weights are well-defined. In regression, the weighted prediction is a weighted mean. In classification, the corresponding weighted class fractions are normalized weighted totals for each class. The predicted class is the class with the largest fraction, subject to a stated class tie rule.

\widehat{f}(x_0)=\frac{\sum_{i\in N_k(x_0)}w_i y_i}{\sum_{i\in N_k(x_0)}w_i}, \qquad \widehat{P}(Y=c\mid x_0)= \frac{\sum_{i\in N_k(x_0)}w_i\,\mathbf{1}(y_i=c)} {\sum_{i\in N_k(x_0)}w_i}, \qquad w_i=d_i^{-p}.

Here \(N_k(x_0)\) is the selected neighbor set when all relevant distances are positive; \(y_i\) is a numerical response for regression or a class label for classification; and \(\mathbf{1}(y_i=c)\) equals 1 if observation \(i\) belongs to class \(c\), and 0 otherwise. At an exact match, the convention above replaces the inverse-power formula: the weights are 1 for the zero-distance records and 0 for all other records. The normalized classification fractions then sum to 1.

Why the Weighted Regression Formula Is a Mean

The weighted regression prediction is not an arbitrary averaging device. For fixed nonnegative weights with a positive total, it is the constant \(a\) that minimizes the weighted sum of squared errors for the selected observations. This gives a short justification for the formula.

Set up the local objective.
For selected observations, define \(Q(a)=\sum_i w_i(y_i-a)^2\). The weights are treated as fixed while choosing \(a\).

Differentiate and set the derivative to zero.
\(Q'(a)=-2\sum_i w_i(y_i-a)=0\), so \(\sum_i w_i y_i=a\sum_i w_i\).

Solve and check it is a minimum.
Since \(\sum_i w_i>0\), \(a=\sum_i w_i y_i/\sum_i w_i\). Also \(Q''(a)=2\sum_i w_i>0\), so this stationary point is the unique minimum.

For exact matches under the stated convention, all selected weights equal 1, so the minimizing constant is simply their arithmetic mean. For classification, normalizing the weighted indicator totals gives the weighted empirical fraction in each class; this is a local proportion, not necessarily a calibrated probability outside the model’s assumptions.

Worked Example: Regression With Two Exact Matches

A hypothetical claim model uses one predictor, \(x\), and distance weighting with \(p=1\). For a query \(x_0=5\), the three nearest records have \((x_i,y_i)=(5,18),(5,30),(6,50)\), with \(k=3\). Find the prediction under the exact-match convention. Name the method and the mistake it avoids.

Method. First identify the distances: \(d_1=|5-5|=0\), \(d_2=|5-5|=0\), and \(d_3=|6-5|=1\). There are \(m=2\) exact matches. The convention uses those two records only, each with weight 1; the positive-distance response is not included.

The prediction is

\widehat{f}(5)=\frac{1(18)+1(30)}{1+1} =\frac{48}{2}=24.

The result is 24. It is the mean response among the two exact matches. The common mistake is to insert \(1/0\) as an inverse-distance weight, or to treat the zero-distance records as ordinary finite-weight observations. Neither operation is defined. Including the response 50 would also violate the stated exact-match convention. Check: the prediction, 24, lies between the two exact-match responses, 18 and 30.

Conflicting Labels at an Exact Match

An exact predictor match does not guarantee identical labels. Under the convention above, each matching record contributes equally, so the local class proportions are the class counts divided by the number of exact matches. The most frequent class is predicted. If the largest proportions tie, the classification tie rule must be applied; a tie rule should not be confused with an instruction to ignore some matching records.

Worked Example: Conflicting Exact-Match Labels and a Class Tie

A hypothetical screening classifier uses \(p=2\). At query vector \(x_0=(3,1)\), four records have exactly that predictor vector: two are labeled Amber and two are labeled Blue. Other nearby records have positive distances. Find the exact-match class proportions and predicted class if the declared class tie rule selects Amber over Blue. Explain the mistake this avoids.

Method. All four exact matches receive weight 1, and the positive-distance observations receive no weight under the stated policy. Thus the total weight is \(1+1+1+1=4\).

\widehat{P}(\text{Amber}\mid x_0)=\frac{2}{4}=0.5, \qquad \widehat{P}(\text{Blue}\mid x_0)=\frac{2}{4}=0.5.

The class proportions tie, so the declared tie rule gives the prediction Amber. The value \(p=2\) does not change this calculation: the inverse-power formula is not used for zero distances. A common mistake is to assume the exact matches must all have the same label, or to claim one class has a larger weighted vote without counting all four records. Check: the two reported class proportions sum to 1, and the tie rule—not a difference in the proportions—determines the predicted label.

Duplicates at Positive Distance Still Count

Repeated predictor vectors matter even when their distance from the query is positive. Under an observation-level KNN rule, each row is an observation and contributes its own response or label. Consequently, repeated records at a location can give that location greater total weight than a location represented once. Removing duplicates changes that empirical weighting unless the data are aggregated while preserving their multiplicity and outcome information.

This distinction is substantive. Repeated rows may represent genuinely repeated observations, or they may be accidental data duplication. If they are valid observations, retaining their multiplicity is part of the fitted rule. If they are errors, they should be corrected as data-quality problems rather than removed silently during prediction.

Worked Example: Duplicate Predictors at Positive Distance

A hypothetical severity model predicts a numerical response at query \(x_0=0\), using \(k=3\) and inverse-distance weights with \(p=1\). The selected records are \((x_i,y_i)=(1,4),(1,10),(2,16)\). Calculate the prediction, and compare it with the result of incorrectly collapsing the duplicate predictor rows to one unweighted row with response equal to their mean.

Method. The distances are \(d_1=1\), \(d_2=1\), and \(d_3=2\). None is zero, so the inverse-distance weights are defined: \(w_1=1\), \(w_2=1\), and \(w_3=1/2\). The observation-level weighted numerator is \(1(4)+1(10)+(1/2)(16)=4+10+8=22\). The denominator is \(1+1+1/2=2.5\). Therefore,

\widehat{f}(0)=\frac{22}{2.5}=8.8.

If the two rows at \(x=1\) are incorrectly collapsed to a single row with response \((4+10)/2=7\) and given only one weight of 1, the result becomes \(\{1(7)+(1/2)(16)\}/(1+1/2)=15/1.5=10\). This differs from 8.8 because the collapsed calculation discarded one observation’s weight. Aggregation can preserve the original result only if it retains the multiplicity and the response information: the two rows at \(x=1\) contribute total weight 2 and weighted response 14. Check: retaining that contribution gives \((14+8)/(2+0.5)=22/2.5=8.8\).

Selection Ties, Class Ties, and Practical Checks

There are two distinct kinds of ties to check. A neighbor-selection tie occurs when multiple observations have the same distance at the boundary of the selected \(k\). A class-vote tie occurs when two or more classes have equal largest weighted proportions after the neighbor set is chosen. A rule for one does not resolve the other.

For example, if a tie at the \(k\)th distance is resolved by a deterministic record ordering, the chosen neighbor set may depend on that ordering. Alternatively, a method may include all observations tied at the boundary, which can make the effective neighbor count exceed \(k\). Either choice should be explicit. The exact-match convention in this tutorial is a separate rule: when any zero-distance records exist, it uses all of them, even if there are more than \(k\).

Check the distances before the weights. If any selected distance is zero, do not evaluate \(d_i^{-p}\) for that record.
Count records, not just distinct predictor vectors. Each valid observation contributes according to the stated procedure.
Report class fractions before the final label. They reveal whether a class tie rule was needed.
State both tie rules. Say how equal distances at the selection boundary are handled and how equal class votes are resolved.
Do not use an unstated small constant. Replacing \(d_i\) by \(d_i+\epsilon\) defines a different weighting rule and may produce results that depend on the chosen \(\epsilon\).

These checks matter in applications where repeated records may arise naturally or through data processing. For a hypothetical insurance classifier, several policy records could share identical measured predictors while having different claim outcomes. Treating their labels as if they could not conflict hides uncertainty; dropping records without preserving their contribution changes the local evidence. A carefully stated rule makes the prediction reproducible and interpretable.

Key takeaway: Inverse-distance weights are undefined at zero distance. Choose and state an exact-match policy; under the all-matches, equal-weight convention, use every zero-distance record and resolve any resulting classification tie explicitly. At positive distances, duplicate observations retain their individual contributions unless aggregation preserves their multiplicity and outcomes.

Check Your Understanding

For each problem, apply the stated convention and show the intermediate calculations before checking the final answer.

A regression query has two exact-match responses, 6 and 12, and a third selected response of 30 at positive distance. Under the all-exact-matches convention, what is the prediction? (Answer: 9)
A classification query has four exact matches: two Green and two Gold. The tie rule selects Gold over Green. What are the class proportions and predicted class? (Answer: Green 0.5000, Gold 0.5000; Gold)
For inverse-distance-squared regression at \(x_0=0\), the selected observations are \((x,y)=(1,2)\) and \((2,8)\), with \(k=2\). Find the prediction. (Answer: 3.2)
For inverse-distance regression at \(x_0=0\), the three selected records are \((1,3),(1,9),(2,12)\). Keep each record’s multiplicity. Find the prediction. (Answer: 7.2)
A query has four exact-match labels: A, A, B, B. The requested \(k\) is 2, and the stated convention uses all exact matches; the class tie rule selects B. Find both class proportions and the prediction. (Answer: A 0.5000, B 0.5000; B)

What You'll Learn