Diagnosing the Influence of Large Weights

Why Examine Large Weights?

Survey weights allow sampled records to represent different numbers of population units. A record with a large weight can therefore have a substantial effect on a weighted description—but the size of the effect also depends on the record’s measured value and on the statistic being calculated. A large weight attached to an outcome close to the weighted mean may have little effect on that mean, while a less extreme weight attached to an unusual outcome may matter considerably.

Diagnosing influence means asking how much the estimate depends on particular weighted records. It is a sensitivity analysis, not a procedure for deciding that a record is wrong or for automatically reducing its weight. The weight may be large for a legitimate design reason, such as a low probability of selection, or because of a documented adjustment. Any proposed change requires investigation of the survey design and weight construction.

Start With Weight Shares and Contributions

For \(n\) eligible records, the sum of the weights is \(\sum_{i=1}^{n} w_i\). The normalized weight share of record \(i\) is its weight divided by that sum. This describes how much of the total weighted mass is assigned to the record; it does not by itself describe the record’s influence on every outcome.

Useful initial summaries include the largest individual weight shares and the combined share held by the largest few weights. These are descriptive flags, not universal cutoffs: whether a share is concerning depends on the purpose of the estimate, the outcome, and the design. For a weighted total, the relevant record-level contribution is \(w_i y_i\), since the total estimate is the sum of those contributions.

If \(y_i\) is nonnegative, a contribution share can be calculated as \(w_i y_i/\widehat{T}_w\), provided the total is positive. When outcomes can be negative or the total is near zero, such shares may be misleading or undefined; report the signed contributions instead, and consider their magnitudes in context. For category totals, the corresponding contribution is \(w_i\) when record \(i\) belongs to the category and zero otherwise.

Compare Means and Proportions by Leaving Out a Record

A weighted mean is a ratio of a weighted outcome total to the corresponding total weight. For a particular record \(i\), define the leave-one-record-out mean by removing that record from both sums while retaining the other records’ original weights. Comparing this estimate with the full weighted mean shows the record’s effect on the ratio. The comparison is descriptive; it does not establish that the record should be removed.

The difference has a useful exact form. Let \(W=\sum_{j=1}^{n}w_j\) and \(T=\sum_{j=1}^{n}w_jx_j\), so that \(\bar{x}_w=T/W\). Removing record \(i\) gives \((T-w_i x_i)/(W-w_i)\). Subtracting the full estimate and combining terms gives:

Thus, a record changes a weighted mean more when its weight is large and its value is far from the full weighted mean. The sign also has a direct interpretation: if \(x_i\) is above \(\bar{x}_w\), removing it lowers the mean; if it is below \(\bar{x}_w\), removing it raises the mean. The same calculation applies to a weighted proportion by setting \(x_i=1\) for records in the category and \(x_i=0\) otherwise.

Quantiles Require Recomputing the Weighted Distribution

A weighted empirical cumulative distribution function (ECDF) records the fraction of total weight at or below each value. For a quantitative outcome \(x_i\), it is defined as the sum of weights for records with \(x_i\leq t\), divided by the total weight. A weighted quantile can then be defined as the smallest value at which this ECDF reaches or exceeds the desired probability. This convention handles discrete data and ties by using the cumulative weight at each distinct value.

To assess a record’s influence on a quantile, remove it, recalculate the total weight and ECDF from the remaining records, and find the quantile again using the same definition. Unlike the mean, a quantile does not generally have a simple contribution formula: its value can stay unchanged over a range of weight changes and then jump when a cumulative-weight threshold is crossed.

How to Use the Diagnostic Responsibly

A practical assessment combines weight concentration with outcome-specific comparisons. First verify that the records are eligible and that the same denominator and analytic population are used throughout. Then identify large weight shares, inspect their weighted contributions for totals, and recompute the relevant mean, proportion, or quantile after excluding each flagged record in turn. Report the full estimate, the comparison estimate, and their difference in meaningful units. For a mean or proportion, a relative change can also be useful when the full estimate is not near zero; always state the reference used to calculate it.

For totals, preserve the original weights when computing the simple leave-one-record-out comparison. Renormalizing the remaining weights changes the total’s scale, so it no longer isolates the removed record’s contribution. For means and proportions, multiplying every remaining weight by the same positive constant leaves the ratio unchanged, but keeping the original weights still makes the comparison procedure explicit and consistent.

A large leave-one-out change should prompt checks of the record, its outcome, eligibility, and the documented construction of its weight. The record may be correct and essential to representing a sparsely sampled part of the population. Conversely, a small change in one estimate does not show that the weights are correct for every other outcome. The diagnostic concerns the estimate and records examined; it does not replace design-based uncertainty assessment or justify a general rule for trimming weights.

Common Pitfall