Reweighting Data to Address Representation

Why Reweighting Changes What the Data Represent

A model is influenced by the observations it sees. If one group is sparsely represented in a training sample, its patterns may have little influence on an ordinary, unweighted fit—even when that group should have greater influence for the decision at hand. Reweighting assigns different numerical importance to observations so that the weighted data reflect a stated target population or target mix of groups.

Reweighting does not add new observations, correct inaccurate labels, or guarantee fair decisions. It changes which population an average or model-fitting objective represents. That makes the choice of target population central: weights appropriate for one question may be inappropriate for another. The methods here are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Suppose the sample contains groups indexed by \(g\). Let \(p_g\) be group \(g\)'s proportion in the sample and \(q_g\) its specified proportion in the target population. If \(p_g>0\), the group representation weight is \(w_g=q_g/p_g\). Each observation in group \(g\) receives weight \(w_g\). These weights adjust the groups' relative contributions; they do not change the observed features, outcomes, or predictions.

For a quantity \(z_j\) measured on each observation \(j\), a normalized weighted mean is \( \sum_j w_j z_j / \sum_j w_j \). A weighted empirical loss uses the same construction with each \(z_j\) replaced by the loss for that observation. Scaling every weight by the same positive constant leaves these normalized quantities unchanged. In fitting a model, however, the chosen software's weighting convention should be checked: weights can have different interpretations in different procedures.

Constructing Weights and Stating the Target

Weights are not selected merely because a group is called underrepresented. First state the target: for example, a desired mix of demographic groups in a portfolio, or a specified population for evaluating prediction error. Then compare the target proportions with the sample proportions. If the target has more of a group than the sample, that group's weight exceeds one; if it has less, its weight is below one.

w_g=\frac{q_g}{p_g}, \qquad \widehat{R}_{\text{target}} =\frac{\sum_{j=1}^{n} w_j L_j}{\sum_{j=1}^{n}w_j},

Here \(L_j\) is an observation's loss, such as absolute prediction error or binary log loss. If the weights are constructed from group proportions and the target and sample proportions each sum to one, the total weight assigned to group \(g\) is proportional to \(q_g\). The weighted calculation therefore gives the target mix—not the observed sample mix—the intended influence.

Worked Example: Reweighting Prediction Error to a Target Group Mix

A hypothetical validation sample has 60 observations from Group North and 40 from Group South. The intended evaluation population is 40% North and 60% South. The sum of absolute prediction errors is 36 for North and 30 for South. Calculate the weighted mean absolute error (MAE), and compare it with the ordinary sample MAE.

Method. Use the ratio of target proportion to sample proportion for each group, then divide the weighted error sum by the total weight. This method avoids treating the sample's 60/40 composition as though it were the stated 40/60 target.

North's sample proportion is \(60/100=0.60\), so its weight is \(0.40/0.60=2/3\). South's sample proportion is \(40/100=0.40\), so its weight is \(0.60/0.40=1.5\).

The total North weight is \(60(2/3)=40\), and the total South weight is \(40(1.5)=60\). The weighted sum of absolute errors is \(36(2/3)+30(1.5)=24+45=69\). The total weight is \(40+60=100\). Thus the weighted MAE is \(69/100=0.69\).

For comparison, the ordinary MAE is \((36+30)/100=0.66\). As a check, North's group MAE is \(36/60=0.60\), and South's is \(30/40=0.75\). Applying the target mix directly gives \(0.40(0.60)+0.60(0.75)=0.24+0.45=0.69\), which agrees with the weighted calculation. Common mistake: multiplying each group's error sum by its target proportion and dividing by the original sample size would not, in general, produce the target-weighted mean. The denominator must reflect the weights too.

Why the Weighted Average Represents the Target Mix

The central justification is a cancellation of the sample proportions. Let \(Z\) be a quantity of interest, and let \(\bar{Z}_g\) be its sample mean within group \(g\). With group-constant weights \(q_g/p_g\), the normalized weighted mean can be written as a weighted average of the group means. When the sample group shares sum to one, its denominator is one on the population-proportion scale.

\frac{\sum_g p_g (q_g/p_g)\bar{Z}_g} {\sum_g p_g(q_g/p_g)} = \frac{\sum_g q_g\bar{Z}_g}{\sum_g q_g} = \sum_g q_g\bar{Z}_g.

The last equality uses \(\sum_g q_g=1\). Thus, if the observed group means estimate the relevant within-group means, the reweighted result estimates the average for the specified target composition. This argument explains what the weights accomplish—and what they do not. They change the mix of group contributions; they do not make an unrepresentative group sample representative within that group.

Using Weights in a Binary Model

Weights can affect model fitting as well as a final summary. For example, a weighted Bernoulli log-likelihood gives more influence to observations with larger weights. In an intercept-only Bernoulli model, the fitted probability is the weighted event proportion. The following calculation shows the distinction between changing group influence and changing any individual's observed outcome.

Worked Example: A Weighted Intercept-Only Event Rate

A hypothetical sample contains 80 policies from Group A, of which 8 have an event, and 20 policies from Group B, of which 10 have an event. The target population assigns equal proportions to A and B. Find the weighted intercept-only Bernoulli estimate of the event probability. Assume the sample observations represent their groups for this outcome.

Method. Fit the intercept-only model by maximizing the weighted Bernoulli log-likelihood. The group weights are \(0.50/0.80=0.625\) for A and \(0.50/0.20=2.5\) for B. The common mistake is to report the raw event rate without accounting for the target group mix.

The total weight for A is \(80(0.625)=50\); its weighted event count is \(8(0.625)=5\). The total weight for B is \(20(2.5)=50\); its weighted event count is \(10(2.5)=25\). Therefore the total weighted event count is \(5+25=30\), and the total weight is \(50+50=100\). The weighted estimate is \(30/100=0.30\).

As a check, the within-group event rates are \(8/80=0.10\) for A and \(10/20=0.50\) for B. Their equal-weight target average is \(0.50(0.10)+0.50(0.50)=0.05+0.25=0.30\). The unweighted rate, \(18/100=0.18\), instead reflects the sample's 80/20 group mix.

Derivation: In an intercept-only Bernoulli model with fitted event probability \(\pi\), let \(y_j\) be 1 for an event and 0 otherwise. The weighted log-likelihood is \(\ell(\pi)=\sum_j w_j[y_j\log(\pi)+(1-y_j)\log(1-\pi)]\). Its derivative is \(\sum_j w_jy_j/\pi-\sum_jw_j(1-y_j)/(1-\pi)\). Setting it equal to zero and multiplying by \(\pi(1-\pi)\) gives \((1-\pi)\sum_jw_jy_j-\pi\sum_jw_j(1-y_j)=0\). Rearranging yields \(\widehat{\pi}=\sum_jw_jy_j/\sum_jw_j\). This result applies to an intercept-only Bernoulli likelihood with positive weights; it is not a formula for every GLM coefficient.

Assumptions, Positivity, and the Meaning of the Target

A representation weight has a defensible interpretation only when its construction and use match the question. Important conditions include:

A specified target. The target group shares must correspond to a clearly described population or evaluation objective. A different target mix produces a different weighted result.
Positivity. Every group with positive target share must be present in the sample: \(q_g>0\) requires \(p_g>0\). If a target group has no sampled observations, a group weight cannot create information about it.
Within-group representativeness. The sample must provide a suitable basis for estimating the outcomes or feature patterns within each weighted group. Group-level weighting alone cannot correct selection differences within a group that matter to the quantity being estimated.
Reliable group definitions and proportions. Misclassified group membership or inaccurate sample and target shares produce inappropriate weights.
A suitable use of weights. The model-fitting or evaluation procedure must implement the intended weighted objective. A weight used in one software function may not have the same meaning in another.

These conditions do not imply that reweighting identifies a causal effect. Nor does a larger target weight prove that a group was excluded unfairly or that the resulting model is fair. Reweighting can change overall error, group error rates, and the populations to which reported metrics apply. Those results should be stated with their target population and checked by group where relevant.

Extreme Weights and Effective Sample Size

A small sample share paired with a much larger target share can produce a large weight. Such observations may dominate a weighted estimate. One diagnostic is the Kish effective sample size, a summary of how uneven weights reduce the information in a weighted sample:

n_{\mathrm{eff}}=\frac{(\sum_j w_j)^2}{\sum_j w_j^2}.

This is a diagnostic, not a universal replacement for a standard error calculation. It is most directly interpreted for independent observations and does not account for every feature of model fitting or dependence in the data. A low value warns that the weighted analysis may be much less stable than the raw sample size suggests.

Worked Example: Diagnosing a Sparse Target Group

A hypothetical sample has 99 observations in Group A and 1 in Group B. The target population is 80% A and 20% B. Calculate the group weights and Kish effective sample size. Explain why the result deserves caution.

Method. Calculate target-to-sample ratios, then substitute all 100 observation weights into the effective-sample-size formula. The common mistake is to treat the sample size of 100 as 100 equally informative observations after reweighting.

The sample proportions are \(p_A=0.99\) and \(p_B=0.01\). Therefore \(w_A=0.80/0.99=80/99\), approximately \(0.8081\), and \(w_B=0.20/0.01=20\). The total weight is \(99(80/99)+1(20)=80+20=100\).

The sum of squared weights is \(99(80/99)^2+20^2=6400/99+400=(6400+39600)/99=46000/99\), approximately \(464.6465\). Hence \(n_{\mathrm{eff}}=100^2/(46000/99)=990000/46000\), approximately \(21.52\). Rounded to two decimals, the effective sample size is 21.52.

The result is far below the raw sample size of 100 because the single Group B observation has weight 20. That observation supplies all the sampled information about B, despite representing a larger share of the target. The calculation cannot establish whether the B outcome is typical; it only signals that the estimate may be sensitive to that observation. The target has positive support in both groups, but the very small B sample remains a serious limitation.

Practical Checks Before Reporting

Before fitting or evaluating with representation weights, document the target mix and the source of the sample proportions. Then inspect the weight distribution, including the largest weights and the effective sample size. If weights are trimmed or capped to reduce instability, explain that choice: the resulting analysis no longer exactly uses the originally specified ratios and may represent a different compromise.

Also compare relevant results before and after weighting and report which population each describes. A change in an overall metric may reflect only the changed group mix, or may also arise because weighting changed model fitting. Neither result alone establishes that group-level errors or other fairness measures have improved. Reweighting is one way to alter influence; it is not a substitute for checking outcomes and limitations in context.

Key takeaway: Reweighting is meaningful only relative to a stated target population. The ratio \(q_g/p_g\) shifts group influence to that target when groups are represented in the sample and their within-group information is suitable. Sparse groups, extreme weights, and an unclear target can make the result unstable or misleading.

Check Your Understanding

For each problem, identify the target mix before calculating. Answers are given so you can check the weighted calculation and its interpretation.

A sample has 30 observations from Group C and 70 from Group D. The target is 50% in each group. What are the group weights? (Answer: C has weight \(0.50/0.30=1.6667\); D has weight \(0.50/0.70=0.7143\).)
A sample's group proportions are 75% and 25%, and the target proportions are 60% and 40%. The within-group mean losses are 0.20 and 0.50. What is the target-weighted mean loss? (Answer: \(0.60(0.20)+0.40(0.50)=0.32\).)
A sample contains 40 events among 200 observations in Group E and 15 events among 50 in Group F. The target is 50/50. What is the weighted intercept-only event estimate? (Answer: \((0.50(40/200)+0.50(15/50))=0.20\).)
A sample has 90 observations in Group A and 10 in Group B, while the target is 80% A and 20% B. What are the weights, and what is the total weight? (Answer: A has weight \(0.80/0.90=0.8889\), B has weight \(0.20/0.10=2\), and total weight is \(90(0.8889)+10(2)=100\), using the unrounded A weight \(8/9\).)
A sample has 49 observations with weight 1 and one observation with weight 7. Calculate Kish effective sample size and state the main caution. (Answer: \(56^2/(49+49)=3136/98=32\); the uneven weights mean the weighted information is summarized by an effective sample size of 32, and the high-weight observation may have substantial influence.)

What You'll Learn