What Trimming Does
Trimming removes entire observations when they meet a specified exclusion rule. The rule might use a value of a predictor, a recorded outcome, or a data-quality criterion. Once an observation is removed, its other recorded variables are removed from that analysis too. Trimming is different from winsorization: winsorization changes an extreme value while retaining its row; trimming discards the row.
A rule should be stated precisely before it is applied. For example, “exclude annual mileage above 45,000” specifies both the variable and the threshold. “Remove outliers” does not: it leaves unclear how an outlier is defined, whether equality counts, and which observations are affected. A clear rule makes the change reproducible, but it does not by itself establish that the excluded observations are errors or irrelevant.
Trimming changes more than the row count. It may change the distribution of a predictor, the proportions of subgroups, the range of outcomes, and the relationship between predictors and outcomes. These changes matter because a model fitted to the retained data describes that selected data unless further assumptions justify interpreting it as describing a broader population.
Quantifying the Change
For a numerical variable \(Y\), the mean among retained observations is
This is not generally the same as the mean in the full sample. The full-sample mean can be decomposed into the retained and excluded parts. Let \(n_D=n-n_R\) be the number deleted, and let \(\bar{Y}_D\) be the mean among deleted observations when \(n_D>0\). Then
The justification is direct: the total of \(Y\) is the retained total plus the deleted total, namely \(n_R\bar{Y}_R+n_D\bar{Y}_D\). Dividing that total by the original sample size gives the full-sample mean. Unless the deleted observations have the same mean as the retained ones, the two means differ.
For a categorical subgroup, its share must be recalculated using the retained sample size as the denominator. If a subgroup has \(n_{g,R}\) retained observations, its retained share is \(n_{g,R}/n_R\), not \(n_{g,R}/n\). This distinction is essential when deletion is concentrated in particular groups.
Worked Example: A Threshold Changes the Sample and Its Composition
A hypothetical insurer reviews 12 claim records. The recorded claim amounts, in thousands of dollars, are: Region A has 8, 12, 16, 20, 24, and 28; Region B has 10, 14, 18, 22, 26, and 100. A review plan written before analysis says to exclude records with amounts above 40. Calculate the sample size, regional shares, and overall mean before and after trimming.
Method. Apply the strict rule “amount above 40” to each whole record. The amounts 8 through 28 and 10 through 26 are retained; the amount 100 is excluded. Equality would be retained if a record had amount 40, because the rule says “above 40.”
Sample size. There are 12 records initially and 1 excluded, so \(n_R=12-1=11\).
Composition. Before trimming, Region A is \(6/12=50.0\%\) of the sample and Region B is \(6/12=50.0\%\). After trimming, Region A is \(6/11=54.5\%\), and Region B is \(5/11=45.5\%\), rounding percentages to one decimal place. Region B’s share falls because the deleted record belongs to that region.
Means. Region A’s total is \(8+12+16+20+24+28=108\), so its mean is \(108/6=18\). Region B’s original total is \(10+14+18+22+26+100=190\), giving a mean of \(190/6=31.6667\). The full total is \(108+190=298\), so the original overall mean is \(298/12=24.8333\). After trimming, Region B’s retained total is \(190-100=90\). The retained total across both regions is \(108+90=198\), and the trimmed mean is \(198/11=18.0000\). Means are rounded to four decimal places here.
Check. The decomposition gives \((11/12)(18)+(1/12)(100)=16.5+8.3333=24.8333\), agreeing with the original mean. The method is direct row-level trimming followed by recalculation; it avoids treating the removed value as if it were still part of the denominator.
When the Rule Selects a Different Kind of Case
A prespecified rule can be applied consistently and still remove valid observations. If the rule is based on a variable associated with the outcome or with a subgroup, the retained data may represent a systematically different mix of cases. In that situation, the trimmed mean or fitted relationship is conditional on passing the rule; it does not automatically answer the original question about all cases.
This is a selection issue, not just a question of whether the arithmetic was performed correctly. The decision should therefore consider why the rule is used, which cases it excludes, and what population the analysis is intended to describe. If an extreme value is a confirmed recording error, excluding or correcting it may be defensible. If it is a valid but unusual case, deleting it changes the analyzed population and should not be justified merely by its size.
Worked Example: Trimming High-Mileage Drivers
A hypothetical portfolio contains eight driver records with annual mileage, driver group, and annual claim count. Four standard-use drivers have counts 1, 2, 2, and 3, and all have mileage below 30,000. Four high-use drivers have mileage of 35,000, 42,000, 48,000, and 55,000, with respective claim counts 3, 4, 5, and 20. An analyst applies the fixed rule “exclude annual mileage above 45,000.” Compare the sample mean claim count and group composition before and after trimming.
Method. The mileage rule excludes the two high-use drivers at 48,000 and 55,000 miles. Their claim counts, 5 and 20, leave the analysis with the other six records. The two high-use drivers at 35,000 and 42,000 miles remain because their mileage does not exceed 45,000.
Sample size and composition. The original sample has 8 drivers: 4 standard-use and 4 high-use, so each group is 50.0%. The retained sample has \(8-2=6\) drivers: 4 standard-use and 2 high-use. The retained high-use share is \(2/6=33.3\%\), while the standard-use share is \(4/6=66.7\%\). Percentages are rounded to one decimal place.
Outcome means. The original claim-count total is \(1+2+2+3+3+4+5+20=40\); the original mean is \(40/8=5\). The retained total is \(40-5-20=15\), so the retained mean is \(15/6=2.5\). Within the retained data, the standard-use mean is \((1+2+2+3)/4=2\), and the high-use mean is \((3+4)/2=3.5\).
Interpretation and check. The rule removes valid high-mileage records under the scenario as stated, including the record with 20 claims. The lower retained mean cannot be described simply as an estimate for all drivers: both the high-use share and the outcome distribution have changed. The calculation checks because the deleted counts sum to \(5+20=25\), and \(40-25=15\). This example uses subgroup and outcome comparisons to assess selection; it avoids assuming that a fixed cutoff makes excluded cases invalid.
Keep Denominators Consistent
When observations represent different amounts of exposure, deleting a row also removes its exposure. A rate calculation must use the retained numerator and the retained denominator together. For example, if the outcome is a claim count and each record has an exposure amount, the crude rate is total claims divided by total exposure. Dividing retained claims by the original exposure would mix two different samples.
Worked Example: Trimming a Long Exposure Record
A hypothetical data set has six policy-period records. Their exposures, in policy-years, are 0.5, 1, 1, 1, 1, and 2; their claim counts are 0, 1, 0, 1, 0, and 3, respectively. A predetermined rule excludes records with exposure above 1.5 policy-years. Calculate the crude claim rate before and after trimming, using total claims divided by total exposure.
Method. The rule removes the sixth record, which has exposure 2 and claim count 3. The other five records are retained.
Original rate. Total exposure is \(0.5+1+1+1+1+2=6.5\) policy-years. Total claims are \(0+1+0+1+0+3=5\). The original crude rate is \(5/6.5=0.7692\) claims per policy-year, to four decimal places.
Retained rate. Retained exposure is \(6.5-2=4.5\) policy-years, and retained claims are \(5-3=2\). The trimmed crude rate is \(2/4.5=0.4444\) claims per policy-year, to four decimal places. Five of the six records remain, but the retained exposure is \(4.5/6.5=69.23\%\) of the original exposure.
Check and interpretation. The deleted record contributes 3 of the 5 claims and 2 of the 6.5 policy-years, so its removal changes both parts of the rate. The rate calculation is internally consistent because its numerator and denominator refer to the same five retained records. The method avoids dividing the trimmed claim count by the original exposure.
Common Mistakes and Practical Checks
- Using the wrong denominator for composition. In the first example, the retained Region B share is \(5/11\), not \(5/12\). The latter divides by the original sample size and does not describe the composition of the retained data.
- Deleting a value but not its row. Trimming removes an observation, including its other variables. Keeping the row’s predictor values while dropping only its outcome creates a different and generally inconsistent data set.
- Assuming an extreme value is an error. A rule identifies records for exclusion; it does not prove that those records are inaccurate. Check the data and context before interpreting unusual but plausible cases as invalid.
- Reporting only the new mean. Also report how many cases were removed and whether removal was concentrated in particular subgroups. A changed mean can reflect a changed sample composition.
- Using unmatched numerator and denominator. In a rate calculation, remove the deleted records’ outcomes and their exposure together.
A useful report states the rule, whether its boundary is inclusive, the number and fraction of observations removed, the retained sample size, subgroup composition before and after trimming, and the main summaries affected. Keeping an exclusion flag and preserving the original records makes the operation auditable; it does not require deleting source data. These checks clarify what the trimmed analysis describes without claiming that the rule has removed all bias.
Check Your Understanding
For each problem, apply the stated rule to whole observations and use the retained observations for post-trimming calculations.
- A set of measurements is 4, 8, 12, 30, and 35. The rule excludes values above 30. How many observations remain, and what is their mean? (Answer: 4 observations; mean \(13.5\).)
- A sample has four Group A values and four Group B values. Group A’s values are 10, 20, 30, and 40; Group B’s are 15, 25, 55, and 65. Exclude values above 50. What fraction of the original sample is Group B, and what fraction of the retained sample is Group B? (Answer: originally \(4/8=50.0\%\); retained \(2/6=33.3\%\).)
- Three records have exposure and claim-count pairs \((1,1)\), \((1,0)\), and \((2,2)\). Exclude records with exposure above 1.5. What are the original and retained crude claim rates, in claims per exposure unit? (Answer: original \(3/4=0.7500\); retained \(1/2=0.5000\).)
- Five records have predictor values 10, 20, 30, 40, and 50, with respective outcomes 2, 3, 4, 5, and 10. A rule retains predictor values at most 30. Find the retained outcome mean and state what share of the original records remains. Why should the retained mean not automatically be treated as the mean for all five cases? (Answer: retained mean \((2+3+4)/3=3\); \(3/5=60.0\%\) remain; the rule selects on a predictor and excludes two cases, so the retained data need not represent all five.)
- A data set has 10 observations, of which 4 belong to a subgroup. A trimming rule removes 3 observations, all from that subgroup. What are the subgroup’s original and retained shares? If the subgroup’s outcome total is 30 originally and the three deleted subgroup outcomes sum to 21, what are its original and retained outcome means? (Answer: original share \(4/10=40.0\%\), retained share \(1/7=14.3\%\); original mean \(30/4=7.5\), retained mean \((30-21)/(4-3)=9\).)