Why Reproducible Outlier Handling Matters
An outlier rule is only useful if another analyst can determine exactly which records it flags and why. A note such as “removed unusual values” is not reproducible: it does not state the threshold, how missing values were handled, whether the rule was recalculated, or what happened to the original observations. In an actuarial analysis, those details matter because outlier treatment can change summaries, fitted relationships, and decisions based on a model.
Earlier material on IQR fences, median and MAD screening, and the context of extreme observations provides several ways to identify values for review. This tutorial focuses on implementing the chosen rule transparently in R. A flag is a review signal, not proof that a value is wrong. Any correction or capping policy requires a separate, documented decision.
In R, reproducibility comes from making choices explicit. For example, a complete rule might say: “For non-missing values of claim amount, calculate type 7 sample quartiles using all eligible records; flag a value only if it is strictly below the lower IQR fence or strictly above the upper fence; do not alter values automatically.” This is more informative than a variable named outlier with no explanation of how it was created.
Encode the Rule Before Reviewing the Results
For the standard IQR rule, let \(Q_1\) and \(Q_3\) be the first and third quartiles and let \(\operatorname{IQR}=Q_3-Q_1\). With multiplier \(k=1.5\), the fences are \(L=Q_1-k\operatorname{IQR}\) and \(U=Q_3+k\operatorname{IQR}\). A non-missing observation \(x\) is flagged when \(x<L\) or \(x>U\). The strict inequalities mean that a value exactly on a fence is not flagged.
R’s quantile() function offers multiple quantile definitions. Its default is type = 7; specifying the type makes the choice visible and avoids relying on an unstated default. The na.rm argument also matters: with na.rm = TRUE, missing values are excluded from the quartile calculation, while the flag should generally remain missing or explicitly unflagged according to the analysis plan. Never let an implicit choice decide this for you.
The reason for the fence calculation is direct: it measures distance from the central half of the observed distribution in units of its interquartile spread. The lower fence is 1.5 IQRs below \(Q_1\), and the upper fence is 1.5 IQRs above \(Q_3\). Thus, the rule identifies values unusually distant from the middle half by this convention. It is a screening definition, not a probability statement or a claim that the data follow a particular distribution.
In a workflow, keep the original variable, the logical flag, and any edited value separate. Use a missing-value test so that an unknown observation is not mistaken for an ordinary non-outlier. Before applying a rule, check that the variable is numeric and that the quartiles can be calculated from the available observations. An all-missing field, for example, cannot produce meaningful IQR fences.
Worked Example: Creating IQR Flags with an Explicit Quantile Rule
A hypothetical insurer reviews twelve recorded claim amounts, in hundreds of dollars: 8, 9, 10, 10, 11, 12, 12, 13, 14, 15, 16, and 60. Use the type 7 IQR rule with \(k=1.5\), flag values strictly outside the fences, and show the R calculation and its result.
Method and rule. The observations are already in increasing order. With R’s type 7 definition, the quantile position is \(h=1+(n-1)p\). Here \(n=12\). For \(Q_1\), \(h=1+11(0.25)=3.75\), so R interpolates 75% of the way from the third to the fourth observation. Both are 10, giving \(Q_1=10\). For \(Q_3\), \(h=1+11(0.75)=9.25\): the ninth value is 14 and the tenth is 15, so \(Q_3=14+0.25(15-14)=14.25\).
The IQR is \(14.25-10=4.25\). The lower fence is \(10-1.5(4.25)=3.625\); the upper fence is \(14.25+1.5(4.25)=20.625\). Only 60 is strictly above 20.625, and no value is below 3.625.
R implementation. The code stores the source values, calculates the fences using the declared quantile type, and creates a flag without changing the amounts.
x <- c(8, 9, 10, 10, 11, 12, 12, 13, 14, 15, 16, 60)
q <- quantile(x, probs = c(0.25, 0.75), type = 7, na.rm = TRUE)
iqr <- unname(q[2] - q[1])
lower <- unname(q[1] - 1.5 * iqr)
upper <- unname(q[2] + 1.5 * iqr)
flag <- !is.na(x) & (x < lower | x > upper)
Check. The resulting quartiles are 10 and 14.25, the fences are 3.625 and 20.625, and the flag vector is FALSE for the first eleven values and TRUE for 60. The original vector x is unchanged. The common mistake avoided is reporting “60 was an outlier” without specifying the quantile definition, fence rule, or strict boundary convention.
Flags Are Not Edits
A flagged record should be reviewed against its source and context. A genuine large claim, a data-entry error, and a value measured on a different scale can all appear extreme numerically, but they call for different treatment. If review confirms an error and establishes the intended value, document the reason and keep the raw value available. If no correction is justified, retaining the value may be more appropriate than changing it.
A practical record-level audit trail includes a stable record identifier, the original value, the flag, the rule version or threshold, the final analysis value, and a reason for any edit. Preserving these fields makes it possible to reproduce the analysis and to explain exactly which observations were changed.
Worked Example: Correcting a Source-Verified Entry While Preserving It
A hypothetical policy dataset stores annual premiums in dollars. A review rule flags any premium above $50,000 for source verification; the threshold is a review trigger, not a deletion rule. Four records have premiums of $8,200, $12,500, $125,000, and $9,100. The source document confirms that the third record was entered with an extra zero and should be $12,500. Show the flag, the corrected analysis value, and the preserved source value.
Method. First retain the imported value in a source column. Then flag values strictly greater than 50,000. After verification, change only record 103’s analysis value, and record the reason. The other observations remain unchanged.
premium <- c(8200, 12500, 125000, 9100)
record_id <- c(101, 102, 103, 104)
dat <- data.frame(record_id = record_id, premium_original = premium)
dat$premium_flag <- !is.na(dat$premium_original) &
dat$premium_original > 50000
dat$premium_analysis <- dat$premium_original
dat$edit_reason <- NA_character_
dat$premium_analysis[dat$record_id == 103 & dat$premium_flag] <- 12500
dat$edit_reason[dat$record_id == 103 & dat$premium_flag] <-
"Source verified: extra zero in entry"
Check. The original values are still 8,200, 12,500, 125,000, and 9,100. The flags are FALSE, FALSE, TRUE, and FALSE. The analysis values are 8,200, 12,500, 12,500, and 9,100. Exactly one analysis value changed, and the reason identifies the evidence for the change.
The common mistake avoided is overwriting premium directly. Doing that would erase the imported value and make it harder to audit or reverse the edit. Another trap is to replace every flagged amount with a cap: the stated rule only requested review, and the evidence supports a specific correction for one record.
Apply Approved Bounds Consistently
Sometimes a documented analysis policy calls for winsorization: values outside fixed lower and upper bounds are replaced by the nearest bound. This is an edit, not merely a flag. The bounds must be justified and stated, and the original value should remain available. When applying a policy to new or validation data, use the already established bounds rather than recalculating them from that data; otherwise the meaning of the transformation changes with the batch being processed.
For a missing \(x\), the edited result should remain missing. A corresponding flag can identify observations outside the bounds before editing. Check the bounds and their order before use: \(L\leq U\) is required. Also inspect the boundary convention; under the rule above, observations equal to either bound are retained and are not flagged.
Worked Example: Applying Fixed Winsorization Bounds
A hypothetical model uses a processing-time predictor in minutes. After review, an analysis policy specifies winsorization at fixed bounds of 10 and 100 minutes. A later batch contains 8, 10, 54, 100, 125, and a missing value. Create flags and edited values using those fixed bounds.
Method. Flag only non-missing values strictly below 10 or strictly above 100. Then replace flagged low values with 10 and flagged high values with 100. The rule preserves values equal to the bounds and leaves missing values missing.
time_original <- c(8, 10, 54, 100, 125, NA_real_)
lower <- 10
upper <- 100
time_flag <- !is.na(time_original) &
(time_original < lower | time_original > upper)
time_edited <- pmin(upper, pmax(lower, time_original))
Check. The flags are TRUE, FALSE, FALSE, FALSE, TRUE, and FALSE. The edited values are 10, 10, 54, 100, 100, and NA. The value 10 is not flagged because it is on the boundary; 8 is flagged and raised to 10; 125 is flagged and reduced to 100. R’s pmin() and pmax() retain the missing value here because the default is not to remove missing arguments.
The common mistake avoided is recalculating bounds from the later batch. The values in that batch do not define the policy: the approved bounds remain 10 and 100. Also, overwriting time_original would lose the information that the two extreme recorded values were 8 and 125.
A Reviewable R Workflow
A disciplined sequence keeps screening, review, and editing distinct. It also makes it easier to rerun the process after a data refresh and compare the results. The key is to store the rule alongside the derived fields, not to rely on memory or an undocumented manual spreadsheet adjustment.
Keep an unchanged source field and a stable record identifier. Record which data extract and variable are being checked.
Record the method, thresholds, strict or inclusive comparisons, treatment of missing values, and—when relevant—quantile type and multiplier.
Review the count and the flagged records. Confirm that the flags agree with the stated rule, especially at boundaries and for missing observations.
Apply only the approved correction or transformation in a separate analysis field. Record its reason and retain the original value.
Check that edits occurred only where intended, missingness was handled as planned, and the resulting values satisfy the policy.
For example, useful verification checks include counting TRUE flags, confirming that corrected records have non-missing reasons, and comparing the source and analysis columns to find changed values. A rule should also be tested on values exactly at its lower and upper thresholds. Such tests catch common implementation errors, including accidentally using <= when the documented rule uses <.
Reproducibility does not mean that every run must produce identical flags if the underlying data have changed. It means that the method and its inputs are clear enough to explain any difference. For a data-dependent rule such as IQR fences, save or report the calculated thresholds and the data version used. For fixed policy bounds, retain the approved values and apply them consistently.
Check Your Understanding
For each situation, identify the correct flag or edit under the stated rule. Treat boundary comparisons and missing values exactly as specified.
- A numeric field contains 4, 7, 9, 11, and 40. An analyst uses fixed review bounds of 5 and 30, flagging only non-missing values strictly outside the bounds. Which values are flagged? (Answer: 4 and 40.)
- A type 7 IQR calculation gives \(Q_1=20\) and \(Q_3=28\). Using \(k=1.5\) and strict fence comparisons, calculate both fences and decide whether 40 is flagged. (Answer: IQR = 8; lower fence = 8; upper fence = 40; 40 is not flagged.)
- A source field contains 12, 999, and 18. A review rule flags values above 100, and the source verifies that 999 should have been 19. What should the flag vector and separate corrected analysis field be? (Answer: Flags FALSE, TRUE, FALSE; analysis values 12, 19, 18, with 999 retained in the source field.)
- A fixed winsorization policy uses bounds 0 and 50. Apply it to -3, 0, 22, 50, 61, and NA, flagging only non-missing values strictly outside the bounds. (Answer: Flags TRUE, FALSE, FALSE, FALSE, TRUE, FALSE; edited values 0, 0, 22, 50, 50, NA.)
- An IQR rule is to be applied to a later validation dataset using thresholds calculated from the training dataset. The training fences are -2 and 14; validation values are -2, 3, 14, and 16. Using strict comparisons, which validation values are flagged, and should the validation fences be recalculated? (Answer: Only 16 is flagged; do not recalculate the fences.)