Sensitivity Analysis for Outlier Treatments

Why Compare Outlier Treatments?

A flagged value can affect an analysis even when it is plausible and has been checked. Retaining it, capping it, or excluding its observation changes the data used by a summary or model. The resulting estimates may differ enough to alter a practical conclusion. Sensitivity analysis makes that dependence visible: repeat the same analysis under defensible treatments and compare the results.

The purpose is not to find the treatment that produces the most convenient estimate. A flag is a reason to investigate a value, not proof that the value is wrong. Context and data quality determine which treatments are reasonable to consider. If the value is verified and relevant to the target population, retaining it may be the primary analysis; capping and exclusion can still help show how much the result depends on it.

The three treatments have different meanings. Retention uses the recorded value unchanged. Capping replaces the value with a stated boundary, such as a defensible upper cap. Exclusion removes the observation from the analysis, so the sample size and potentially the composition of the analyzed data change. These are not interchangeable data edits: each changes what the analysis represents.

A fair comparison starts with one analysis question and one set of flagged observations. Do not quietly change the outcome, predictor set, inclusion rules, or flag definition between runs. Record the estimate, the sample size, and the conclusion under each treatment. If a treatment is not defensible for the data, do not include it merely to complete a three-column table.

What Changes When a Value Is Capped or Excluded?

For a numeric outcome \(y_1,\ldots,y_n\), the sample mean is the sum of the included values divided by the number of included observations. Suppose one flagged value is \(y_j\), and a proposed cap replaces it with \(c\). The original and capped means are:

Subtracting the first expression from the second gives a useful direct comparison:

This identity holds because capping changes one value by \(c-y_j\) while leaving the denominator \(n\) unchanged. It does not describe exclusion: exclusion removes both \(y_j\) and one observation from the denominator. If \(T=\sum_{i=1}^{n}y_i\), the mean after excluding observation \(j\) is \((T-y_j)/(n-1)\), provided \(n>1\).

For a model, the same idea applies at the level of the chosen model estimate. A regression coefficient may change because a treatment changes an outcome, a predictor, or which rows are included. Unlike the mean identity above, there is generally no one-value shortcut that gives the new coefficient; refit the model to the treated data.

Repeat the Model, Not the Question

In a simple linear regression of a quantitative response \(Y\) on a quantitative predictor \(X\), the fitted slope is the estimated change in the mean response associated with a one-unit increase in the predictor under the model. The ordinary least squares slope is the sample covariance of \(X\) and \(Y\), using divisor \(n\), divided by the sample variance of \(X\), also using divisor \(n\). Equivalently, it is \(S_{XY}/S_{XX}\), where \(S_{XY}=\sum_i(x_i-\bar{x})(y_i-\bar{y})\) and \(S_{XX}=\sum_i(x_i-\bar{x})^2\). This formula requires \(S_{XX}>0\); if all predictor values are identical, the slope cannot be estimated.

A treatment that changes a response can change \(S_{XY}\), and therefore the slope. Excluding a row can also change \(\bar{x}\), \(\bar{y}\), \(S_{XY}\), and \(S_{XX}\). Consequently, compare refitted models on their own treated data, and state the sample size for each. In a practical interpretation, distinguish whether the direction of association changes from whether its estimated magnitude changes.

Interpret the Difference, Not Just the Numbers

Sensitivity is a property of the conclusion relative to a specified treatment set. It is useful to ask whether estimates keep the same sign, remain in a similar range, or cross a decision threshold that matters in context. A small numerical change may be important near a threshold, while a larger change may not alter the operational decision. Always explain the comparison in the units and context of the outcome.

Exclusion deserves particular care. If extreme observations are more likely in a subgroup or under certain conditions, removing them can change which cases the analysis represents. The excluded-data result may describe a selected subset rather than the intended population. This is why sensitivity analysis should accompany, not replace, a defensible primary treatment and a clear description of the target population.

Likewise, capping changes the meaning of the recorded variable: all values above the cap are represented at the boundary. A cap should have a stated rationale, such as a known measurement ceiling or a pre-specified analytic rule. Choosing a cap solely because it produces a preferred conclusion is not a defensible analysis.

A useful report does not merely say that results are “robust” or “not robust.” It specifies what remained stable and what changed. For example: “The estimated association remained positive under all three treatments, while its magnitude decreased substantially when the flagged response was capped or excluded.” That statement separates directional stability from magnitude sensitivity.

Do not interpret treatment comparisons as a formal test of whether an observation is an outlier, nor as evidence that the data-generating process follows a particular distribution. They are conditional comparisons: given the stated data and analysis, how do results change under these alternative handling choices? Further modeling or validation may be needed to answer different questions.