Least Trimmed Squares and High-Breakdown Fitting

Why Trim the Residuals?

Least squares uses every squared residual in its objective. As a result, even one extremely large response residual can dominate the fit. Robust regression methods change how residual size contributes to fitting; least trimmed squares (LTS) takes a different approach from methods that gradually reduce the contribution of large residuals. It minimizes the sum of only the $h$ smallest squared residuals.

The central idea is to let the fit be determined by a substantial subset that is internally consistent, rather than requiring every observation to agree with that fit. If a minority of responses have been contaminated, their residuals may be large and may therefore be left out of the objective. The observations are not thereby proved erroneous: trimming is a fitting rule, not a data-cleaning decision.

Definition: For $n$ observations and a regression model with coefficient vector $\boldsymbol{\beta}$, let $e_i(\boldsymbol{\beta})=y_i-\hat{y}_i(\boldsymbol{\beta})$ be the residual for observation $i$. Sort the squared residuals from smallest to largest as $e_{(1)}^2(\boldsymbol{\beta})\leq\cdots\leq e_{(n)}^2(\boldsymbol{\beta})$. The least trimmed squares objective for a chosen integer $h$ is $$ Q_h(\boldsymbol{\beta})=\sum_{j=1}^{h}e_{(j)}^2(\boldsymbol{\beta}). $$ An LTS estimate is a coefficient vector that minimizes $Q_h(\boldsymbol{\beta})$.

Here $h$ is the number of residuals retained in the objective, not the number of coefficients. A smaller $h$ permits more observations to be excluded from the sum, but also bases the fit on fewer observations. For a model with $p$ coefficients, including an intercept when present, $h$ must be large enough that retained subsets can identify the coefficients; in ordinary settings this requires at least $p$ observations with a full-rank predictor design. The choice $h=n$ gives the least-squares objective. Values below $n$ allow trimming.

What the Objective Does

The ordering in the definition depends on the candidate coefficients. As the coefficients change, the residuals change, and so may the set of $h$ observations with the smallest squared residuals. Thus LTS does not first choose a permanent subset and then fit it once. It searches for coefficients and a subset that work well together.

A useful equivalent expression makes this explicit. For any fixed candidate coefficients, the sum of the $h$ smallest squared residuals is the smallest sum obtainable by selecting any subset of $h$ observations. Therefore,

Q_h(\boldsymbol{\beta}) = \min_{\substack{S\subseteq\{1,\ldots,n\}\\|S|=h}} \sum_{i\in S}e_i^2(\boldsymbol{\beta}).

To justify the equality, list all $n$ squared residuals in increasing order. Any subset of $h$ entries must include $h$ values from that list. Its $j$th-smallest selected value cannot be less than the $j$th-smallest value in the full list. Summing those $h$ inequalities shows that no subset can have a smaller sum than the first $h$ values. Selecting exactly those first $h$ values attains the minimum. Consequently, LTS can also be understood as finding the best-fitting subset of size $h$, while optimizing the regression coefficients.

Worked Example: Ordering Residuals for an LTS Objective

Suppose a candidate regression fit has residuals $-0.2, 0.1, 0.3, -0.4, 5,$ and $-8$, and the chosen retained count is $h=4$. Find this candidate's LTS objective.

First square each residual. In the original order, the squared residuals are $0.04, 0.01, 0.09, 0.16, 25,$ and $64$. In increasing order, they are $0.01, 0.04, 0.09, 0.16, 25,$ and $64$.

The objective adds the first four:

$$ Q_4=0.01+0.04+0.09+0.16=0.30. $$

The two large squared residuals, $25$ and $64$, do not contribute to this candidate's objective. This calculation does not mean those observations are permanently discarded: a different candidate fit could change the residual ordering.

Resistance to Contaminated Responses

Response contamination means that some observed response values have been replaced or corrupted, while the predictor values are treated as fixed. Such observations can have very large residuals under a fit to the uncontaminated pattern. Least squares squares and sums every residual, so large values can strongly pull its coefficients. LTS can limit that effect when the contaminated observations fall among the residuals excluded from its $h$-term sum.

This resistance has a precise but conditional meaning. For a fixed predictor design in general position, the finite-sample replacement breakdown fraction of LTS is commonly expressed, for $p$ fitted coefficients, as

\varepsilon^*_{\mathrm{LTS}} = \min\left\{\frac{n-h+1}{n},\frac{h-p+1}{n}\right\}.

The replacement breakdown fraction is the smallest fraction of observations that can be replaced so that the estimate can become arbitrarily large or otherwise unbounded. In this formula, the first term reflects how many replacements can force the fit to rely on contaminated observations among the $h$ residuals it retains. The second reflects the need for enough uncontaminated observations to constrain a model with $p$ coefficients. The expression is a finite-sample result under its design conditions, not a guarantee against every form of unusual data.

The two terms pull in opposite directions. Increasing $h$ reduces the amount trimmed and increases the number of observations that must fit well, but it also changes how many replacements can disrupt the retained subset. Choosing $h$ near the balance between the two terms can give a high breakdown fraction. The examples below illustrate the arithmetic; the dimensions and design conditions matter as much as the number alone.

Worked Example: A High-Breakdown Choice with Twenty Observations

Consider $n=20$ observations, a straight-line regression with an intercept and slope ($p=2$), and $h=11$. Under the stated fixed-design and general-position conditions, calculate the breakdown fraction.

Substitute the values in the formula:

\varepsilon^*_{\mathrm{LTS}} = \min\left\{\frac{20-11+1}{20},\frac{11-2+1}{20}\right\} = \min\left\{\frac{10}{20},\frac{10}{20}\right\} = \frac{10}{20}=0.50.

Thus, under those conditions, ten replacements reach the breakdown count associated with this fraction; nine replacements are below that count. The remaining-observation arithmetic is important: replacing ten of twenty leaves $20-10=10$ observations unreplaced, while replacing nine leaves $20-9=11$. These counts describe how many observations remain, not additional breakdown fractions. The result illustrates substantial response-contamination resistance, but it does not imply that every set of ten replacements will make the fit unbounded.

Worked Example: How the Choice of h Changes the Limit

Take $n=12$ observations and a regression with $p=2$ coefficients. Compare the breakdown fractions for $h=11$ and $h=7$, assuming the same design conditions.

For $h=11$,

\varepsilon^*_{\mathrm{LTS}} = \min\left\{\frac{12-11+1}{12},\frac{11-2+1}{12}\right\} = \min\left\{\frac{2}{12},\frac{10}{12}\right\} = \frac{1}{6}\approx 0.167.

For $h=7$,

\varepsilon^*_{\mathrm{LTS}} = \min\left\{\frac{12-7+1}{12},\frac{7-2+1}{12}\right\} = \min\left\{\frac{6}{12},\frac{6}{12}\right\} = \frac{1}{2}=0.50.

The smaller retained count gives the larger breakdown fraction in this comparison. For $h=11$, the first term is the limiting one; for $h=7$, the two terms are equal. A high breakdown fraction is therefore not obtained simply by choosing the smallest possible $h$: retaining enough uncontaminated observations to identify the coefficients also matters.

A Small Location Example

A one-coefficient intercept-only model helps show concretely how an extreme response can be left out of the objective. In this model, every fitted value is the same location, and the residual for an observation is its response minus that location. The example is illustrative rather than a claim about a real data set.

Worked Example: Four Retained Responses Among Six

Suppose the six responses are $2,3,3,4,4,100$, and use LTS with $h=4$. Find a minimizing location and compare its objective with the least-squares location.

For any selected group of four responses, the sum of squared deviations is minimized at that group's mean. The four values $3,3,4,4$ have mean $3.5$, and their sum of squared deviations is

$$ (3-3.5)^2+(3-3.5)^2+(4-3.5)^2+(4-3.5)^2 =0.25+0.25+0.25+0.25=1. $$

At the location $3.5$, the six squared residuals are $2.25,0.25,0.25,0.25,0.25,$ and $9312.25$. The four smallest sum to $1$. For any fixed location, the four smallest squared residuals can be selected from four consecutive responses in sorted order; therefore, a global minimum can be found by minimizing the within-group sums over the three consecutive blocks of four. The other possible groups of four consecutive values in the ordered responses have larger within-group sums: $2,3,3,4$ has sum $2$, and $3,4,4,100$ has a much larger sum. Thus the group $3,3,4,4$ gives a minimizing location, $3.5$, with LTS objective $1$.

For comparison, the least-squares location is the mean of all six responses:

\bar{y}=\frac{2+3+3+4+4+100}{6}=\frac{116}{6}\approx 19.33.

The extreme response makes the all-observation mean much larger than the location determined by the four central values. LTS instead minimizes the squared residuals of four observations; it does not assert that the response $100$ is invalid. Whether that observation reflects an error, a rare but genuine outcome, or a different data-generating pattern requires contextual investigation.

Interpretation, Assumptions, and Limits

LTS is a fitting method, not an automatic test for bad observations. Its response-contamination protection is strongest when the uncontaminated observations follow a coherent regression pattern, enough such observations remain to identify the coefficients, and the contaminated responses produce large residuals under that pattern. The breakdown formula also assumes the specified fixed predictor design and general-position conditions. It does not establish resistance to every possible configuration of predictors and responses.

Because LTS selects a subset through residual ordering, the minimizing subset can be sensitive to small changes when several candidate subsets fit similarly. Different choices of $h$ can also produce different coefficients. A robust fit should therefore be interpreted alongside the data, the modeling goal, and appropriate sensitivity checks. A coefficient from LTS describes the fitted robust pattern; it should not automatically be described as the same target as a least-squares coefficient.

LTS is also distinct from methods that assign continuously varying residual weights. Its defining operation is to minimize the squared residual sum over the smallest $h$ residuals. This simple rule is useful in the presence of a substantial fraction of response outliers, but computing the global minimum can require searching among many subsets. Practical algorithms are designed to find good solutions efficiently; the mathematical definition remains the global minimization stated above.

Common Mistake

A common mistake is to say, “The largest residuals were trimmed, so those observations should be deleted.” That conclusion does not follow. LTS excludes some residuals from its fitting objective to reduce their influence on the coefficients; it neither proves that the corresponding records are mistakes nor decides how they should be handled in the analysis. Retain and inspect the observations, check their context and measurement process, and explain clearly whether a reported fit is an LTS fit or a fit using all observations.

Key takeaway: LTS minimizes the sum of the $h$ smallest squared residuals. Choosing $h$ controls the balance between trimming potentially contaminated responses and retaining enough observations to estimate the model. A high breakdown fraction describes resistance under specified conditions, not a license to ignore or delete unusual data.

Check Your Understanding

Work through these questions to check your understanding of least trimmed squares.

For residuals $0.5,-1,2,-4$ and $h=2$, what is the LTS objective for this candidate fit?
Explain why the sum of the $h$ smallest squared residuals equals the minimum squared-residual sum over all subsets of size $h$.
With $n=16$, $p=2$, and $h=9$, calculate the finite-sample breakdown fraction using the stated formula.
What does the LTS objective’s exclusion of a large residual establish about the observation that produced it, and what does it not establish?
Why can making $h$ smaller increase resistance to response contamination while also creating a concern about estimating the regression coefficients?