Robust Regression Methods for Clinical Data

Introduction

Clinical regression analyses are often presented as though the main task is to fit a model and report a coefficient. In practice, clinical datasets frequently contain features that make ordinary least-squares regression unnecessarily sensitive: extreme laboratory values, influential observations, heavy-tailed residuals, heterogeneous variability, or a small number of subjects with unusual combinations of covariates.

Robust regression provides a family of methods designed to make regression estimates less sensitive to observations that would otherwise have disproportionate influence on the fitted model. The goal is not to automatically discard unusual patients. Instead, robust methods change how much influence observations receive or change the loss function used to estimate the regression parameters.

Key idea: An observation can be clinically unusual without being erroneous. Robust regression is therefore best viewed as a sensitivity and modeling strategy, not as a mechanism for deleting inconvenient observations.

Why Ordinary Least Squares Can Be Sensitive

Consider the usual linear regression model:

Y_i=\beta_0+\beta_1X_i+\varepsilon_i.

Ordinary least squares estimates the coefficients by minimizing the sum of squared residuals:

\operatorname*{min}_{\beta_0,\beta_1} \sum_{i=1}^{n}(Y_i-\beta_0-\beta_1X_i)^2.

Because residuals are squared, a very large residual receives disproportionately large weight in the objective function. A single extreme observation can therefore move the fitted line substantially, particularly in a small clinical dataset.

This sensitivity is not necessarily a defect. Least squares has excellent statistical properties under its usual assumptions and is often the appropriate primary analysis. The issue arises when a small number of observations have much greater influence than the rest of the data and the scientific conclusion depends heavily on them.

Important distinction: A large residual is not automatically an outlier that should be removed. First determine whether the observation is valid, whether the value is clinically plausible, whether a data-processing error exists, and whether the observation is influential enough to affect the scientific conclusion.

What Does “Robust” Mean?

Robust methods attempt to reduce sensitivity to departures from ideal modeling conditions. In regression, this can mean reducing the contribution of large residuals, reducing the leverage of unusual predictor values, or obtaining standard errors that remain reliable under certain forms of heteroscedasticity.

Approach	Main problem addressed	Core idea
Robust regression	Influential residuals / outliers	Use a loss function that penalizes large residuals less aggressively
Huber regression	Large residuals	Quadratic loss near zero, approximately linear loss for large residuals
MM-estimation	Outliers and leverage	Combine high-breakdown estimation with efficient robust estimation
Quantile regression	Non-normal or heterogeneous outcomes	Model conditional quantiles rather than the conditional mean
Robust standard errors	Heteroscedasticity	Retain the fitted mean model but use variance estimates less sensitive to unequal variance

These approaches should not be treated as interchangeable. Robust regression changes the fitted regression estimator; robust standard errors primarily change inference about coefficients; and quantile regression answers a different scientific question.

When Robust Regression Is Useful in Clinical Data

Several recurring situations can motivate a robust analysis.

Continuous biomarkers contain a small number of very large values.
Clinical outcomes have heavy-tailed distributions.
Residual-versus-fitted plots show a few observations with unusually large residuals.
A small clinical study contains influential subjects whose inclusion materially changes the estimated treatment effect.
Measurement error or biological variability produces occasional extreme but plausible observations.
A sensitivity analysis is needed to determine whether a primary regression conclusion depends on a small number of observations.

For example, suppose a study compares change from baseline in a biomarker between treatment groups while adjusting for baseline biomarker level and prognostic covariates. If several subjects have unusually large changes, an ordinary least-squares estimate may be driven partly by those subjects. A robust estimator can provide a useful sensitivity analysis.

Start With Diagnostics, Not a Robust Estimator

The first step is to understand why a conventional regression appears problematic. A useful diagnostic sequence is:

Check the data: identify impossible values, unit errors, duplicate records, coding errors, and transcription problems.

Fit the prespecified or conventional model: examine the coefficient estimates, standard errors, residuals, and fitted values.

Inspect influence: examine leverage, studentized residuals, Cook's distance, and other influence diagnostics.

Investigate unusual subjects: determine whether the observations are clinically valid and whether they have a plausible scientific explanation.

Choose the sensitivity method: robust regression, transformed outcome, quantile regression, robust standard errors, or another appropriate model may be considered.

Compare conclusions: assess whether the treatment effect, uncertainty, direction, and substantive interpretation are stable.

Residuals, Leverage, and Influence

Three concepts are especially important.

Residual

The residual is the difference between an observed outcome and its fitted value:

e_i=Y_i-\widehat{Y}_i.

A large residual indicates that the model predicts the observation poorly, but it does not by itself mean that the observation strongly changes the regression coefficients.

Leverage

Leverage measures how unusual an observation's predictor values are relative to the rest of the design matrix. A subject can therefore have high leverage even when its outcome is close to the fitted regression surface.

Influence

Influence describes how much an observation affects the fitted results. Influential observations often combine unusual predictor values with a substantial residual.

Clinical interpretation: A patient with an extreme outcome is not necessarily influential, and a patient with unusual covariates is not necessarily an outlier. Robust regression is most useful when the distinction among residual size, leverage, and influence is understood.

Huber Regression

One of the most common robust estimators uses the Huber loss. Instead of squaring every residual, the loss is quadratic for small residuals and becomes approximately linear for large residuals.

\rho_c(r)= \begin{cases} \frac{1}{2}r^2, & |r|\le c,\\ c|r|-\frac{1}{2}c^2, & |r|>c. \end{cases}

Here \(r\) is a standardized residual and \(c\) is a tuning constant. Near zero, the Huber loss behaves like ordinary least squares. For large residuals, its growth is linear rather than quadratic.

The resulting estimator minimizes:

\operatorname*{min}_{\boldsymbol{\beta}} \sum_{i=1}^{n} \rho_c\!\left( \frac{Y_i-\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}} {\widehat{\sigma}} \right).

Thus, ordinary observations retain approximately least-squares behavior, whereas observations with very large residuals have reduced influence.

How Huber Weighting Works

Huber regression can be understood through iteratively reweighted least squares. Observations with small standardized residuals receive weight near one. As the residual becomes larger than the tuning threshold, its weight decreases approximately in proportion to \(1/|r|\).

Standardized residual	Approximate treatment under Huber loss
Small	Behaves like ordinary least squares
Moderate	Transition region
Very large	Reduced influence relative to squared-error loss

The important point is that the observation remains in the analysis. Its contribution is downweighted rather than automatically deleted.

Example: A Biomarker With an Extreme Observation

Suppose a randomized clinical study evaluates a continuous biomarker change using treatment group and baseline biomarker as covariates. Consider the following simplified data:

Subject	Treatment	Baseline	Change
01	Placebo	42	-3
02	Placebo	47	-5
03	Placebo	51	-4
04	Treatment	45	-12
05	Treatment	49	-11
06	Treatment	52	-13
07	Treatment	50	-42

The value of -42 should not be deleted simply because it is extreme. It should first be checked against the source data, units, visit date, treatment history, laboratory records, and clinical context. If it is valid, it represents a real patient observation.

A useful sensitivity analysis is to compare the conventional least-squares estimate with a robust estimate. If the treatment effect remains directionally and clinically similar, confidence in the conclusion may increase. If the results differ materially, that difference is itself an important finding.

Robust Regression Is Not Outlier Deletion

A common mistake is to use a robust method as a disguised data-cleaning procedure. These are different operations.

Action	What happens to the observation?	Scientific implication
Data correction	Erroneous value is corrected using source evidence	Addresses a data-quality problem
Exclusion	Observation is removed under a prespecified rule	Changes the analysis population
Robust regression	Observation remains, but its influence may be reduced	Provides resistance to unusual observations
Transformation	Outcome scale is changed	Changes the model's interpretation and error structure

Rule of thumb: If a value is wrong, correct it according to the data-management process. If it is valid but unusual, do not remove it merely to make a model look better. Consider robust or alternative modeling as a sensitivity analysis when justified.

MM-Estimation

Huber regression is useful, but robust estimation can face a difficult tradeoff: an estimator that is highly resistant to extreme observations may sacrifice some efficiency when the data are nearly ideal.

MM-estimation addresses this by combining a high-breakdown initial estimate with a subsequent efficient robust estimate. The exact implementation depends on the statistical software and tuning choices.

Conceptually, the procedure is:

Obtain an initial robust estimate of location and scale with strong resistance to contamination.

Use the robust scale estimate to standardize residuals.

Refine the regression coefficients using a bounded influence function.

Obtain an estimator that retains high robustness while recovering good efficiency under less contaminated conditions.

MM-estimation is particularly useful when there is concern about both contaminated outcomes and observations with high leverage. It is more specialized than simply fitting a Huber regression and should be described explicitly in the statistical methods.

Robust M-Estimation and Influence Functions

More generally, robust regression can be written as an M-estimation problem:

\widehat{\boldsymbol{\beta}} = \operatorname*{arg\,min}_{\boldsymbol{\beta}} \sum_{i=1}^{n} \rho(r_i),

where \(r_i\) is a standardized residual and \(\rho\) is a chosen loss function. Equivalently, estimation can be expressed through a score or influence function \(\psi\):

\sum_{i=1}^{n} \psi(r_i)\mathbf{x}_i=0.

For ordinary least squares, \(\psi(r)=r\), so very large residuals continue to exert increasingly large influence. Robust methods use a bounded or less rapidly increasing \(\psi\), limiting the effect of extreme residuals.

Robust Standard Errors Are Different

A frequent source of confusion is the phrase robust regression. Sometimes analysts actually mean heteroscedasticity-consistent standard errors.

Suppose the conditional mean model is:

E(Y\mid X)=X\beta,

but the variance changes with \(X\):

\operatorname{Var}(Y\mid X)=\sigma^2(X).

A heteroscedasticity-consistent covariance estimator can provide more reliable standard errors without replacing the least-squares coefficient estimator. This is fundamentally different from fitting a Huber or MM regression.

Question	Robust regression	Robust standard errors
Changes coefficient estimator?	Yes	Usually no
Addresses influential residuals?	Yes	Not primarily
Addresses heteroscedasticity?	Not necessarily	Yes, for inference under suitable conditions
Changes fitted values?	Potentially	No, if coefficients remain OLS

Do not conflate the two: If the problem is unequal variance, robust standard errors may be appropriate. If the problem is that a few observations dominate the coefficient estimates, a robust regression estimator may be more directly relevant.

Quantile Regression: A Different Robust Strategy

Quantile regression is often discussed alongside robust regression because it does not require the conditional outcome distribution to be centered around a mean with constant variance. However, it answers a different question.

Ordinary regression targets the conditional mean:

E(Y\mid X=x).

Quantile regression can instead model a conditional quantile \(Q_\tau(Y\mid X)\), such as the median when \(\tau=0.50\):

Q_\tau(Y\mid X=x)=x^\mathsf{T}\beta_\tau.

For clinical outcomes with skewness or heavy tails, median or upper/lower quantile effects may be more clinically informative than a mean effect. Quantile regression should therefore be selected because the target estimand is appropriate, not simply because it is called “robust.”

Robust Regression With Treatment Effects

A common clinical model might be:

Y_i= \beta_0+\beta_1\operatorname{Treatment}_i+ \beta_2\operatorname{Baseline}_i+ \beta_3X_{i3}+\varepsilon_i.

Here \(\beta_1\) represents the adjusted treatment-group difference under the specified coding and estimand. A robust regression sensitivity analysis re-estimates the same scientific contrast using a robust loss function.

The key comparison is not whether the robust coefficient is numerically identical to the OLS coefficient. The important questions are:

Does the estimated treatment effect have the same direction?
Is the magnitude clinically similar?
Does the uncertainty materially change?
Is statistical significance driven by a small number of subjects?
Does the robust analysis support the same clinical interpretation?

Clinical Example: Change From Baseline

Suppose a Phase II study compares mean change in a continuous biomarker. The primary ANCOVA-style model is:

\operatorname{Change}_i = \beta_0+ \beta_1\operatorname{Treatment}_i+ \beta_2\operatorname{Baseline}_i+ \varepsilon_i.

The conventional analysis may be appropriate as the primary analysis. A robust regression can then be specified as a sensitivity analysis if diagnostics show that a small number of valid observations have substantial influence.

The sensitivity analysis should preserve the same outcome definition, treatment contrast, covariates, and analysis population whenever possible. The principal difference should be the estimation method.

Analysis	Treatment estimate	Standard error	Interpretation
Primary OLS	Report prespecified estimate	Report conventional SE	Primary inferential result
Huber regression	Robust estimate	Method-specific robust SE	Sensitivity to large residuals
MM regression	Robust estimate	Method-specific robust SE	Sensitivity to contamination/influence
Quantile regression	Median or other quantile effect	Quantile-regression inference	Different estimand

R Example

In R, robust regression can be implemented with packages such as MASS and robustbase. For example, a Huber-type M-estimate can be fit with MASS::rlm().

library(MASS)

fit_ols <- lm(change ~ treatment + baseline, data = dat)

fit_huber <- rlm(
  change ~ treatment + baseline,
  data = dat,
  psi = psi.huber
)

summary(fit_ols)
summary(fit_huber)

The two models should be compared deliberately rather than by simply choosing the one with the smaller p-value.

For an MM-type estimator, a common approach is:

library(robustbase)

fit_mm <- lmrob(
  change ~ treatment + baseline,
  data = dat
)

summary(fit_mm)

Programming note: The exact default tuning constants, weighting rules, convergence criteria, and variance estimation depend on the implementation. For a regulated clinical analysis, document the software version, package version, estimation method, tuning choices, and any convergence or exclusion rules.

SAS Example

SAS provides several procedures and options for robust or resistant regression. The appropriate implementation depends on the specific estimator and analysis objective. For example, robust covariance estimation can be considered when the concern is heteroscedasticity rather than influential observations.

proc genmod data=dat;
    class treatment;
    model change = treatment baseline / dist=normal;
    repeated subject=subject / type=ind;
run;

For a robust regression estimator specifically, analysts should use the SAS procedure and options corresponding to the chosen estimator rather than describing any model with a sandwich covariance matrix as “robust regression.” The statistical programming specification should identify the estimator explicitly.

Interpreting a Robust Regression Output

A robust regression table should contain enough information for a reviewer to understand what was estimated.

Item	What to report
Outcome	Clinical endpoint and analysis time point
Model	Predictors, treatment coding, and covariate adjustment
Estimator	Huber M-estimation, MM-estimation, or other specified method
Tuning	Tuning constant or software default, if applicable
Effect estimate	Treatment coefficient or other prespecified contrast
Uncertainty	SE, confidence interval, and inference method
Diagnostics	Influential observations and relevant sensitivity findings

Confidence Intervals and Inference

Robust point estimates do not automatically imply robust inference. The variance estimator and confidence-interval construction must be compatible with the chosen regression method.

Depending on the estimator and sample size, inference may use asymptotic variance estimates, sandwich-type covariance estimates, or bootstrap methods. Small clinical datasets deserve particular caution because asymptotic approximations may be less reliable.

Small-sample caution: A robust point estimate can be useful even when the corresponding p-value or confidence interval is unstable. In small studies, emphasize the effect estimate, its uncertainty, diagnostic sensitivity, and clinical interpretation rather than treating a single robust p-value as definitive.

Bootstrap Sensitivity Analysis

Bootstrap methods can be useful for studying the sampling variability of robust estimators, particularly when the analytic variance calculation is complicated.

A basic nonparametric bootstrap repeatedly resamples subjects with replacement, refits the model, and examines the empirical distribution of the treatment effect:

\widehat{\beta}_1^{*(1)},\widehat{\beta}_1^{*(2)},\ldots, \widehat{\beta}_1^{*(B)}.

The resulting bootstrap distribution can be used to construct an appropriate confidence interval. However, bootstrap methods are not a universal solution for small samples, clustered data, missing data, or complex trial designs. The resampling unit must respect the study design.

Robust Regression for Repeated Clinical Measurements

Repeated measurements introduce another layer of complexity. If each patient contributes multiple observations, ordinary regression treating all rows as independent is generally inappropriate.

A robust strategy may instead involve a mixed-effects model, generalized estimating equations, or another repeated-measures framework, with robustness introduced through an appropriate residual or covariance strategy.

The principle remains the same: do not solve an outlier problem by ignoring the correlation structure of the clinical data.

Key point: Robust regression does not replace the need to model clustering, repeated measurements, treatment-by-time interactions, missingness, or other structural features of a clinical dataset.

Robust Regression for Survival Outcomes

Standard linear robust regression is not appropriate for time-to-event endpoints simply because the data contain extreme survival times. Survival outcomes require survival-specific models.

For example, a Cox proportional hazards model can be paired with robust variance estimation in settings involving clustering or correlated observations. That is a different use of the word “robust” from replacing the Cox partial-likelihood estimator with an M-estimator for continuous outcomes.

Therefore, the endpoint type should be established before selecting a robust method.

Robust Regression and Missing Data

Robust regression does not automatically solve missing-data problems. If subjects have missing outcomes or covariates, the missingness mechanism and analysis strategy still need to be addressed.

For a clinical trial, a sensitivity analysis should distinguish:

robustness to influential observed values;
robustness to modeling assumptions;
robustness to missing-data assumptions; and
robustness to analysis-population definitions.

These are separate dimensions of sensitivity analysis and should not be merged into one generic claim that an analysis is “robust.”

Choosing Among Robust Approaches

Observed issue	Potential approach	Primary question
One or a few very large residuals	Huber or another M-estimator	Does the coefficient remain stable when large residuals are downweighted?
Concern about leverage and contamination	MM-estimation / high-breakdown methods	Is the result resistant to influential contamination?
Unequal residual variance	Robust covariance / heteroscedasticity-consistent SE	Is inference reliable without constant variance?
Skewed outcome or interest in median	Quantile regression	What is the treatment effect on a conditional quantile?
Nonlinear mean relationship	Alternative functional form or nonlinear model	Is the mean model itself misspecified?
Repeated observations	Mixed model, GEE, or repeated-measures robust method	How should within-subject correlation be handled?

Common Mistakes

1. Calling every sandwich estimator “robust regression”

A sandwich covariance estimator can protect standard errors against certain variance misspecification while leaving the coefficient estimates unchanged. That is not the same estimator as Huber or MM regression.

2. Removing an observation before examining it

Clinical outliers should be investigated. A valid extreme response may be clinically important and may contain information about treatment heterogeneity.

3. Selecting the method after seeing which gives the desired p-value

Choosing a robust method solely because it produces statistical significance is poor practice. The rationale should come from the data-generating concerns, the estimand, the protocol or SAP, and the intended sensitivity analysis.

4. Treating robust regression as proof that the data are “clean”

Robust estimation reduces sensitivity; it does not certify that the data are correct or that the model is correctly specified.

5. Ignoring leverage

Some robust M-estimators primarily reduce the impact of large residuals. High-leverage observations require separate attention because an observation can have an unusual predictor pattern without an initially large residual.

6. Reporting only the robust result

When robust regression is a sensitivity analysis, the conventional primary analysis and the robust analysis should generally be presented together so readers can understand the impact of the modeling choice.

Robustness Checks for a Clinical Report

A useful sensitivity-analysis package can include several complementary comparisons:

Primary least-squares model.
Robust Huber or MM regression when justified.
Influence diagnostics.
Alternative outcome transformation when scientifically appropriate.
Robust covariance estimates when heteroscedasticity is the concern.
Alternative clinically reasonable covariate specifications.
Appropriate missing-data sensitivity analyses.

The objective is not to find one model that “wins.” The objective is to understand which assumptions are driving the estimated treatment effect.

Worked Interpretation

Suppose the primary analysis estimates an adjusted treatment difference of \(-8.4\) units with a 95% confidence interval of \(-12.7\) to \(-4.1\). A robust analysis gives \(-7.9\) units with a 95% confidence interval of \(-11.9\) to \(-3.8\).

The numerical estimates are not identical, but the clinical interpretation is similar: both analyses indicate a treatment-associated reduction of roughly eight units, and the uncertainty intervals overlap substantially.

Now suppose instead that the robust analysis gives \(-2.1\) units with a wide confidence interval spanning zero. That discrepancy deserves investigation. It suggests that the primary estimate may depend strongly on a small number of observations.

Interpretation principle: A disagreement between conventional and robust analyses is not a failure of the robust method. It is evidence that the estimated treatment effect is sensitive to the observations or assumptions being changed.

How to Report Influential Observations

A clinical report should avoid language suggesting that patients were removed because they were inconvenient. A more transparent description is:

identify the number of influential observations;
describe the diagnostic used;
state whether the observations were clinically valid;
explain whether they remained in the robust analysis;
quantify the effect on the treatment estimate; and
state whether the overall conclusion changed.

Patient-level identifiers should generally not appear in public-facing tutorial examples or reports unless they are necessary and appropriately protected.

Recommended Statistical Analysis Plan Language

When a robust regression is prespecified, the SAP should specify the estimator rather than merely stating that a “robust regression” will be performed.

Example framework: The primary analysis will use a linear regression model for change from baseline including treatment group and prespecified baseline covariates. As a sensitivity analysis, a robust regression estimator will be fitted using the specified robust loss function and tuning constant. The treatment contrast, covariate set, analysis population, and outcome definition will remain unchanged. The robust estimate and confidence interval will be compared with the primary analysis to assess sensitivity to influential observations.

The exact estimator, software implementation, variance method, confidence interval procedure, convergence criteria, and treatment of missing values should be specified before unblinded results are used to select among methods.

Practical Workflow

Define the clinical estimand and endpoint before choosing the regression method.

Fit the prespecified primary model.

Inspect residuals, leverage, and influence diagnostics.

Investigate unusual observations using clinical and data-management information.

Determine whether the issue is outlying residuals, leverage, heteroscedasticity, nonlinearity, skewness, dependence, or another modeling problem.

Select a robust or alternative model that directly addresses the identified issue.

Refit the model without changing unrelated analysis specifications.

Compare treatment estimates, confidence intervals, and clinical interpretation.

Document the estimator, tuning parameters, software, and inference method.

Report discrepancies transparently rather than selecting the result with the most favorable p-value.

Key Takeaways

Robust regression reduces the influence of observations that would otherwise disproportionately affect a regression fit.
Huber regression uses a loss function that is quadratic for small residuals and less aggressive for large residuals.
MM-estimation can provide strong resistance to contamination while retaining useful efficiency.
Robust standard errors and robust regression are different concepts.
Quantile regression is related to robust analysis but targets a different estimand.
Valid extreme observations should not automatically be deleted.
Diagnostics should precede the choice of a robust estimator.
For clinical trials, the primary model, estimand, analysis population, and covariates should remain clear when a robust sensitivity analysis is performed.
Disagreement between conventional and robust estimates is scientifically informative.
Software, tuning constants, variance estimation, confidence intervals, and convergence rules should be documented.

Bottom line: Robust regression is most valuable when it answers a clearly defined sensitivity question. In clinical research, the strongest practice is not to replace every ordinary regression with a robust estimator, but to understand the influence structure of the data, select an estimator appropriate to the identified problem, and determine whether the scientific conclusion remains stable.

References

Huber, P.J. (1964). Robust estimation of a location parameter. Annals of Mathematical Statistics, 35(1), 73–101.
Huber, P.J. & Ronchetti, E.M. (2009). Robust Statistics, 2nd ed. Wiley.
Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J. & Stahel, W.A. (1986). Robust Statistics: The Approach Based on Influence Functions. Wiley.
Rousseeuw, P.J. & Leroy, A.M. (1987). Robust Regression and Outlier Detection. Wiley.
Maronna, R.A., Martin, D.R., Yohai, V.J. & Salibián-Barrera, M. (2019). Robust Statistics: Theory and Methods (with R), 2nd ed. Wiley.
Koenker, R. & Bassett, G. (1978). Regression quantiles. Econometrica, 46(1), 33–50.