Sensitivity Fits for Competing Mean Structures

Why Fit More Than One Mean Structure?

A regression model summarizes how the mean response changes with predictors. A straight-line relationship may be a useful first description, but residual patterns can suggest that the mean changes in a curved way or differs across groups. Fitting a small number of sensitivity fits helps show whether a reasonable alternative changes the fitted relationship and whether it leaves a clearer or less systematic residual pattern.

The aim is not to try every possible formula until one appears significant. Candidate models should be motivated by the question, the way variables were measured, or a specific diagnostic clue. For example, a curved term may be considered when residuals show a curved pattern against a predictor; an interaction may be considered when residual patterns differ by group. Earlier material on specification checks and tests helps identify such candidates. Here the emphasis is on comparing a limited set of defensible fits.

Conditions for a Fair Comparison

Write a candidate mean structure as \(E(Y_i\mid \mathbf{x}_i)=m(\mathbf{x}_i;\boldsymbol{\beta})\), where \(Y_i\) is the response for observation \(i\), \(\mathbf{x}_i\) denotes its predictors, and \(\boldsymbol{\beta}\) is a vector of coefficients. The fitted value is \(\hat{y}_i\), and the residual is \(e_i=y_i-\hat{y}_i\). A residual is the observed response minus the fitted response; positive residuals mean the observation is above its fitted value.

For ordinary least squares comparisons in this tutorial, assume the observations are independent, the conditional mean is represented adequately by the candidate structure, and the error variance is constant across predictor settings. Normal errors are additionally needed for standard small-sample inference, but a residual-pattern comparison is not itself a significance test. The regression diagnostic assumptions should be considered for each candidate; changing the mean structure does not repair dependence or unequal variance automatically.

A fair comparison uses the same response, observations, and measurement scales. If one candidate uses fewer observations because of missing predictor values, a change in RSS may reflect the changed data set rather than the changed structure. Interpret fitted relationships over the predictor range actually represented in the data. An apparently plausible curve or group-specific line within that range does not establish that it will continue beyond it.

The residual sum of squares is \(\mathrm{RSS}=\sum_{i=1}^{n}e_i^2\). For ordinary least squares, adding terms cannot increase RSS when the smaller model is nested in the larger one and both are fitted to the same observations. This follows because the larger model can reproduce every fit available to the smaller model, while also allowing additional fits. Minimizing squared residuals over a larger set of possible fits therefore cannot yield a larger minimum. That fact alone does not show that the extra terms are useful: added flexibility usually lowers RSS, even when the apparent improvement is mostly fitting noise.

Compare the Relationships, Not Just the Fit Statistic

For each candidate, examine whether residuals still show recognizable structure against predictors or fitted values, and whether the fitted relationship is sensible in context. Predictor-specific residual smooths, component-plus-residual plots, and residual strata can help reveal the relevant pattern. These displays complement, rather than replace, consideration of the data collection and the substantive meaning of the model.

RSS summarizes squared residual size, but it does not show where those residuals occur. A model may have a smaller RSS while still leaving a systematic pattern, or may flatten a pattern by using extra parameters without a convincing reason. The residual standard error, \(\sqrt{\mathrm{RSS}/(n-p)}\), where \(p\) counts the estimated coefficients including the intercept, describes residual size on the response scale under the model. When models have different numbers of coefficients, its denominator changes too, so it should be read as a descriptive summary rather than a universal selection rule.

Compare Group Relationships With and Without an Interaction

An additive model with a group indicator and a quantitative predictor allows groups to have different levels but imposes a common slope. Adding an interaction allows the slope to differ by group. When residual patterns differ across groups or predictor ranges, comparing these fitted relationships can reveal whether the common-slope restriction is visibly inadequate.

Use a Small, Purposeful Set of Alternatives

The candidates should answer a specific diagnostic question. A linear and quadratic fit address whether curvature may matter; additive and interaction fits address whether a common predictor relationship across groups may be too restrictive. If there are several plausible alternatives, state why each was included before comparing their residual patterns. This keeps the comparison interpretable and reduces the temptation to select a model simply because it happens to look best among many trials.

For each candidate, record the terms included, the observations used, the fitted relationship, and the main residual features. Then ask whether the alternative addresses the suspected pattern without introducing implausible behavior or unnecessary complexity. A more elaborate model can improve in-sample fit while making the fitted relationship difficult to interpret, especially where data are sparse.

Common Mistakes and Why They Matter

• Choosing the model with the smallest RSS automatically. A larger model has more freedom and, when nested, cannot have higher minimum RSS. Check whether it resolves a meaningful residual pattern and whether its extra terms are defensible.
• Reading residual size without checking residual order or grouping. A single summary can conceal curvature, group-specific patterns, or drift. Inspect residuals in relation to the predictors and fitted values that motivated the candidate.
• Comparing fits based on different observations as if only the formulas changed. Different data can change RSS and residual patterns. Use the same observations where possible, and disclose when that cannot be done.
• Calling a sensitivity fit a causal result. Changing the mean structure changes the statistical description; it does not, by itself, establish that a predictor causes a response change.

A small arithmetic slip can also alter a comparison. For example, if six residuals each have magnitude \(0.5\), their contribution to RSS is \(6(0.5^2)=6(0.25)=1.5\), not 1. Always square each residual before summing.

Sensitivity fits matter because conclusions about the mean pattern can depend on a modeling choice. Reporting a small, motivated comparison makes that dependence visible: readers can see which relationships were considered, what residual structures remained, and why one specification was judged more useful for the stated purpose. The next step in a broader analysis may be to study how coefficients change across those specifications, but that is a distinct question from the residual-pattern comparison made here.