What a Regression Coefficient Interval Leaves Out

What a Coefficient Interval Conditions On

A regression coefficient interval is useful, but it answers a narrower question than researchers sometimes suppose. A conventional interval describes uncertainty in a coefficient estimate under a specified regression model, with the variables as measured and the target population defined by the analysis. It does not automatically account for every uncertainty involved in choosing that model, measuring those variables, or deciding whom the result should describe.

For a coefficient \(\beta_j\), a conventional two-sided interval has the form

\hat{\beta}_j \mathbin{\pm} t^* SE(\hat{\beta}_j)

Here \(\hat{\beta}_j\) is the fitted coefficient, \(SE(\hat{\beta}_j)\) is its standard error under the fitted model, and \(t^*\) is a critical value chosen for the confidence level and the relevant degrees of freedom. The conditions behind conventional coefficient intervals include an appropriate model and sampling process; the earlier material on those conditions explains them in detail.

The important word is under. The interval allows for the coefficient estimate to vary across repeated samples when the analysis setup is treated as fixed. It is not automatically an interval for uncertainty about the setup itself. Three questions help make the distinction concrete:

Would the interval change if a different reasonable model had been chosen?
Are the recorded predictor and outcome values accurate enough to treat as known?
Does the population represented by the analysis match the population the conclusion is meant to describe?

If an answer raises a concern, the conventional interval may still be calculated correctly for its stated setup. The concern is that it may not capture all the uncertainty relevant to the broader scientific or practical question.

A Conventional Interval as a Conditional Summary

Worked Example: Reading the Scope of an Interval

Suppose a hypothetical, prespecified linear regression relates a patient's age in years to a measured recovery time in days. The fitted age coefficient is \(2.40\) days per year, its standard error is \(0.80\) days per year, and the degrees of freedom are \(24\). Assume the model and sampling conditions for a conventional interval are reasonable: the relationship is approximately linear, residual variability is roughly constant, observations are independent, and the residual distribution is adequate for the small-sample \(t\) procedure. The study's measurement protocol and target population are treated as fixed for this calculation.

For a 95% interval, take \(t^*=2.064\). The margin of error is

2.064(0.80)=1.6512\ \text{days per year}.

Therefore,

2.40\mathbin{\pm}1.6512=(0.7488,\ 4.0512)\ \text{days per year}.

Rounded suitably, the conventional interval is \(0.749\) to \(4.05\) additional recovery days per year of age, within the specified regression analysis. Its ordinary repeated-sampling interpretation relies on the model and conditions just stated. It does not show how much the estimate would change under another model, how accurate the measurements are, or whether patients outside the study's target population would have the same relationship. Those are separate questions, not extra conclusions supplied by these endpoints.

Model-Choice Uncertainty

A conventional interval is calculated after a model has been specified. If the model was selected after examining many candidate models or predictors, an ordinary interval usually treats the selected model as though it had been fixed in advance. It then reflects coefficient uncertainty within that fitted model, not the additional uncertainty created by the search that led to it.

Model choice can matter because different specifications may estimate different quantities or may produce different values for a coefficient with the same label. For example, an age coefficient adjusted for one set of predictors is not necessarily the same quantity as an age coefficient adjusted for another set. The intervals from alternative models should not be treated as a single formal interval for one parameter unless that parameter and the method for combining results have been explicitly defined.

Worked Example: A Coefficient After Choosing Between Models

In a hypothetical analysis of \(60\) patients, an analyst examines the data and considers two models for recovery time. Model A includes age alone. Model B includes age and a second predictor. Suppose the ordinary two-sided critical value is approximately \(2.00\) for each fit. In Model A, the age coefficient is \(1.80\) days per year with standard error \(0.70\). In Model B, it is \(0.90\) days per year with standard error \(0.80\). Assume each fitted model is otherwise adequate for its own conventional interval.

For Model A, the margin is \(2.00(0.70)=1.40\), giving

1.80\mathbin{\pm}1.40=(0.40,\ 3.20)\ \text{days per year}.

For Model B, the margin is \(2.00(0.80)=1.60\), giving

0.90\mathbin{\pm}1.60=(-0.70,\ 2.50)\ \text{days per year}.

Each interval summarizes coefficient uncertainty conditional on its model. Neither interval accounts for the fact that the analyst looked at the data before choosing between the models. Nor should the analyst report the union of the two intervals as though it were a calculated confidence interval: the coefficients may have different interpretations, and that union has no stated coverage guarantee. A response to model-choice uncertainty requires an analysis that acknowledges the selection process, or a clearly justified model specified independently of these data.

Measurement Uncertainty

A conventional coefficient interval generally treats the recorded data as the values being analyzed. If a measuring instrument is poorly calibrated, or if a variable is recorded with substantial error, the resulting coefficient can be uncertain for reasons not represented by its usual standard error. Measurement error can also change the coefficient's target or systematically shift the estimate; it is not always just an extra amount of random variation that can be added to the interval.

The following example isolates a simple calibration issue. It assumes a known relationship between the recorded and true predictor scales, and shows how uncertainty in that calibration can be carried through an estimate. This calculation is an illustration under those assumptions, not a general correction for all measurement error.

Worked Example: An Uncertain Predictor Calibration

Suppose a hypothetical instrument records a predictor \(X_{\text{obs}}\) in units related to the true scale \(X_{\text{true}}\) by \(X_{\text{obs}}=cX_{\text{true}}\). A regression using the recorded scale gives a coefficient \(4.00\) outcome units per recorded unit, with standard error \(0.50\). An independent calibration gives \(c=1.10\) recorded units per true unit, with standard error \(0.05\). Assume these estimates are independent, the scale relationship is appropriate, and the calibration uncertainty is reasonably summarized by its standard error.

If \(c\) were known exactly, the coefficient per true unit would be \(c\hat{\beta}_{\text{obs}}\), because

\hat{\beta}_{\text{true}}=1.10(4.00)=4.40\ \text{outcome units per true unit}.

But \(c\) is estimated, not known exactly. For independent estimates, a first-order approximation to the variance of their product includes uncertainty from both estimates:

SE(\hat{\beta}_{\text{true}}) \mathrel{\approx} \sqrt{c^2SE(\hat{\beta}_{\text{obs}})^2+ \hat{\beta}_{\text{obs}}^2SE(c)^2}.

Substituting the values gives

\sqrt{1.10^2(0.50^2)+4.00^2(0.05^2)} = \sqrt{0.3025+0.0400} = 0.5852\ \text{outcome units per true unit}.

Using \(2.024\) as an approximate 95% critical value, the propagated interval is

4.40\mathbin{\pm}2.024(0.5852) \mathrel{\approx} (3.216,\ 5.584)\ \text{outcome units per true unit}.

This approximate interval is wider than one that would treat \(c=1.10\) as exact, because it includes calibration uncertainty. It depends on the stated scale relationship and independence assumption. More complicated measurement error may require a different model or validation data; a conventional interval based only on the regression fit cannot settle that issue.

Target-Population Uncertainty

A regression coefficient is meaningful relative to a population and a model. A conventional interval does not, by itself, establish that a sample represents every population to which someone might wish to apply the result. If the sample's composition differs from the intended target population, the coefficient relevant to that target may differ as well. This is a question of population definition and generalizability, not merely the width of a sampling interval.

Worked Example: A Different Mix in the Target Population

Suppose a hypothetical analysis reports a weighted summary of age slopes across two patient groups. The estimated group-specific slopes are \(2.0\) and \(6.0\) recovery days per year of age. The study sample is 80% from Group 1 and 20% from Group 2. For this example, define the sample summary explicitly as the weighted average of these group slopes, and suppose its conventional standard error is \(0.60\) days per year. Assume the conventional interval conditions are adequate for this summary.

The sample-weighted estimate is

0.80(2.0)+0.20(6.0)=2.80\ \text{days per year}.

Using an approximate critical value of \(2.01\), its conventional 95% interval is

2.80\mathbin{\pm}2.01(0.60) = 2.80\mathbin{\pm}1.206 = (1.594,\ 4.006)\ \text{days per year}.

Now suppose the intended target population is 40% Group 1 and 60% Group 2. Under the same explicitly defined weighted-summary approach, its value using these group slopes is

0.40(2.0)+0.60(6.0)=4.40\ \text{days per year}.

The sample interval was calculated for the sample's 80%/20% mix. It is not automatically an interval for the target population's 40%/60% summary. The difference does not mean that the sample interval was calculated incorrectly; it means that the two summaries answer different population questions. In practice, group slopes, target-population weights, and sampling uncertainty may all need to be estimated. This simplified calculation isolates the effect of using different weights.

A Short Justification: Why the Scope Is Conditional

The usual repeated-sampling argument can be stated simply. Fix a model, the measurements, and a target population, and suppose the assumptions for the conventional interval hold. Then the procedure is designed so that, over repeated samples under that same setup, it covers the specified coefficient at its stated rate. That guarantee concerns repetitions with the setup held fixed.

To see the distinction, denote the setup by \(S\), including the chosen model, measurement process, and target. The conventional coverage statement is about

P(\beta_j(S)\text{ is in the interval}\mid S)\mathrel{\approx}0.95.

If the model is instead chosen anew from the data, or if calibration and target weights are uncertain, the setup is not simply fixed in this way. The conditional coverage statement alone does not tell us the overall coverage after those additional sources of uncertainty are included. This does not prove that every ordinary interval will fail; it identifies what the interval's usual justification does and does not establish.

Common Pitfall and Practical Use

Wrong reasoning: “The 95% coefficient interval is narrow, so model choice, measurement quality, and generalizability must all be unimportant.” The interval's width reflects the standard error calculated under the fitted analysis. It cannot be used to assess uncertainties that were not part of that calculation.

Correction: State the model, measured variables, and target population that the coefficient interval concerns. Then assess model selection, measurement quality, and population match separately. If any of these are uncertain, make that limitation explicit and use an analysis suited to the additional uncertainty where possible. For example, compare defensible prespecified models, collect calibration or validation data, or define and justify the target population and its weights.

This distinction matters because a precisely estimated coefficient for a particular model and sample population can still be a limited answer to a broader question. Precision is valuable, but it does not substitute for a sound model, trustworthy measurements, or a well-defined target.

Check Your Understanding

Answer each question by identifying what the conventional interval does and does not cover.

A coefficient interval is calculated after the analyst tries many predictor combinations and reports the most promising model. What additional uncertainty is not automatically represented?
An instrument's calibration factor is uncertain, but the regression standard error treats recorded predictor values as exact. What source of uncertainty is left out?
Why can an interval calculated for a sample with one group composition fail to answer a question about a target population with a different composition?
In the calibration example, what assumption allows uncertainty in the regression coefficient and calibration estimate to be combined using the displayed product approximation?
State, in one sentence, what is held fixed in the ordinary repeated-sampling interpretation of a conventional coefficient interval.