Coefficient Estimates and Sampling Uncertainty

One Model, Two Kinds of Coefficient

A regression analysis reports numbers calculated from observed data. Those numbers describe the fitted relationship in the sample, but they are also used to learn about a relationship in a larger population. Keeping these two roles separate is essential: the coefficient produced by one sample is an estimate, while the population coefficient it aims to learn about is a parameter.

For example, a fitted slope of \(-1.6\) mmHg per hour is a number calculated from one dataset. It is not automatically the exact population slope. If another sample were collected from the same population and the same analysis repeated, its fitted slope could differ. This variation is present even when the research question, model, measurement procedures, and fitting method remain unchanged.

The word “fixed” does not mean that a population coefficient is known. It means that, for a defined population and model, its value does not change merely because a different sample is drawn. The estimate can change because the observed individuals and their outcomes change. A coefficient also depends on how the model is specified: changing the outcome, predictor, adjustment variables, or functional form can change which population quantity is being described.

The Model and the Source of Uncertainty

For a simple linear regression, write the population model as

Here \(Y_i\) is the outcome for individual \(i\), \(x_i\) is the predictor value, \(\beta_0\) is the population intercept, and \(\beta_1\) is the population slope. The error term \(\varepsilon_i\) represents the part of the outcome not accounted for by the linear mean. The condition \(E(\varepsilon_i\mid X_i=x)=0\) says that the population mean of the error is zero at each predictor value; equivalently, the conditional mean of \(Y\) follows the stated line.

A sample is used to fit a line, producing estimates such as \(\hat{\beta}_0\) and \(\hat{\beta}_1\). The fitting rule can stay exactly the same while the estimated line changes, because the rule is applied to different observed values. The term sampling uncertainty refers to this uncertainty about the population coefficient that remains because the analysis uses a sample rather than every member of the population.

This idea assumes a clear sampling framework. For a repeated-sample comparison to isolate sampling variation, imagine drawing samples from the same target population using the same design, measuring the same variables in the same way, and fitting the same model with the same method. If the target population, selection process, or model changes, differences in estimates may reflect those changes too; they cannot be attributed to sampling variation alone.

Sampling uncertainty is not the same as every other kind of uncertainty. A measurement error, an omitted important predictor, a poorly chosen functional form, or a change in who is included can affect the analysis in additional ways. A coefficient estimate should therefore be understood relative to its population, model, and data-collection process, not as a context-free number.

Why the Estimate Changes Across Samples

In least-squares simple regression, the fitted slope is calculated from the sample's predictor and outcome values. In particular, when the predictor varies, the slope can be written as

The formula makes the sample dependence visible: changing the observed \(x_i\)'s or \(y_i\)'s can change the numerator, denominator, or both. Even when the same population line underlies the data, different samples will generally contain different individuals and different outcome deviations from that line.

A short algebraic identity makes the role of those deviations clearer. Suppose the model is \(y_i=\beta_0+\beta_1x_i+\varepsilon_i\), and the sample contains at least two different predictor values, so the slope denominator is not zero. Substitute this model into the least-squares slope formula. Since \(\sum_{i=1}^{n}(x_i-\bar{x})=0\), constant terms disappear from the numerator; also, \(x_i=\bar{x}+(x_i-\bar{x})\). Therefore,

The last term depends on the particular sample's errors and predictor values. Different samples can produce different values of that term, and hence different slope estimates. This identity explains why a fitted slope need not equal the population slope in a given sample. It does not, by itself, tell us the full pattern or size of the estimates across repeated samples.

Coefficients in an Adjusted Model

The same distinction applies to multiple regression. Consider a population model relating an outcome \(Y\) to an indicator \(T\) for treatment and a predictor \(A\) for age:

Here \(\beta_1\) describes the model-based difference in conditional mean outcome between \(T=1\) and \(T=0\) at the same age, provided the stated model and its assumptions are appropriate. Fitting that model to a sample produces \(\hat{\beta}_1\), an estimate of this population coefficient. Holding the model fixed does not make the estimate identical across samples; each sample supplies different data to the same fitting procedure.

Why the Distinction Matters

A fitted coefficient is useful because it gives a concrete summary of the data and serves as an estimate of a population quantity. But reporting only the fitted value can invite overconfidence: readers may mistake a sample result for an exact population fact. In regression inference, later procedures quantify how much sampling uncertainty accompanies an estimate and use that information to make inferential statements. The starting point is to recognize that a coefficient estimate is not the population coefficient itself.

It is also important not to treat all differences between reported coefficients as sampling variation. Estimates from models with different adjustment variables may target different conditional relationships. Estimates from different populations or measurement procedures may not be directly comparable. Before interpreting a difference, check whether the target population, outcome, predictors, model, and fitting method are genuinely held fixed.

Common Mistakes

• Calling the fitted coefficient the population truth. A slope of \(-1.6\) calculated from a sample is \(\hat{\beta}_1\), an estimate. It is not automatically the exact population value \(\beta_1\).
• Assuming the same method must give the same answer. The method can be fixed while the data change. Because the estimate is calculated from the observed sample, new observations can produce a different coefficient.
• Calling every coefficient difference sampling uncertainty. If the population, model, or measurement process changed, the difference may reflect those changes as well. A repeated-sampling comparison isolates sampling variation only when the setup stays the same.
• Reading an association as a causal effect. A regression coefficient describes a model-based relationship. Its value alone does not establish that changing a predictor causes a change in the outcome.
• Treating a small set of estimates as the population parameter. The mean of a few sample estimates is simply their arithmetic mean. It is not necessarily the parameter they aim to estimate.