Fixed Predictors And Random-Predictor Inference

Two Ways to View the Predictor

In regression, the observed predictor values may be treated as fixed features of the study design, or as random observations from a population. This distinction affects what is being conditioned on, but it does not by itself decide whether a regression t or F procedure is valid. The key question is whether the response model and error assumptions hold, either for the fixed design or conditional on the observed predictors.

Consider simple linear regression with response \(Y_i\) and predictor \(x_i\), for observations \(i=1,\ldots,n\). The conventional model writes the response as a linear mean plus an error. In a fixed-predictor analysis, the \(x_i\) values are treated as known constants. In a random-predictor analysis, the \(X_i\) values are random, but inference can still be made conditional on the values that were observed.

“Fixed” describes the probability model used for inference; it does not mean that the predictor was necessarily controlled by an experimenter. Conversely, a predictor being random does not automatically make conventional regression inference invalid. The conditions on the response errors, and the population quantity being estimated, are what matter.

Conditions for Conventional Regression Inference

For fixed predictor values, the simple linear regression model is

The standard t and F procedures require assumptions about the errors. For the usual finite-sample, exact procedures, a standard set of conditions is that, conditional on the predictors, the errors are independent and normally distributed with mean zero and a common variance \(\sigma^2\). Thus \(E[\varepsilon_i\mid x_1,\ldots,x_n]=0\), \(\operatorname{Var}(\varepsilon_i\mid x_1,\ldots,x_n)=\sigma^2\), and the errors are conditionally independent. The predictor values must also contain variation: \(S_{xx}=\sum_i(x_i-\bar{x})^2>0\). There must be more observations than fitted parameters to estimate the residual variance.

The zero conditional mean says that the fitted line describes the response mean at each predictor value. Equal conditional variances give the usual pooled residual variance its stated role. Independence rules out unaccounted dependence among observations. Conditional normality supplies exact finite-sample t and F distributions; without it, those exact reference distributions are not generally guaranteed. Standard large-sample approximations require their own conditions and are a separate issue.

With an intercept and slope, let \(\hat{\beta}_1\) be the least-squares slope and let \(\mathrm{SSE}=\sum_i(Y_i-\hat{Y}_i)^2\). The conventional estimated error variance is \(s^2=\mathrm{SSE}/(n-2)\). Under the conditions above, the slope standard error is \(s/\sqrt{S_{xx}}\), and a test of \(H_0:\beta_1=\beta_{1,0}\) uses

For multiple regression, the same principle applies conditional on the design matrix: the design must have full column rank, and the errors must satisfy the corresponding zero-mean, equal-variance, independent conditions. Exact classical F tests for \(q\) linear restrictions use the reduction in residual sum of squares from imposing those restrictions, divided by \(q\), relative to the unrestricted residual variance. Their usual exact reference distribution also relies on conditional normality.

Why Conditioning on the Predictors Works

The role of conditioning can be seen directly from the least-squares slope. For fixed \(x_i\), it can be written as

The first equality is the usual slope formula. For the second, substitute \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\). The intercept term disappears because \(\sum_i(x_i-\bar{x})=0\), and the slope term contributes \(\beta_1 S_{xx}/S_{xx}=\beta_1\). Conditional on the predictors, the errors have mean zero, so the conditional mean of the slope estimate is \(\beta_1\). If the errors are conditionally independent with common variance \(\sigma^2\), then

This explains why the usual slope standard error depends on the observed predictor spread. If predictors were sampled randomly, the same calculation applies after conditioning on their realized values, provided the error assumptions hold conditional on those values. In the usual normal-error model, the conditional t and F reference distributions do not depend on the particular predictor values beyond the required rank and degrees of freedom. Randomness in the predictors therefore does not, on its own, invalidate the procedures.

Random Predictors: Condition on What Was Observed

Suppose observations are sampled as pairs \((X_i,Y_i)\). A useful random-predictor model states that the conditional response mean is linear:

For ordinary regression t and F inference, the error \(\varepsilon_i=Y_i-(\beta_0+\beta_1X_i)\) must satisfy the relevant assumptions conditional on the predictor values. In particular, the conditional mean must be zero, the conditional variance must be constant for conventional standard errors, and the errors must be independent across observations. For exact finite-sample t and F procedures, conditional normality is also required. Random sampling of the predictors is compatible with these conditions; a nonlinear conditional mean, dependence between observations, or changing error variance may not be.

The regression coefficient is also tied to the target. When the conditional mean is truly linear, \(\beta_1\) describes its change per unit of \(X\). If the conditional mean is not linear, a fitted line may instead summarize a linear projection over the sampled predictor distribution. That projection can depend on the distribution of \(X\); it need not be a universal slope describing the conditional mean at every \(x\).

When the Conventional Procedures Do Not Apply

If the conditional mean is wrong, or if errors have nonzero conditional means, the slope may no longer estimate the coefficient claimed by the model. If conditional variances differ, the pooled estimate \(s^2\) does not generally provide the conventional standard error. If observations are dependent, the usual variance calculation can also fail. In these cases, the familiar t or F formula does not become valid merely because the predictors are treated as fixed.

A particularly important problem is dependence between a predictor and the error. In a population, the slope of the best linear projection of \(Y\) on \(X\), when \(\operatorname{Var}(X)>0\), is \(\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\). Under \(Y=\beta_0+\beta_1X+\varepsilon\), this is

Thus a nonzero \(\operatorname{Cov}(X,\varepsilon)\) shifts the population projection slope away from the stated \(\beta_1\). Conditioning on observed predictors does not remove this problem: the required zero conditional mean has failed.

Common Exam Traps

• Equating random predictors with invalid inference. If the response assumptions hold conditional on the observed predictors, the usual conditional regression calculations remain available.
• Equating fixed predictors with automatically valid inference. A fixed design does not repair heteroskedasticity, dependence, a misspecified conditional mean, or nonzero error means.
• Ignoring the target parameter. A fitted linear projection and a coefficient in a correctly specified conditional mean are not necessarily the same scientific quantity.
• Claiming exact t or F results without the needed distributional assumption. The conventional exact finite-sample reference distributions require conditional normal errors, in addition to the other model conditions.