When the Least-Squares Slope Is Not Identifiable

Why Predictor Variation Matters

In simple linear regression, the fitted line is often summarized by an intercept and a slope. The slope describes how the fitted response changes as the predictor changes. But a data set cannot determine that change if the predictor never changes: when every observed predictor value is identical, the observations contain no comparison between different predictor levels.

This is a question about whether the least-squares coefficients are uniquely determined by the observed predictor values. It is not a question about whether the response values vary, nor does it require a probability distribution for the errors. The least-squares calculations below apply to the observed pairs $(x_i,y_i)$, for $i=1,\ldots,n$.

Definition: For a simple least-squares regression with an intercept, the predictor sum of squares is $$ S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2, $$ where $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i$. The least-squares slope is identifiable from the observed data when the minimizing slope has one unique value. This occurs exactly when $S_{xx}>0$, or equivalently, when at least two observed predictor values differ.

The usual model used to describe simple linear regression is $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, where $x_i$ is the predictor value, $\beta_0$ and $\beta_1$ are model coefficients, and $\varepsilon_i$ is an error term. For the least-squares identifiability question, the key issue is the predictor values themselves. Normality or independence assumptions are not needed to determine whether the least-squares slope is unique.

Deriving the Uniqueness Condition

For candidate coefficients $a$ and $b$, least squares minimizes the residual sum of squares $Q(a,b)=\sum_{i=1}^{n}(y_i-a-bx_i)^2$. The intercept $a$ and slope $b$ are chosen together. The following derivation shows why the denominator of the slope formula is $S_{xx}$, and what fails when that denominator is zero.

Minimize over the intercept.
For a fixed slope $b$, differentiating $Q(a,b)$ with respect to $a$ and setting the derivative to zero gives $\sum_{i=1}^{n}(y_i-a-bx_i)=0$. Rearranging yields $a=\bar{y}-b\bar{x}$.

Substitute the best intercept for that slope.
With $a=\bar{y}-b\bar{x}$, each residual is $(y_i-\bar{y})-b(x_i-\bar{x})$. Thus the residual sum of squares as a function of $b$ is $\sum_{i=1}^{n}[(y_i-\bar{y})-b(x_i-\bar{x})]^2$.

Minimize the resulting quadratic.
Differentiating with respect to $b$ and setting the derivative to zero gives $bS_{xx}=S_{xy}$, where $S_{xy}=\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})$. If $S_{xx}>0$, this equation has the unique solution $b=S_{xy}/S_{xx}$.

The condition $S_{xx}=0$ has a precise implication. Each term $(x_i-\bar{x})^2$ is nonnegative, so their sum is zero only if every term is zero. Therefore, $x_i=\bar{x}$ for every observation: all predictors are identical. In that case $S_{xy}=0$ too, and the equation $bS_{xx}=S_{xy}$ reduces to $0=0$. It does not select one slope.

S_{xx}=0 \quad\Longleftrightarrow\quad x_1=x_2=\cdots=x_n=c

When every $x_i=c$, a line with intercept $a$ and slope $b$ predicts $a+bc$ at every observed point. The data can determine the best common prediction, but they cannot separately determine $a$ and $b$. For any chosen slope $b$, setting $a=\bar{y}-bc$ gives the same fitted value $\bar{y}$ for every observation. Consequently, there are infinitely many minimizing coefficient pairs.

Worked Examples

Worked Example: Finding the Least-Squares Lines When All Predictors Equal Four

A hypothetical analyst records response values $7,9,10,$ and $14$ for four observations. The predictor value is $4$ for each observation. Determine whether the least-squares slope is identifiable, give the full set of least-squares coefficient pairs, and calculate the minimized residual sum of squares.

Method. Calculate the predictor sum of squares and response mean, then use the condition for a minimizing intercept and slope. The common mistake to avoid is assigning a slope of zero just because the slope formula gives a zero denominator.

First, $\bar{x}=(4+4+4+4)/4=4$, so $S_{xx}=(4-4)^2+(4-4)^2+(4-4)^2+(4-4)^2=0$. The response mean is $\bar{y}=(7+9+10+14)/4=40/4=10$. Also, $S_{xy}=0$, since every $x_i-\bar{x}$ is zero. Thus the expression $S_{xy}/S_{xx}$ is $0/0$, which is undefined; it is not a slope estimate of zero.

For any selected slope $b$, the intercept that makes the fitted value equal to $\bar{y}$ at $x=4$ is $a=10-4b$. For example, $b=0$ gives $a=10$, while $b=2$ gives $a=2$; both lines predict $10$ whenever $x=4$. In fact, every pair $(a,b)$ satisfying $a+4b=10$ minimizes the residual sum of squares.

The minimized residual sum of squares is $(7-10)^2+(9-10)^2+(10-10)^2+(14-10)^2=9+1+0+16=26$. As a check, both example lines predict $10$ at all four observed predictors and therefore give the same residual sum of squares, $26$. The slope is not identifiable, although the best fitted value at the observed predictor is determined.

Worked Example: Correcting an Undefined Slope Calculation

A hypothetical calibration study has three response measurements, $2,5,$ and $11$, all taken at predictor value $3$. A report calculates $S_{xy}=0$ and $S_{xx}=0$, then concludes that the fitted slope is zero. Assess the conclusion and find the least-squares predictions and residual sum of squares.

Method. Check whether the reported ratio is defined, and then minimize the residual sum of squares using the common-predictor condition. The trap is confusing “the slope is zero” with “the slope cannot be determined.”

The predictor mean is $\bar{x}=3$, so $S_{xx}=\sum_{i=1}^{3}(3-3)^2=0$, and $S_{xy}=0$. The proposed calculation $S_{xy}/S_{xx}=0/0$ has no numerical value. Therefore, it does not justify a zero slope.

The response mean is $\bar{y}=(2+5+11)/3=18/3=6$. For any slope $b$, choose $a=6-3b$; then $a+3b=6$, so the fitted prediction at every observed point is $6$. The response residuals are $2-6=-4$, $5-6=-1$, and $11-6=5$. The minimized residual sum of squares is $(-4)^2+(-1)^2+5^2=16+1+25=42$.

As a check, the line with $b=0,a=6$ and the line with $b=1,a=3$ both predict $6$ at $x=3$, so both have residual sum of squares $42$. A zero slope is one possible minimizing choice, not the uniquely determined slope.

Worked Example: A Small but Nonzero Predictor Spread

A hypothetical instrument is tested at predictor settings $100.0,100.1,$ and $100.2$, producing responses $20.0,20.5,$ and $21.5$, respectively. Calculate the least-squares slope and intercept, and determine whether the slope is identifiable.

Method. Center the predictor and response values and calculate $S_{xx}$ and $S_{xy}$. The mistake to avoid is treating a small predictor spread as if it were exactly zero.

The means are $\bar{x}=(100.0+100.1+100.2)/3=100.1$ and $\bar{y}=(20.0+20.5+21.5)/3=62.0/3$. To keep the arithmetic transparent, the centered response values are $20.0-62/3=-2/3$, $20.5-62/3=-1/6$, and $21.5-62/3=7/6$. The centered predictor values are $-0.1,0,0.1$.

Therefore, $S_{xx}=(-0.1)^2+0^2+(0.1)^2=0.02$, and $S_{xy}=(-0.1)(-2/3)+0(-1/6)+(0.1)(7/6)=1/15+7/60=11/60$. The slope is $b=(11/60)/0.02=(11/60)/(1/50)=55/6$, approximately $9.1667$. The intercept is $a=\bar{y}-b\bar{x}=62/3-(55/6)(100.1)=-1820.25$.

The slope is identifiable because $S_{xx}=0.02>0$, even though the predictor values are close together. As a check, the fitted predictions at $100.0,100.1,$ and $100.2$ are approximately $9.1667,10.0833,$ and $11.0000$. Subtracting the intercept and multiplying by $x$ gives the same predictions, subject to displayed rounding. The small spread is not the same mathematical condition as zero spread.

What Nonidentifiability Does—and Does Not—Mean

When $S_{xx}=0$, every observation is at the same predictor value. Replicating that predictor value more times may improve the precision with which the response mean at that single point is described, but it does not create a comparison across predictor levels. The data still cannot determine how a fitted line changes away from that point.

The fitted value at the observed predictor is nevertheless determined: it is $\bar{y}$. The fitted intercept and slope individually are not determined, because many combinations produce that same fitted value. This distinction matters when interpreting regression coefficients: a numerical pair output by software in a rank-deficient case may reflect a software convention, but it is not a unique least-squares slope established by the data.

Conversely, a constant response does not by itself make the slope unidentifiable. If the predictors vary, then $S_{xx}>0$; if all responses equal the same constant, then $S_{xy}=0$, and the unique least-squares slope is $0$. The question is whether the predictors vary, not whether the responses vary.

Key takeaway: The least-squares slope with an intercept is $S_{xy}/S_{xx}$ only when $S_{xx}>0$. If $S_{xx}=0$, all predictors are identical, the ratio is undefined, and infinitely many intercept–slope pairs produce the same least-squares fit at the observed predictor value.

Common Exam Traps

Calling $0/0$ a zero slope. A zero numerator and zero denominator make the ratio undefined. The correct conclusion is that the slope is not uniquely determined.
Assuming response variation fixes the problem. Different response values do not help identify a slope if every predictor value is the same.
Assuming a constant response prevents a slope from being identified. With predictor variation, $S_{xx}>0$; a constant response gives $S_{xy}=0$, so the unique slope is zero.
Treating near-zero variation as zero variation. Any positive $S_{xx}$ gives a unique least-squares slope, though a very small $S_{xx}$ can make the calculation sensitive to the observed data.

Check Your Understanding

For each problem, distinguish a uniquely determined slope from an undefined slope calculation. Use the least-squares model with an intercept.

Three observations have predictor value $5$ and responses $1,4,7$. Find $S_{xx}$, the mean response, and all minimizing intercept–slope pairs. (Answer: $S_{xx}=0$, $\bar{y}=4$, and every pair satisfying $a+5b=4$ minimizes.)
Predictor values are $1,1,4,4$, and responses are $3,5,9,11$. Find $S_{xx}$, $S_{xy}$, the slope, and the intercept. (Answer: $S_{xx}=9$, $S_{xy}=18$, $b=2$, $a=2$.)
Four observations have predictor value $9$ and responses $2,4,8,10$. Find the minimizing prediction and residual sum of squares, and state whether the slope is identifiable. (Answer: prediction $6$ at $x=9$, residual sum of squares $40$, and the slope is not identifiable; $a+9b=6$.)
Predictor values are $0,0,0,2$, and responses are $1,3,5,9$. Calculate $S_{xx}$, $S_{xy}$, and the fitted slope. (Answer: $S_{xx}=3$, $S_{xy}=9$, and $b=3$, so the slope is identifiable.)
The predictor values are $2,4,6$, and all three responses equal $5$. Is the slope identifiable, and what is its value? (Answer: Yes; $S_{xx}=8>0$, $S_{xy}=0$, and $b=0$.)

What You'll Learn