Two-Stage Least Squares As Projection

Why View 2SLS as a Projection?

An endogenous regressor can contain variation related to the structural error. Instrumental variables estimation uses variation in the regressors that is supported by the instruments instead. Two-stage least squares (2SLS) makes this idea explicit: it first projects the regressor columns onto the space spanned by the instruments, then uses those projected regressors to estimate the structural coefficients.

The projection is not a new set of observations or a change to the structural model. It is a matrix operation that separates the part of each regressor lying in the instrument space from the part orthogonal to that space. The algebra clarifies exactly which variation enters the coefficient calculation.

Definition: Let $y$ be an $n \times 1$ outcome vector, $X$ an $n \times k$ matrix of structural regressors, and $Z$ an $n \times l$ matrix of instruments. Assume $Z$ has full column rank. The projection matrix onto the column space of $Z$ is $$ P_Z = Z(Z'Z)^{-1}Z'. $$ The projected regressors are $\hat{X}=P_ZX$. Each column of $\hat{X}$ is the least-squares projection of the corresponding column of $X$ onto the space spanned by the columns of $Z$.

The instruments in $Z$ include any excluded instruments and any included exogenous regressors used as instruments. A column of ones is included when the structural model has an intercept. This setup accommodates the instrument conditions with exogenous controls without needing a separate projection formula.

For the usual structural model $y=X\beta+u$, the instrument assumptions concern the relationship between the instruments and the structural error, and the relevance conditions require enough instrument-supported variation to identify the coefficients. The projection algebra below defines the estimator; it does not by itself establish that the instruments are valid or that a causal interpretation is justified.

Properties of the Projection Matrix

The matrix $P_Z$ is symmetric and idempotent: $P_Z'=P_Z$ and $P_Z^2=P_Z$. These properties express that projection is an orthogonal projection. The projected vector lies in the instrument space, while the difference between the original and projected vectors is orthogonal to that space.

P_Z'=P_Z,\qquad P_Z^2 =Z(Z'Z)^{-1}Z'Z(Z'Z)^{-1}Z' =Z(Z'Z)^{-1}Z' =P_Z.

For any vector $v$, the projected vector $P_Zv$ is a linear combination of the columns of $Z$, because it is $Z$ multiplied by a vector of coefficients. Also,

Z'(v-P_Zv) =Z'v-Z'Z(Z'Z)^{-1}Z'v =0.

Thus $v-P_Zv$ is orthogonal to every column of $Z$. Together, these facts characterize $P_Zv$ as the orthogonal projection of $v$ onto the instrument space. For the regressor matrix $X$, this operation retains the components of its columns that lie in that space.

Deriving the 2SLS Formula

In the first stage, project every structural regressor onto the instruments, producing $\hat{X}=P_ZX$. In the second stage, regress $y$ on $\hat{X}$. The ordinary least-squares coefficient formula for this regression is

\hat{\beta}_{2SLS} =(\hat{X}'\hat{X})^{-1}\hat{X}'y.

Substitute $\hat{X}=P_ZX$. Since $P_Z$ is symmetric and idempotent, $\hat{X}'\hat{X}=X'P_Z'P_ZX=X'P_ZX$, and $\hat{X}'y=X'P_Zy$. Therefore,

\hat{\beta}_{2SLS} =(X'P_ZX)^{-1}X'P_Zy.

This expression is defined when $X'P_ZX$ is invertible, equivalently when the projected regressors have full column rank. The sample rank condition matters: if projection leaves some structural regressors linearly dependent, the coefficients cannot all be estimated by this formula.

There is another useful way to read the same calculation. Because $P_Z$ is symmetric and idempotent, minimizing the squared length of the projected residual gives the 2SLS criterion:

\hat{\beta}_{2SLS} =\underset{b}{\operatorname{argmin}}\, (y-Xb)'P_Z(y-Xb) =\underset{b}{\operatorname{argmin}}\, \|P_Z(y-Xb)\|^2.

Differentiating the quadratic with respect to $b$ gives the first-order condition $X'P_Z(y-Xb)=0$. Solving it yields the matrix formula above when $X'P_ZX$ is invertible. 2SLS therefore fits the part of the structural residual that can be seen in the instrument space. It does not choose coefficients by minimizing the ordinary squared residual length $(y-Xb)'(y-Xb)$.

Worked Example: Projection With a Binary Instrument

Consider three hypothetical observations from the model $y_i=\beta_0+\beta_1x_i+u_i$. Suppose the instrument is $z_i$, and use an intercept in both the structural regressors and the instrument matrix.

Observation	$z$	$x$	$y$
1	0	1	2
2	0	3	5
3	1	4	8

The instrument matrix has an intercept and $z$. Its column space contains vectors that are constant within each of the two instrument groups. Projecting $x$ onto this space gives the within-group means: $2$ for the first two observations and $4$ for the third. Thus $\hat{x}=(2,2,4)'$. The second stage regresses $y$ on an intercept and $\hat{x}$.

The mean of $\hat{x}$ is $8/3$, and the mean of $y$ is $5$. The second-stage slope is

\hat{\beta}_1 =\frac{\sum_i(\hat{x}_i-\bar{\hat{x}})(y_i-\bar{y})} {\sum_i(\hat{x}_i-\bar{\hat{x}})^2} =\frac{(-2/3)(-3)+(-2/3)(0)+(4/3)(3)} {(-2/3)^2+(-2/3)^2+(4/3)^2} =\frac{6}{8/3} =2.25.

The intercept is $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{\hat{x}}=5-2.25(8/3)=-1$. The estimated structural line is therefore $\hat{y}=-1+2.25x$. The numerical coefficient is based on the projected regressor, not on the original values $1,3,4$.

Worked Example: Projection With More Instruments Than Regressors

Now consider four hypothetical observations, one structural regressor $x$ plus an intercept, and three instruments: an intercept and two contrasts. Let $z_1=(1,-1,0,0)'$ and $z_2=(0,0,1,-1)'$. The instrument matrix $Z$ has columns $\mathbf{1},z_1,z_2$. These columns are mutually orthogonal, and projection onto their span replaces the first pair of entries by their pair mean and the last pair by their pair mean.

Observation	$x$	$y$
1	1	2
2	3	4
3	2	5
4	6	9

The projection matrix and projected regressor are

P_Z= \begin{pmatrix} 1/2&1/2&0&0\\ 1/2&1/2&0&0\\ 0&0&1/2&1/2\\ 0&0&1/2&1/2 \end{pmatrix}, \qquad \hat{x}=P_Zx= \begin{pmatrix}2\\2\\4\\4\end{pmatrix}.

Regressing $y$ on an intercept and $\hat{x}$, the means are $\bar{\hat{x}}=3$ and $\bar{y}=5$. The slope is

\hat{\beta}_1 =\frac{(-1)(-3)+(-1)(-1)+(1)(0)+(1)(4)} {(-1)^2+(-1)^2+(1)^2+(1)^2} =\frac{8}{4}=2, \qquad \hat{\beta}_0=5-2(3)=-1.

The projected regressor has rank two when combined with the intercept, so both coefficients are estimable. Here there are three instrument columns for two structural regressors. The projection calculation remains the same: it forms fitted regressors from the full instrument space before estimating the structural coefficients.

What Projection Changes: A Comparison With OLS

Projection can change the coefficient because it removes the component of each regressor orthogonal to the instruments. The following comparison uses the same hypothetical observations as the preceding example. It shows the algebraic distinction between regressing on $x$ itself and regressing on its projection.

Worked Example: Original Regressor Versus Projected Regressor

For the data $x=(1,3,2,6)'$ and $y=(2,4,5,9)'$, ordinary least squares with an intercept gives $\bar{x}=3$ and $\bar{y}=5$. Its slope is

\hat{\beta}_{OLS} =\frac{(-2)(-3)+(0)(-1)+(-1)(0)+(3)(4)} {(-2)^2+0^2+(-1)^2+3^2} =\frac{18}{14} \approx 1.286.

By contrast, the instrument-space projection in the preceding example gives $\hat{x}=(2,2,4,4)'$ and the 2SLS slope is $2$. The two estimates differ because the original regressor includes variation not used by 2SLS's projected-regressor calculation. Under the structural model and valid, relevant instrument conditions, 2SLS targets the structural coefficient using instrument-supported variation; the numerical difference alone does not establish that either set of assumptions holds.

Reading the Two Stages Correctly

The phrase “two stages” can invite a misleading description. In the first stage, the regressors are projected onto the instruments. In the second stage, the outcome is regressed on those projected regressors to obtain the coefficient estimate. The projection matrix also implies the compact expression $(X'P_ZX)^{-1}X'P_Zy$, so the first stage is not merely a verbal instruction to substitute arbitrary fitted values.

The projected columns are fitted values from least-squares regressions on $Z$, but their role here is to construct the 2SLS coefficient. The projection does not make the structural error disappear, and the formula does not validate the instrument assumptions. Nor does the fact that the second-stage coefficient has an OLS form mean that the overall estimator is ordinary least squares on the original regressors: its regressor matrix is $P_ZX$, not $X$.

Common Mistakes

Using $X$ where the formula requires $P_ZX$. This gives the ordinary least-squares coefficient $(X'X)^{-1}X'y$, not 2SLS.
Treating projection as a transformation of the outcome alone. The defining second-stage regression uses projected structural regressors. The equivalent criterion also projects the residual, as in $\|P_Z(y-Xb)\|^2$.
Assuming a projection guarantees valid instruments. The matrix $P_Z$ can be calculated for a chosen $Z$, but its existence does not establish instrument relevance or the required relationship between $Z$ and the structural error.
Ignoring rank. If the projected regressors do not have full column rank, $X'P_ZX$ is singular and the displayed inverse does not exist.

Key takeaway: 2SLS projects the columns of $X$ onto the column space of $Z$, then estimates the structural coefficients using those projected regressors. Symmetry and idempotence of $P_Z$ yield the formula $\hat{\beta}_{2SLS}=(X'P_ZX)^{-1}X'P_Zy$, provided the projected regressors have full rank.

Check Your Understanding

Use the projection interpretation to answer each question.

In one sentence, what does $P_ZX$ represent?
Why is $P_Z^2=P_Z$ a useful property when deriving the 2SLS formula?
What rank condition is needed for $(X'P_ZX)^{-1}$ to exist?
Why is an OLS regression of $y$ on the original $X$ not generally the 2SLS second stage?
Does the ability to compute $P_Z$ establish that the instruments are valid? Explain.

Observation	\(z\)	\(x\)	\(y\)
1	0	1	2
2	0	3	5
3	1	4	8