Zellner’s G-Prior And Regression Shrinkage

Why use a design-based prior?

In regression, a prior distribution on coefficients should be expressed in units that make sense for the predictors. Assigning the same prior variance to every coefficient can be awkward: changing a predictor from dollars to thousands of dollars changes the numerical coefficient, even though it does not change the underlying relationship. Zellner’s g-prior addresses this by letting the observed design matrix determine the prior covariance. Its single scale parameter, \(g\), then controls how strongly the data update a prior centered at zero.

The resulting Gaussian posterior has a particularly clear form: the posterior mean of each slope is the ordinary least squares estimate multiplied by \(g/(1+g)\). This is shrinkage toward zero, not a claim that the true coefficients are zero. The prior is a modeling choice, and its implications depend on the regression specification and the value assigned to \(g\).

Model and definition

Consider the Gaussian linear regression model \(y=\alpha\mathbf{1}+X\beta+\varepsilon\), where \(y\) is an \(n\)-vector of outcomes, \(\alpha\) is an intercept, \(X\) is an \(n\times p\) matrix of predictors, and \(\beta\) is a \(p\)-vector of slope coefficients. Assume the errors are independent with \(\varepsilon\sim N(0,\sigma^2 I_n)\), where \(\sigma^2>0\). For the standard g-prior below, assume the columns of \(X\) have been centered, \(X\) has full column rank, and the intercept is handled separately. Thus \(X'\mathbf{1}=0\), and \(X'X\) is invertible.

Centering the predictors separates the intercept from the slopes. With a flat prior on \(\alpha\), the likelihood estimate of the intercept is \(\bar{y}\); the slope calculation can be written using the centered response \(y_c=y-\bar{y}\mathbf{1}\). Define the ordinary least squares estimate of the slopes as \(\hat{\beta}=(X'X)^{-1}X'y_c\).

The covariance matrix \((X'X)^{-1}\) has the same design-based form as the covariance matrix of the ordinary least squares estimate under the stated Gaussian model, apart from the factor \(\sigma^2\). The g-prior therefore adjusts the overall scale of coefficient uncertainty with one parameter rather than assigning unrelated variances to the slopes.

Deriving the posterior

First suppose \(\sigma^2\) is known. Conditional on the model and the design, the likelihood for \(\beta\), up to a factor not depending on \(\beta\), is proportional to \(\exp\{- (y_c-X\beta)'(y_c-X\beta)/(2\sigma^2)\}\). The prior contributes a quadratic form in \(\beta\). Multiplying likelihood and prior gives the posterior kernel:

Expand the first quadratic. Its terms involving \(\beta\) are \(\beta'X'X\beta-2\beta'X'y_c\). Adding the prior term gives a quadratic coefficient \((1+1/g)X'X\) and a linear term \(-2\beta'X'y_c\). Completing the square identifies the posterior precision and mean. In particular, the precision matrix is \((1+1/g)X'X/\sigma^2\), and multiplying its inverse by \(X'y_c/\sigma^2\) gives the mean.

Thus, under these assumptions, the full conditional posterior is \(\beta\mid y_c,X,\sigma^2,g\sim N\!\left(\frac{g}{1+g}\hat{\beta},\frac{g}{1+g}\sigma^2(X'X)^{-1}\right)\). If \(\sigma^2\) is unknown, this remains the conditional posterior of \(\beta\) given \(\sigma^2\), provided the same g-prior is used. A joint posterior that also updates \(\sigma^2\) requires specifying a prior for the variance; the conditional formula alone is not that joint posterior.

How g controls shrinkage

The shrinkage factor \(g/(1+g)\) lies strictly between zero and one for \(g>0\). A small \(g\) gives a prior concentrated near zero relative to the likelihood scale and pulls the posterior mean strongly toward zero. A large \(g\) gives a more diffuse prior; as \(g\) grows without bound, the posterior mean approaches \(\hat{\beta}\). The limit \(g\to0\) collapses the prior onto \(\beta=0\), although \(g=0\) is not a nonsingular Gaussian prior.

The same scalar factor multiplies the ordinary least squares covariance matrix to give the conditional posterior covariance. This does not mean that smaller \(g\) simply means “less uncertainty” in every practical sense: the posterior mean changes too, and uncertainty about the outcome or a future observation is a separate calculation. Here the result concerns uncertainty about the regression slopes conditional on the stated model, design, variance, and \(g\).

What the design-based scale means

The prior covariance is not a universal variance for every coefficient. If a column of \(X\) is rescaled, the corresponding coefficient is rescaled in the opposite direction, and the matching diagonal element of \((X'X)^{-1}\) changes accordingly. This makes the prior responsive to the predictor’s units. It also means the coefficient covariance should be read together with the design matrix, rather than as a standalone numerical specification.

One way to see the prior’s implication for the fitted signal is to consider \(X\beta\). Under the g-prior,

Here \(P_X\) is the projection matrix onto the columns of \(X\). Along directions in the predictor space, the prior variance of the fitted signal is scaled by \(g\sigma^2\). This shows why one common \(g\) controls the overall prior signal scale, while the design determines the relevant directions.

Assumptions, limitations, and common mistakes

The standard formula requires a Gaussian linear likelihood, a specified positive error variance for the conditional derivation, \(g>0\), and a full-rank predictor matrix after separating the intercept. If predictors are perfectly collinear, \(X'X\) is singular and the displayed prior cannot be used as written. One must change the model or use a suitable alternative construction rather than silently invert a singular matrix. Centering predictors is a convenient way to make the standard slope prior separate cleanly from an unshrunk intercept.

• Mistake: treating the shrinkage factor as evidence that a coefficient is zero. The posterior mean is pulled toward the prior center because of the prior specification. The result alone neither proves a zero effect nor establishes a causal interpretation.
• Mistake: assigning the prior covariance as \(g\sigma^2 X'X\). The standard g-prior uses the inverse, \((X'X)^{-1}\). A larger amount of predictor variation generally makes the corresponding coefficient more precisely estimated and gives it a smaller prior variance under this construction.
• Mistake: shrinking the intercept automatically. The standard setup here applies the prior to slopes and treats the intercept separately. If the intercept were included in a different prior specification, its treatment and resulting posterior would need to be stated explicitly.
• Mistake: interpreting \(g\) without the design and variance. The same numerical \(g\) acts through \(g\sigma^2(X'X)^{-1}\). Its coefficient-scale implications therefore depend on the units and configuration of the predictors as well as on \(\sigma^2\).