Contrast Matrices and Identifiability

Why Contrast Matrices Need Two Conditions

A factor with \(K\) levels can be described by \(K\) level effects, but including all \(K\) effects alongside an intercept introduces redundancy: adding a constant to every level effect and subtracting it from the intercept leaves every fitted linear predictor unchanged. Contrast coding removes this redundancy by representing level effects in a \(K-1\)-dimensional space. The key is to distinguish two conditions: the columns must sum to zero, and they must be linearly independent.

This tutorial focuses on how those conditions encode factor effects and what they do—and do not—establish about identifiability. Identifiability is ultimately a property of the design actually used in the model, not just of a coding matrix considered in isolation.

The zero-sum condition says that every represented level-effect vector satisfies \(\mathbf{1}_K^T\alpha=0\). The rank condition says that the columns represent \(K-1\) independent directions, rather than repeating or omitting some direction. Together, these conditions make the columns a basis for the space of zero-sum level effects.

What the Rank and Zero-Sum Conditions Establish

The space of zero-sum vectors in \(\mathbb{R}^K\) has dimension \(K-1\): its elements satisfy one linear constraint, and the first \(K-1\) coordinates can be chosen freely while the last is determined by their sum. A full-rank contrast matrix has \(K-1\) independent columns in this space, so it spans the whole space.

Proof. Since \(\mathbf{1}_K^TC=0\), every vector \(C\gamma\) has sum zero. The zero-sum space has dimension \(K-1\), and the columns of \(C\) are \(K-1\) independent vectors in that space. They therefore form a basis for it, so every zero-sum \(\alpha\) can be written as \(C\gamma\). For uniqueness, suppose \(C\gamma=C\tilde{\gamma}\). Then \(C(\gamma-\tilde{\gamma})=0\). Full column rank implies that the null space of \(C\) contains only the zero vector, so \(\gamma=\tilde{\gamma}\). \(\square\)

In a factor-only model, the linear predictor for level \(k\) is \(\eta_k=\beta_0+\alpha_k\). With \(\alpha=C\gamma\), the \(K\) level predictors are represented as \(\boldsymbol{\eta}=\beta_0\mathbf{1}_K+C\gamma\). Because the columns of \(C\) sum to zero, their average is \(\beta_0\). Thus the intercept is the unweighted average of the level-specific linear predictors, while \(\gamma\) gives coordinates for their deviations from that average. This is an average on the linear-predictor scale; for a nonidentity link it is not generally the average of the response means.

From a Contrast Basis to Identifiable Coefficients

A full-rank contrast matrix guarantees that the coefficients uniquely encode the level effects. It does not, by itself, guarantee that a fitted model can distinguish all coefficients from the observations. For that, the design matrix must also have adequate rank. For a GLM, the linear predictor has the form \(\boldsymbol{\eta}=X\beta\), where \(X\) is the observed design matrix. If two coefficient vectors give the same \(\boldsymbol{\eta}\), the model cannot distinguish them through the linear predictor.

Proof. If \(X\) has full column rank, its null space contains only zero. If \(X\beta=X\tilde{\beta}\), then \(X(\beta-\tilde{\beta})=0\), hence \(\beta-\tilde{\beta}=0\), so \(\beta=\tilde{\beta}\). Conversely, if \(X\) does not have full column rank, there is a nonzero vector \(v\) with \(Xv=0\). For any \(\beta\), \(X(\beta+v)=X\beta+Xv=X\beta\); therefore distinct coefficient vectors give the same linear predictor. For the GLM statement, equality of response distributions at the design points implies equality of their means under the stated distribution-identifiability condition. A one-to-one link then gives equality of the corresponding linear predictors. Full column rank implies equality of the coefficient vectors by the first part. \(\square\)

For a factor represented by an intercept and a contrast matrix, the design matrix has a row \((1,c_k^T)\) for an observation at level \(k\), where \(c_k^T\) is row \(k\) of \(C\). If all levels are observed and \(C\) is a full contrast basis, these factor columns have full rank. Indeed, if \(a_0+c_k^Tb=0\) at every level, summing those equations over \(k\) gives \(Ka_0=0\), because the columns of \(C\) sum to zero. Thus \(a_0=0\). The equations then say \(Cb=0\), and full column rank gives \(b=0\). This argument concerns the factor-only design; when other predictors are included, the rank of the complete design matrix must be checked.

What to Check in an Applied GLM

A useful check separates the coding question from the observed-design question. First, verify that the proposed contrast columns sum to zero and have rank \(K-1\); this confirms that they form a basis for zero-sum level effects. Then form the complete model matrix for the observations and check its column rank. A factor can be coded with a valid contrast basis and still fail to have identifiable coefficients in a particular model—for example, if the observed design makes a factor column redundant with other predictors.

Common mistake: “The columns sum to zero, so the factor coefficients must be identifiable.” The conclusion does not follow. Zero sums constrain the effects to have average zero, but the columns might be linearly dependent, as in the preceding example. Even a full-rank contrast basis cannot prevent a loss of rank caused by the observed design or other model columns. The correction is to check both the contrast matrix and the complete design matrix, and to be clear about which parameters are being claimed to be identifiable.