Diagnosing Rank-Deficient Information Matrices

Why Diagnose the Information Matrix?

A GLM fitting algorithm may need to solve a system involving its information matrix at every iteration. If that matrix is singular or nearly singular, the problem is not simply that a solver needs more precision: some coefficient combination may be unidentifiable from the design, or the current likelihood may provide very little information in a direction. These situations call for different diagnoses. An exact alias is a property of the design columns; weak information can also arise because the current working weights make some observations contribute very little.

The useful object to inspect is not only the information matrix itself, but also the weighted design from which it is formed. Its singular values reveal the sizes of the directions represented in the data, while its right singular vectors identify the corresponding coefficient combinations.

Information and the Weighted Design

Let \(X\) be the \(n \times p\) GLM design matrix, whose \(i\)th row is \(x_i^\mathsf{T}\). At a specified coefficient value, let \(W=\operatorname{diag}(w_1,\ldots,w_n)\) be the diagonal matrix of finite, nonnegative working weights. In the usual independent-observation GLM working-information calculation, the information matrix, apart from any common positive dispersion factor, is

For example, without additional prior weights, the usual Fisher-scoring working weight for observation \(i\) is \(w_i=(d\mu_i/d\eta_i)^2/\operatorname{Var}(Y_i)\), evaluated at the current fitted mean. A common positive factor such as \(1/\phi\) changes the scale of \(I\), but not its rank. In the diagnostics below, the weighted design means \(A=W^{1/2}X\); rows with zero weights contribute no information at that iteration.

If \(A=U\Sigma V^\mathsf{T}\) is a singular-value decomposition (SVD), its singular values are \(\sigma_1\geq\cdots\geq\sigma_p\geq0\), allowing zeros when \(A\) has fewer than \(p\) independent columns. The corresponding information eigenvalues are \(\sigma_j^2\). A right singular vector \(v_j\) describes a coefficient direction: a small \(\sigma_j\) means that changing coefficients along \(v_j\) changes the weighted linear predictors only slightly.

Proof. For any \(b\in\mathbb{R}^p\),

Thus \(b\) is in the nullspace of \(I\) exactly when \(W^{1/2}Xb=0\). If every weight is positive, \(W^{1/2}\) is invertible, so \(W^{1/2}Xb=0\) exactly when \(Xb=0\). The nullspaces, and hence the ranks, agree. If weights may be zero, \(Xb=0\) still implies \(W^{1/2}Xb=0\), but the reverse need not hold: a nonzero value of \(Xb\) confined to zero-weight rows is erased by multiplication by \(W^{1/2}\). This proves both claims. \(\square\)

Using SVD and Pivoted QR

For a numerical diagnosis, compute the singular values of \(A\) when feasible. A common scale-relative rule declares \(\sigma_j\) numerically negligible when it is no larger than \(\tau\sigma_1\), where \(\tau\) is a stated tolerance. A machine-precision rule may take \(\tau\) on the order of \(\max(n,p)\epsilon\), with \(\epsilon\) machine precision; applications may need a larger tolerance to reflect measurement precision or the accuracy required of the fit. There is no universal cutoff that turns a small singular value into a proof of exact rank deficiency.

A pivoted QR factorization is often a less expensive first diagnostic, especially for a tall design matrix. It permutes columns and computes \(AP=QR\), where \(Q\) has orthonormal columns and \(R\) is upper trapezoidal. Very small diagonal entries near the end of \(R\), relative to the leading scale, suggest dependent or nearly dependent columns. Pivoted QR is useful for locating candidate columns, but its diagonal entries are not a substitute for an SVD when the rank decision is delicate.

A right singular vector associated with a zero singular value gives an exact relation \(Xb=0\) when the design itself is rank deficient. In a fitted model, this means that adding any multiple of \(b\) to the coefficient vector leaves every linear predictor unchanged. Pivoted QR can point to which columns participate, but the specific column labelled “dependent” can vary with column order, scaling, and pivot choices. The underlying dependent subspace is the more fundamental diagnosis.

Worked Diagnostics

What the Diagnosis Does—and Does Not—Say

A rank diagnostic describes the information matrix at the weights and coefficient values used to form it. In logistic regression, for example, fitted probabilities near zero or one make \(p_i(1-p_i)\) small. If such observations are the ones needed to distinguish a coefficient direction, that direction can become weakly informed. Near-separation is one possible model-level reason for this pattern, but small weights alone do not prove near-separation: inspect the outcomes, predictors, and coefficient path as well. The reverse caution also matters: an ill-conditioned unweighted design can remain ill-conditioned after weighting, so weak singular values in \(A\) cannot automatically be blamed on \(W\).

For exact aliases, the remedy is to reconsider the parameterization or remove a redundant column, guided by the scientific meaning of the predictors. For a small but nonzero singular value, first check scaling and near-dependence in \(X\), then assess whether the current weights further weaken the direction. A numerical rank cutoff can flag a concern; it does not decide the scientific acceptability of a model or convert an approximate dependence into an exact one.

Common Mistake

Wrong reasoning: “The design matrix has full column rank, so a tiny singular value of \(W^{1/2}X\) must have been caused by the weights.” Full rank only rules out exact linear dependence. The unweighted design may already be poorly conditioned, even when all weights equal one.

Correction: Compare the singular values and conditioning of \(X\) with those of \(W^{1/2}X\), using consistent column scaling. Attribute additional weakness to the weights only when that comparison supports it. If the unweighted design is already weak in the same direction, investigate its scaling or near-dependence before drawing conclusions about the weights.