High-Dimensional Regression Regimes and Sparsity

Why Predictor Dimension Changes the Regression Problem

In a regression with a modest number of predictors, the sample size usually grows while the number of coefficients stays fixed. Contemporary economic data can present a different regime: the number of candidate predictors may grow with the sample, approach its size, or exceed it. Examples include many possible controls, interactions, lags, or text-derived variables. The central question is then not simply how to estimate a long list of coefficients. It is whether the regression has useful structure, and whether the goal is prediction or inference about particular coefficients.

Let \(Y_i\) be an outcome and let \(X_i=(X_{i1},\ldots,X_{ip_n})'\) be a vector of \(p_n\) predictors for observation \(i\), where \(i=1,\ldots,n\). Write the linear part of the model as \(X_i'\beta^0\), with coefficient vector \(\beta^0\). An intercept may be included separately or removed by centering; here the displayed predictors do not include it. The number of predictors is allowed to depend on the sample size.

When \(p_n\) is fixed and \(n\) grows, the number of unknown coefficients does not keep pace with the information in the sample. If \(p_n\) is comparable to \(n\), estimation and approximation errors can behave differently from fixed-dimensional theory. If \(p_n>n\), an unrestricted coefficient vector generally cannot be uniquely determined from the observed regression equations. This last fact is about the design matrix, not evidence that every high-dimensional problem is unsolvable.

Exact and Approximate Sparsity

High-dimensional methods often rely on a concise description of the relationship between predictors and the outcome. Sparsity is one such description: only a relatively small number of predictors, or a small number of terms in a useful approximation, carry substantial information. The relevant comparison is between the number of important terms and the total number of candidates.

A useful exact-sparsity model might have \(p_n=2{,}000\) candidate predictors but only \(s_0=8\) nonzero coefficients. The coefficient vector is high-dimensional in its list of possible entries, yet its true support is small. Exact sparsity is an idealization: in an economic application, many small effects may be more plausible than a large set of effects that are precisely zero.

This definition makes an important point: approximate sparsity is not just the observation that many coefficients look small. It concerns the quality of a sparse approximation to the conditional mean under the distribution of predictors. Two coefficient vectors with the same small entries can have different prediction consequences if the predictors have different scales or dependence patterns. A small remainder for prediction also need not be adequate for inference on a particular coefficient.

Why Coefficients Can Be Non-Unique

The basic linear algebra explains the issue when \(p_n>n\). The \(n\)-row design matrix \(X\) has rank at most \(n\). By the rank-nullity theorem, its null space has dimension \(p_n-\operatorname{rank}(X)\), which is at least \(p_n-n>0\). Therefore, there is a nonzero vector \(v\) such that \(Xv=0\). If \(\beta\) fits the data, then

So the coefficient vector is not unique among exact fits. More generally, least-squares coefficient vectors can differ by null-space directions while producing the same fitted vector. The fitted vector itself is unique: it is the projection of \(y\) onto the column space of \(X\). This distinction matters. A unique fit to the observed sample does not imply unique coefficients, nor does it guarantee accurate predictions away from the observed design.

Even when \(p_n\le n\), uniqueness is not automatic: it requires the columns of \(X\) to be linearly independent. Near-dependence can also make coefficient estimates unstable, even if exact algebraic uniqueness holds. Dimension and sparsity therefore describe part of the problem, but design properties and identification remain important.

Approximation Error and Prediction

Prediction asks how close a fitted conditional mean is to the outcome’s conditional mean at relevant predictor values. One way to describe the target is the population mean squared prediction error, \(E[(X_{\mathrm{new}}'\hat\beta-E[Y_{\mathrm{new}}\mid X_{\mathrm{new}}])^2]\), for a new observation drawn from the target population. It is not the same as asking whether each coefficient has been estimated precisely.

Approximate sparsity creates a trade-off. A shorter representation is easier to estimate, but may omit relevant contributions and leave a larger approximation remainder. A longer representation can reduce that remainder while requiring more information to estimate its coefficients. The balance depends on the design, noise, and how well the omitted effects can be approximated; sparsity by itself does not give a universal prediction guarantee.

Prediction and Inference Are Different Goals

Prediction concerns the accuracy of a fitted conditional mean over a relevant predictor distribution. Inference concerns uncertainty about an unknown quantity, such as one coefficient \(\beta^0_j\), a contrast of coefficients, or a treatment effect. Good prediction does not automatically provide valid uncertainty statements about individual coefficients: different coefficient vectors can generate similar predictions, especially when predictors are highly correlated.

Inference requires more than a small support. The target must be identified from the observed information, and the design and error conditions must support a valid uncertainty calculation. In particular, highly correlated or duplicated predictors can make separate coefficients indistinguishable even when the regression relationship is sparse. Some targets may remain identifiable while others do not. The relevant question is therefore not simply whether \(p_n>n\), but whether the chosen target can be learned under the model and design assumptions being used.

Common Pitfall: Treating Sparsity as a Complete Solution

A common mistake is to reason, “There are more predictors than observations, but only a few coefficients matter, so the important coefficients and their standard errors are automatically recoverable.” The correction is that sparsity is a structural assumption, not a complete identification or inference argument. A sparse model may still have multiple sparse representations, as the examples show; a useful approximation may still have a substantial remainder; and a prediction that works in one part of the predictor space may not work elsewhere.

Before interpreting a high-dimensional regression, specify the goal, the relevant sparsity notion, and the design assumptions. For prediction, assess the approximation remainder and the target population for predictions. For inference, ask whether the particular coefficient or contrast is identified and whether the uncertainty procedure is justified. The ratio \(p_n/n\) signals that conventional fixed-dimensional reasoning may not apply, but it does not by itself determine whether the analysis succeeds or fails.