Why Represent a Model as a Subset?
When several candidate predictors are available, a regression model can be identified by which predictors it includes. This representation separates the structure of a model from the numerical values of its estimated coefficients. For example, the model using age and annual premium is a different specification from the model using age alone, even before either model is fitted to data.
A precise subset notation helps avoid a frequent ambiguity: the intercept is part of the regression equation, but it is not ordinarily a selectable predictor. Model size counts the selected predictors under a stated convention; it does not automatically count the intercept.
The Regression Equation and the Intercept
For observations \(i=1,\ldots,n\), let \(y_i\) be the response and \(x_{ij}\) the value of candidate predictor \(j\) for observation \(i\). The model specified by subset \(S\) is
Here \(\beta_0\) is the intercept, \(\beta_j\) is the coefficient for selected predictor \(j\), and \(\varepsilon_i\) is the error. For the standard linear regression setup, assume the errors have mean zero, common variance \(\sigma^2\), and are independent across observations; normality is an additional assumption when required for exact finite-sample inference, not for defining the subset. Predictors not in \(S\) have no coefficient in this model equation. Equivalently, one may write a coefficient of zero for each excluded predictor, but the subset notation makes clear that it is not selected.
The intercept represents the fitted mean response when all included predictors equal zero, provided that this interpretation is meaningful for the predictor values. Its interpretation does not change the counting rule: if \(S=\{2,4\}\), then the model size is two, although the equation contains three regression coefficients, \(\beta_0,\beta_2,\beta_4\). The error variance \(\sigma^2\), when estimated, is also not a selected predictor and is not included in model size.
Worked Example: Translating a Subset Into an Equation
A hypothetical insurer models annual claim cost using four candidate predictors: \(x_1\), vehicle age in years; \(x_2\), annual distance driven in thousands of miles; \(x_3\), a zero-one indicator for an advanced safety feature; and \(x_4\), the driver's years licensed. The specified subset is \(S=\{1,3\}\). State the model equation, its size, and the number of regression coefficients, including the intercept.
Method. Use the index set to retain exactly the predictors whose indices are in \(S\). The intercept remains in the model, whether or not it appears in the subset.
Since \(1\in S\) and \(3\in S\), include \(x_1\) and \(x_3\). Since \(2\notin S\) and \(4\notin S\), omit \(x_2\) and \(x_4\). The equation is
The size is \(|S|=2\), because \(S\) contains two indices. There are \(2+1=3\) regression coefficients: one intercept and two predictor coefficients. The error variance is not part of this count.
Check. The equation contains precisely the two selected predictors, and its three regression coefficients agree with the two selected predictors plus the intercept. Common mistake avoided: counting the intercept and reporting size three. That would count coefficients, not selected predictors.
What Counts as a Candidate Predictor?
The subset notation is only unambiguous after the candidate predictors have been specified. A candidate may be a numerical variable, a transformed variable such as log income, an indicator, or a constructed term such as a product of two variables. The index refers to the candidate term as defined for the model, not necessarily to the original raw data field from which it was constructed.
For example, if age and age squared are listed as two separate candidates, they have separate indices and may be represented separately in \(S\). If an age-by-region interaction is listed as another candidate term, it too receives an index. Whether a particular analysis permits every possible combination of terms is a separate modeling restriction. The notation \(S\subseteq\mathcal{P}\) records the selected terms; by itself it does not impose restrictions such as requiring main effects whenever an interaction is included.
Categorical predictors need a stated convention. A factor with several levels is often represented in the regression equation by several indicator columns relative to a reference level. A model specification may treat those columns as a single selectable factor block, or may treat individual columns as separate candidates. The definition of size must match the convention being used. If the problem says a factor is entered as one block, count the factor as one selectable term, even though its equation uses multiple coefficients. If candidates are explicitly defined as individual design-matrix columns, count the selected columns instead.
Worked Example: A Grouped Categorical Predictor
A hypothetical pricing model has three selectable units: \(x_1\), driver experience; \(x_2\), annual mileage; and \(G\), a vehicle-use category entered as a single block. The category has three levels, with private use as the reference level and two indicator columns \(z_{i1}\) and \(z_{i2}\) for the other levels. A proposed model includes \(x_1\) and the whole \(G\) block, but excludes \(x_2\). Using the stated selectable-unit convention, write the equation and give both the model size and the number of regression coefficients, including the intercept.
Method. First identify the selectable units, rather than counting equation columns. Here the units are \(\{x_1,x_2,G\}\). The selected subset is \(S=\{x_1,G\}\), so it contains two units.
The factor block requires two indicator coefficients because it has two non-reference levels. The model equation is
The model size under the given convention is \(|S|=2\): one unit for experience and one for the grouped factor. The equation has \(1+1+2=4\) regression coefficients: the intercept, one experience coefficient, and two indicator coefficients. Mileage has no coefficient because its unit is excluded.
Check. The two selected units produce three predictor coefficients, because the category block contributes two; adding the intercept gives four regression coefficients. Common mistake avoided: reporting size three by counting the factor's two indicator columns separately despite the problem explicitly defining the factor as one selectable unit.
Why the Subset Identifies the Specification
Once the candidate terms and their coding have been fixed, the subset gives a direct rule for constructing the model equation. Define \(I(j\in S)\) to equal 1 when candidate \(j\) is selected and 0 otherwise. Then the same model can be written with all candidates displayed as
For \(j\in S\), the indicator is 1, so the term \(\beta_jx_{ij}\) appears. For \(j\notin S\), the indicator is 0, so that term contributes zero. This establishes why the subset is sufficient to specify which candidate terms are included. It also shows why the intercept is handled separately: its term is \(\beta_0\), not one of the indexed selectable terms.
The subset identifies a specified model, not necessarily a unique fitted relationship in every data set. If two candidate columns are identical, or one is an exact linear combination of other columns, distinct specifications can produce the same set of fitted values or fail to identify separate coefficients. This is a design-matrix issue associated with collinearity. The subset notation still records which terms were specified, but it does not guarantee that every coefficient is separately estimable.
Worked Example: Separating the Intercept From Selectable Terms
A hypothetical study uses four candidate terms: \(x_1\), policy duration; \(x_2\), log of insured amount; \(x_3\), a claim-history indicator; and \(x_4=x_1x_3\), an interaction term. The proposed equation is
Identify the subset, the model size, the excluded candidate, and the number of regression coefficients including the intercept.
Method. Match each predictor term in the equation to its candidate index. The equation includes \(x_1\), \(x_3\), and \(x_4\), so \(S=\{1,3,4\}\). The fact that \(x_4\) was constructed as an interaction does not remove it from the candidate set: it is explicitly listed as candidate 4.
The size is \(|S|=3\). Candidate 2, log insured amount, is excluded because its term does not appear. The equation contains three selected-predictor coefficients, \(\beta_1,\beta_3,\beta_4\), plus the intercept \(\beta_0\), for a total of \(3+1=4\) regression coefficients.
Check. The subset has exactly the three indices corresponding to the three displayed predictor terms. Common mistake avoided: counting the interaction as part of the intercept or overlooking it because it is derived from other variables. Under the stated candidate list, it is a selectable term with its own index.
Common Traps and Practical Checks
Before interpreting a model label or comparing model sizes, verify what its candidates mean. In particular, keep the following distinctions clear:
- Predictor count versus coefficient count. Model size ordinarily counts selected predictors, excluding the intercept. A grouped factor can make the coefficient count larger than the number of selected units.
- Raw variables versus candidate terms. A transformation or interaction counts as a candidate if it is separately defined in the candidate set. Do not silently combine it with its source variables.
- Model specification versus estimability. A subset records the terms specified. Redundant or linearly dependent design columns can prevent unique coefficient estimation, even though the subset itself is clear.
- Term restrictions versus subset notation. A subset can describe a collection of terms without asserting that the collection satisfies any additional hierarchy or scientific constraints. Such restrictions must be stated separately.
A reliable reading procedure is to write down the candidate units and their labels, mark which indices appear in the equation, form \(S\), and then calculate \(|S|\). Count the intercept only if the question asks for the number of regression coefficients or explicitly defines a different size convention.
Check Your Understanding
For each problem, use the candidate units and counting convention stated in that problem.
- A model has candidates \(x_1\) (age), \(x_2\) (distance), and \(x_3\) (a binary indicator). Its equation is \(y_i=\beta_0+\beta_2x_{i2}+\varepsilon_i\). Give \(S\), model size, and the number of regression coefficients including the intercept. (Answer: \(S=\{2\}\), size \(1\), and \(2\) regression coefficients.)
- There are five candidate terms, including a separately listed squared predictor \(x_5\). A model includes candidates 1, 3, and 5. State its size and which candidates are excluded. (Answer: size \(3\); candidates 2 and 4 are excluded.)
- A three-level factor is defined as one selectable block, represented by two indicator columns. A model includes that block and one numerical predictor, but excludes two other numerical candidates. Give the model size and the number of predictor coefficients. (Answer: size \(2\); \(3\) predictor coefficients.)
- Four candidates are listed: \(x_1\), \(x_2\), \(x_3\), and the separately listed interaction \(x_4=x_1x_2\). An equation contains the intercept and terms \(x_1,x_2,x_4\). Give the subset, size, and total regression-coefficient count. (Answer: \(S=\{1,2,4\}\), size \(3\), and \(4\) regression coefficients.)
- A model uses two selected candidate columns and an intercept. The data show that one selected column is exactly twice the other. How many candidate predictors are specified, and does the subset alone guarantee that both predictor coefficients are separately identifiable? (Answer: \(2\) candidate predictors; no, exact linear dependence means the separate coefficients are not identifiable.)