Notation for Samples, Features, and Responses

Why Notation Matters

A supervised learning data set pairs an input with a response for each observed case. Notation makes those pairs precise: it tells us what kind of values are possible, which values belong to the same case, and which quantities are observed. It also distinguishes the data that could be generated by an experiment from the particular values that were recorded.

We will write \(\mathcal X\) for the feature space and \(\mathcal Y\) for the response space. These spaces describe possible values, not the observed data themselves. An individual feature vector belongs to \(\mathcal X\), and its paired response belongs to \(\mathcal Y\). The feature space may contain vectors of numbers, categories, or other structured inputs; the response space may contain numbers or labels.

Definition: For case \(i\), let \(X_i\) denote its feature value and \(Y_i\) its response value. The pair \(Z_i=(X_i,Y_i)\) is the observation for that case. If the feature and response are recorded as \(x_i\) and \(y_i\), respectively, then \(z_i=(x_i,y_i)\) is the realized observation. The feature and response spaces are \(\mathcal X\) and \(\mathcal Y\), with \(X_i,x_i\in\mathcal X\) and \(Y_i,y_i\in\mathcal Y\).

Uppercase letters conventionally denote random variables, while lowercase letters denote specified or observed values. Thus \(X_i\) is not the same thing as \(x_i\): the former refers to a quantity that can take values in \(\mathcal X\), and the latter is one particular value. In a data table, lowercase notation is usually natural because the entries have already been observed. When describing the data-generating experiment, uppercase notation makes clear that the values are not yet specified.

Indexing Cases and Features

The index \(i\) labels cases. With \(n\) cases, \(i\) ranges from \(1\) through \(n\). Each case contributes one feature value and its corresponding response. The pairing is essential: \(x_i\) goes with \(y_i\), because they were recorded for the same case. The notation does not say that different cases are independent or identically distributed; those are additional assumptions, not consequences of using indices.

When each feature value has \(p\) numerical coordinates, we can write \(X_i=(X_{i1},\ldots,X_{ip})\), and its realized value as \(x_i=(x_{i1},\ldots,x_{ip})\). Here \(j\) labels a feature coordinate, so \(X_{ij}\) is the \(j\)th feature of case \(i\). This coordinate notation applies when the feature space is a subset of \(\mathbb R^p\). For a more general feature space, \(X_i\) can be treated as one object without breaking it into numerical coordinates.

Worked Example: Reading a Two-Feature Observation

Suppose a study records two features for each patient: age in years and a treatment indicator coded as 0 or 1. The response is a numerical measurement. For one patient, the recorded feature vector is \(x_2=(54,1)\), and the recorded response is \(y_2=12.4\).

There are \(p=2\) feature coordinates. Coordinate \(j=1\) is age, so \(x_{21}=54\). Coordinate \(j=2\) is the treatment indicator, so \(x_{22}=1\). Together, these make the feature vector \(x_2=(x_{21},x_{22})=(54,1)\). The response paired with this feature vector is \(y_2=12.4\), and the realized observation is \(z_2=(x_2,y_2)=((54,1),12.4)\).

The index 2 identifies the case; it does not mean that the patient has two features. The coordinate index \(j\) identifies a feature within the case.

Writing the Complete Dataset

The complete sample consists of all \(n\) paired observations. It is useful to distinguish the random data object from its realized values. Write \(\mathcal D_n=((X_1,Y_1),\ldots,(X_n,Y_n))\) for the random ordered sample, and \(\mathcal d_n=((x_1,y_1),\ldots,(x_n,y_n))\) for the observed ordered data. The subscript \(n\) indicates the number of cases.

\mathcal D_n=((X_1,Y_1),\ldots,(X_n,Y_n)), \qquad \mathcal d_n=((x_1,y_1),\ldots,(x_n,y_n)).

For numerical features, the same observed information can be displayed as a feature matrix and response vector. The \(i\)th row of the matrix is the feature vector for case \(i\); the \(i\)th entry of the response vector is that case’s response.

\mathbf X= \begin{pmatrix} x_{11} & \cdots & x_{1p}\\ \vdots & \ddots & \vdots\\ x_{n1} & \cdots & x_{np} \end{pmatrix} \in\mathbb R^{n\times p}, \qquad \mathbf y= \begin{pmatrix} y_1\\ \vdots\\ y_n \end{pmatrix} \in\mathbb R^n.

This matrix display presumes numerical, \(p\)-coordinate features; it is not a requirement that every feature space have that form. In particular, a categorical response can be recorded as a label rather than a numerical entry, so the response vector need not belong to \(\mathbb R^n\). The paired-list notation works in either situation.

Worked Example: From a Table to Dataset Notation

Three apartments are described by square footage and number of bedrooms. Their recorded values and monthly rents are:

Case \(i\)	Square footage \(x_{i1}\)	Bedrooms \(x_{i2}\)	Rent \(y_i\)
1	600	1	1,200
2	850	2	1,650
3	700	1	1,400

There are \(n=3\) cases and \(p=2\) features. The realized feature vectors are \(x_1=(600,1)\), \(x_2=(850,2)\), and \(x_3=(700,1)\). The corresponding responses are \(y_1=1200\), \(y_2=1650\), and \(y_3=1400\). Therefore, the realized ordered dataset is \(\mathcal d_3=(( (600,1),1200),((850,2),1650),((700,1),1400))\).

The feature matrix has three rows and two columns, one row per apartment and one column per feature. The response vector has three entries, in the same case order:

\mathbf X= \begin{pmatrix} 600&1\\ 850&2\\ 700&1 \end{pmatrix}, \qquad \mathbf y= \begin{pmatrix} 1200\\ 1650\\ 1400 \end{pmatrix}.

The second row of \(\mathbf X\) must be read with the second entry of \(\mathbf y\): the apartment with 850 square feet and two bedrooms has recorded rent 1,650.

Two Useful Notation Facts

The feature-matrix representation is simply a rearrangement of the same observations when features have numerical coordinates. For fixed \(n\) and \(p\), the list of \(n\) feature vectors and \(n\) responses determines the matrix and vector, and those rows and entries recover the list. This is a representation fact, not an assumption about how the values were generated.

Derived fact: A numerical feature matrix for \(n\) cases and \(p\) coordinates has \(np\) entries. Appending one scalar response for each case gives \(n(p+1)\) scalar data entries in total.

To verify the count, each of the \(n\) rows contains \(p\) feature entries, giving \(np\) feature entries. There are \(n\) scalar responses, one for each row. Thus the total is \(np+n=n(p+1)\). For the apartment example, this is \(3(2+1)=9\): six feature entries and three rent entries.

A second useful fact concerns case order. The ordered list records which observation has index \(i\), while an unordered collection suppresses that ordering. Reordering all cases together does not change the collection of observed pairs, but it changes their indices. For example, if case 2 is moved to the first row, its response must move with its feature vector.

Derived fact: Let \(\pi\) be a permutation of \(1,\ldots,n\). The reordered list \(((x_{\pi(1)},y_{\pi(1)}),\ldots,(x_{\pi(n)},y_{\pi(n)}))\) contains exactly the same paired observations as \(\mathcal d_n\), only in a different order. Consequently, for any real-valued function \(g\) defined on the observed pairs, the sum of its values is unchanged by reordering.

For the sum claim, set \(a_i=g(x_i,y_i)\). A permutation uses each index from 1 to \(n\) exactly once. Therefore, the terms \(a_{\pi(1)},\ldots,a_{\pi(n)}\) are precisely the terms \(a_1,\ldots,a_n\) in a different order, and addition is unchanged by order:

\sum_{i=1}^{n}g(x_{\pi(i)},y_{\pi(i)}) = \sum_{i=1}^{n}a_{\pi(i)} = \sum_{i=1}^{n}a_i = \sum_{i=1}^{n}g(x_i,y_i).

This is why a notation such as \(\{(x_i,y_i)\}_{i=1}^n\) is often used informally for a dataset. Strictly, ordinary set notation does not record order and does not retain repeated identical elements. The indexed list or multiset interpretation keeps the number of cases and any duplicate observations visible.

Worked Example: Duplicate Records and Case Order

Suppose three recorded observations are \(z_1=(2,A)\), \(z_2=(2,A)\), and \(z_3=(5,B)\), where the first component is a feature and the second is a response label. The first two cases happen to have identical recorded values, but they are still two cases. The ordered sample is \(((2,A),(2,A),(5,B))\), with \(n=3\).

If the first and third cases are reordered, the list becomes \(((5,B),(2,A),(2,A))\). The feature and response in each pair stay together, and the same three observations remain present. Writing only the ordinary set \(\{(2,A),(5,B)\}\) would misleadingly display two distinct values and hide that \((2,A)\) was observed twice. The indexed list preserves both the multiplicity and the sample size.

What the Notation Does Not Assume

The symbol \(n\) identifies the sample size in the notation; it does not, on its own, say whether the size was fixed in advance or could vary. Likewise, writing \(Z_1,\ldots,Z_n\) does not establish that the observations are independent, identically distributed, or drawn from any particular probability distribution. The notation identifies the objects and their pairing. Statements about their joint distribution require separate assumptions.

The same caution applies to the feature matrix. Its rows are organized by cases, and its columns by feature coordinates. That arrangement alone does not establish that the feature values are random or fixed under a particular design. It is safe to use the matrix as a compact way of recording realized numerical features without making a claim about the experiment that produced them.

These distinctions matter because an analysis may refer either to possible data before observation or to the particular data in hand. Using \(X_i,Y_i\) for random quantities and \(x_i,y_i\) for realizations makes that distinction explicit. It also prevents a common ambiguity: an expression involving \(X_i\) may describe a random object, while the same expression with \(x_i\) describes a value calculated from observed data.

Common Mistake

Wrong reasoning: “The dataset is written with braces, so it is an ordinary set, and repeated rows can be discarded.” Correction: Braces indexed by \(i=1,\ldots,n\) are often shorthand for the complete indexed sample. Repeated records still correspond to distinct cases. Use the ordered list or an explicit index when multiplicity matters.
Wrong reasoning: “Because the data are in a matrix, the feature values must have been fixed by the experiment.” Correction: A matrix describes the arrangement of numerical values. Whether those values are treated as random or fixed is a separate feature of the sampling setup.
Wrong reasoning: “\(X_{ij}\) labels case \(j\).” Correction: The first index \(i\) labels the case, and the second index \(j\) labels a feature coordinate within that case.

Key takeaway: Use \(\mathcal X\) and \(\mathcal Y\) for possible feature and response values, uppercase symbols for random quantities, and lowercase symbols for realized values. Keep each \(x_i\) paired with its \(y_i\), and let the index \(i\) identify the case throughout the data representation.

Check Your Understanding

Answer each question using the notation introduced above.

For a sample of \(n\) cases with \(p\) numerical features, what do the row and column indices of \(x_{ij}\) identify?
Explain the distinction between \(Y_4\) and \(y_4\).
If a response is a class label rather than a number, can the paired-list notation \(\mathcal d_n\) still be used? Why?
What information can be lost by writing a dataset as an ordinary set when two cases have identical feature-response pairs?
Does writing \(Z_1,\ldots,Z_n\) imply independence? State what the notation does establish.