Why Representation Matters
Before specifying an imputation model for nested data, establish what each row represents and which unit each variable describes. A dataset might contain patient records nested within clinics, students within schools, or repeated measurements within people. If a clinic-level characteristic is copied onto every patient row, the file contains multiple entries for that characteristic, but it still describes clinics rather than individual patients. The data layout must make that distinction recoverable.
This tutorial focuses on representing two-level data: level-1 units nested within level-2 clusters. It does not specify which imputation model is appropriate for a particular estimand. Its purpose is more basic: to identify the level of each column and to check whether the chosen layout faithfully represents the data. A sound layout prevents software from confusing a repeated cluster attribute with a separately measured attribute for every individual.
A variable's level is determined by what it measures and how it is defined, not by its storage type or by whether a particular sample happens to show variation. For example, clinic treatment policy is a level-2 variable if one policy applies to each clinic. Patient age is level 1 because it is measured for individual patients. A numerical patient ID is still an identifier, not a quantitative patient characteristic. Likewise, a cluster-level variable can happen to have the same value in every observed clinic; that sample pattern does not change the unit it describes.
The basic two-level representation assumes that each level-1 record belongs to exactly one level-2 cluster, that cluster membership is known, and that any cluster-level attribute attached to level-1 records has one consistent value within each cluster. These are data-structure assumptions, not claims that clusters are statistically independent or that a particular imputation model is adequate. If a person can belong to multiple clusters, or if the hierarchy has more than two levels, a simple one-cluster-per-row identifier may not be sufficient.
Long Format: One Row per Level-1 Unit
In long format, each row represents one level-1 unit. A cluster identifier appears on every row so that records from the same cluster can be grouped. Level-1 variables usually have one entry per row. A level-2 variable may also appear on every row, with the same cluster-level value repeated for all level-1 units in that cluster.
Worked Example: Mapping Columns in a Long Dataset
Suppose a hypothetical dataset records two patients from each of three clinics. The patient outcome and age are measured for each patient. Clinic type is recorded once for each clinic in the underlying source information, then copied to the clinic's patient rows for analysis.
| clinic_id | patient_id | age_years | outcome_score | district_type |
|---|---|---|---|---|
| C1 | P11 | 34 | 8 | urban |
| C1 | P12 | 51 | 6 | urban |
| C2 | P21 | 43 | 7 | rural |
| C2 | P22 | 28 | 9 | rural |
| C3 | P31 | 62 | 5 | urban |
| C3 | P32 | 39 | 8 | urban |
Mapping. clinic_id is the cluster identifier: it links each patient row to a clinic. patient_id identifies a level-1 unit and is not an analysis variable unless the study has a specific use for that identity. age_years and outcome_score are level-1 variables because each value describes a patient. district_type is level 2 because it describes the clinic's district, not an independently measured attribute of each patient.
There are three school-level district_type entries, one per clinic, even though the long-format column contains six rows; because district_type is urban or rural, there are at most two distinct values. The two repeated rows per clinic do not create two independent measurements of clinic type. They are repeated representations of the same clinic attribute.
For a basic row-count check, the dataset has \(J=3\) clinics and \(n_j=2\) patients in each clinic, so the total number of long-format rows is \(\sum_{j=1}^{J} n_j=2+2+2=6\). The six rows represent six patients, not six clinics.
A useful rule is to ask, column by column: “What is one value attached to?” If the value is attached to a person, patient, visit, or other nested unit, it is level 1. If it is attached to the cluster as a whole, it is level 2. If it exists only to identify membership or records, it is an identifier. This rule is more reliable than judging by the number of distinct values in the current sample.
Wide Format: One Row per Cluster
In wide format, each row represents one level-2 cluster. Cluster-level variables appear once per row. Values for level-1 units are stored in separate sets of columns, often indexed by a within-cluster position such as 1, 2, or 3. A wide file therefore needs a convention that makes clear which columns are repeated measurements or records, and how those records align across variables.
Worked Example: Reading and Converting a Wide Dataset
Consider two hypothetical clinics with different numbers of patients. The wide file reserves space for three patients per clinic. The last patient slot for clinic C2 is unused.
| clinic_id | policy | patient_id_1 | score_1 | patient_id_2 | score_2 | patient_id_3 | score_3 |
|---|---|---|---|---|---|---|---|
| C1 | A | P11 | 8 | P12 | 6 | P13 | 7 |
| C2 | B | P21 | 5 | P22 | 9 |
Here, clinic_id identifies the cluster, and policy is a level-2 variable. Each patient_id_k identifies the level-1 unit in slot \(k\), while each score_k is a level-1 outcome for that same patient. The suffix \(k\) is essential: score_2 belongs to patient_id_2, not to an unspecified second clinic.
To convert to long format, create one row for every occupied patient slot and carry the clinic identifier and policy onto each resulting row. Clinic C1 contributes three rows and clinic C2 contributes two, for \(3+2=5\) long-format rows. The empty third slot for C2 is a structural blank: no third patient record is present in that cluster. It is not a missing score for an existing patient. If C2 had a third patient ID but no recorded score, that score entry would instead be missing data for an existing level-1 unit.
The wide and long versions can encode the same observed records, provided the slot alignment is correct and no information is lost in conversion. Wide format can make the cluster-level structure visible, while long format makes the level-1 units explicit as rows. Neither layout, by itself, states how missing values should be imputed.
A Short Justification for the Long-Format Mapping
The row-count relationship helps explain why a repeated level-2 column must not be interpreted as new cluster information. Let \(J\) be the number of clusters, and let \(n_j\) be the number of level-1 units in cluster \(j\). In long format, the total number of rows is
If \(Z_j\) is a level-2 attribute for cluster \(j\), then the long-format entry in the row for unit \(i\) in cluster \(j\) is \(Z_{ij}=Z_j\). Thus, the \(n_j\) entries in cluster \(j\) are copies of one cluster value. The number of recorded cells in that column is \(N\), but the number of cluster-level source values is \(J\). For example, if three clinics have patient counts 2, 2, and 3, then the long file has \(N=2+2+3=7\) rows and seven displayed entries for a clinic attribute, but those entries correspond to three clinics.
This is a representation identity, not a statistical independence argument. It explains what the rows mean; it does not determine how the attribute should enter an imputation model or a subsequent analysis.
Auditing Columns Before Imputation
A practical audit starts with the unit represented by each row, then assigns a role to every column. It is useful to record the variable's meaning, level, and expected repetition pattern. For a long file, a level-2 variable should be constant within each cluster if it is copied to level-1 rows. For a wide file, the suffix or position convention should consistently associate each level-1 record's variables. These checks detect representation errors before they are confused with statistical-model problems.
Write down whether a row represents a person, visit, patient, clinic, or another unit. Do not infer this solely from the file's appearance.
Check that each level-1 record has one valid cluster identifier and that the identifier is stable across all variables describing that record.
Separate substantive level-1 variables, substantive level-2 variables, and identifiers. Note the unit and, where relevant, the units of measurement.
In long data, confirm cluster-level values are consistent within cluster. In wide data, verify that each numbered set of columns refers to the same level-1 unit.
Determine whether a blank means an existing unit has an unobserved value or whether that unit or slot does not exist.
Worked Example: Diagnosing a Mixed-Level Data File
A hypothetical long-format file contains a row for each patient and columns for clinic_id, patient_id, treatment_group, and symptom_score. In one clinic, the first three rows show treatment group A, A, and B. The clinic's protocol is said to assign one treatment group to the entire clinic.
The intended mapping is: clinic_id is the cluster identifier; patient_id is a level-1 identifier; symptom_score is level 1; and treatment_group is intended to be level 2 because the protocol is assigned to the clinic. The entries A, A, and B violate the required within-clinic consistency for a copied clinic-level variable. This is not fixed by declaring the column level 1: the definition in the study says the assignment belongs to the clinic.
The data manager should check the source records. If the B is a transcription or merge error, correct it using the verified source. If treatment was actually assigned separately to patients, then the study description and variable definition must be revised and the variable is level 1. Until this discrepancy is resolved, the file does not faithfully encode the stated design. The audit identifies a data-definition or data-quality problem; it does not decide which value is true.
Common Mistakes and Limitations
- Counting repeated entries as new cluster observations. In long format, a clinic variable may occupy many cells, but those repeated cells represent the clinics' values, not a new clinic measurement for each patient.
- Assigning level by data type. A numeric code can be an identifier, and a text category can be a substantive variable. The level depends on the unit described.
- Assigning level from observed variation alone. A level-2 variable may have only one observed value in a small dataset. Conversely, inconsistent copies of a cluster-level value are a warning to investigate, not proof that it is a level-1 variable.
- Treating every blank alike. A missing measurement for an existing person differs from an empty wide-format slot where no person exists.
- Assuming a layout is an imputation model. Long or wide storage does not ensure that an imputation procedure respects clustering. Representation is necessary groundwork, not a guarantee of valid imputation.
Long format is often convenient when software expects one record per level-1 unit and uses a cluster identifier. Wide format can be useful when the number of nested records is fixed and the software expects one row per cluster. The main alternative is not “correct versus incorrect”: both can be valid if the data structure and variable meanings are encoded unambiguously. Their limitations differ. Long files repeat level-2 entries and require a reliable cluster ID; wide files can become unwieldy with many or unequal numbers of level-1 units and require careful alignment. Conversion is safe only when identifiers, slot mappings, and missing-versus-structural distinctions are preserved.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A long file has 4 schools, with 5, 6, 4, and 5 students respectively. How many rows does it have if each row is one student? How many school-level values underlie a school funding category repeated on every student row?
- A wide file has one row per clinic and columns for
patient_id_1,weight_1,patient_id_2, andweight_2. State the level of each column's information and explain how the suffixes are used. - A clinic-level region variable is repeated on patient rows, but within one clinic the entries are “north” and “south.” What should be concluded, and what should be checked?
- In a wide row, the third patient ID is blank and the third outcome is blank. Does this establish a missing outcome for a patient? Explain what additional information is needed.
Solutions. (1) The row count is \(5+6+4+5=20\). The repeated funding-category entries represent 4 school-level values, one per school, even if some schools share the same category. (2) The clinic row identifies the cluster. Each patient ID identifies a level-1 unit in the numbered slot, and each corresponding weight is a level-1 measurement for that same unit. Suffix 1 aligns the first ID with the first weight, and suffix 2 aligns the second ID with the second weight. (3) The stated level-2 definition requires one region value per clinic, so the inconsistent entries signal a possible data or definition problem. Check the source and the intended unit of measurement; do not silently recode the variable as level 1. (4) No. If there is no third patient, both blanks describe an unoccupied slot. If a third patient exists but the outcome was not recorded, the outcome is missing for an existing unit. The patient roster or other source information is needed to distinguish these cases.