Why Specify a Dataset Before Analyzing It?
A dataset can look orderly and still be unsuitable for analysis. A time column might be measured in minutes when the analysis expects hours; a concentration field might contain text; or a required identifier might be absent from some records. Without a written specification, these problems are easy to miss or to handle inconsistently across files and analysts.
A dataset schema makes the expected structure explicit. It describes the fields and the rules that incoming data must satisfy before they are used. Schema validation checks whether a particular dataset conforms to that contract. It does not establish that the measurements are scientifically plausible, that different fields agree with one another, or that the data are free of source errors. Those are separate quality-control questions.
What a Field Specification Must Say
Field names should be stable and unambiguous. For example, TIME is more useful when accompanied by a definition such as “time from the stated study reference, in hours” than when its meaning is left to the file creator. A schema should also state whether spelling and capitalization matter. If two files use SUBJID and ID for the same concept, an explicit mapping or a consistent naming rule is needed; analysts should not have to guess.
The declared type should describe the values the field is intended to contain, such as text, integer, or numeric. An identifier may consist of digits but still be a text field: arithmetic on subject identifiers is not meaningful. Units must be stated for measured quantities. A numeric value alone cannot establish whether, for example, 2.5 represents hours or minutes. When a file carries unit metadata, validation can compare it with the schema. When units are not carried with the values, the unit must be established by the data-transfer contract or documented source metadata; it cannot be inferred reliably from the numbers.
For fields with a finite vocabulary, specify the permitted values exactly. For example, a visit label might be limited to baseline, week4, or week8. This differs from a numeric range check: the schema checks membership in a specified set, while checks on plausible numerical magnitudes and distributions are separate.
Requiredness needs particular care. A field can be required at the column level, at the record-value level, or both. A required field may be expected as a column in the file and must have a nonmissing value in every applicable record. An optional field may be absent from the file. If an optional field is supplied, however, its values must still meet its type, unit, and allowed-value rules. Whether a blank or null value is permitted is a separate rule that should be stated explicitly. Optional does not automatically mean that any supplied value, including an invalid one, is acceptable.
| Schema element | Question it answers | Example specification |
|---|---|---|
| Field name | Which column is being described? | TIME |
| Type | What kind of values are valid? | Numeric |
| Unit or definition | What does a value represent? | Elapsed time, hours |
| Allowed values | Which categories or codes are accepted? | baseline, week4, week8 |
| Requiredness and missingness | Must the column and its values be present? | Required column; nonmissing for each record |
A Practical Validation Sequence
Validation is easier to reproduce when it follows a defined sequence. First identify the schema version and the file to which it applies. Then check the file’s structure, followed by its field values. Preserve the original input and report errors rather than silently dropping records, renaming fields, changing units, or replacing unexpected values. If a documented transformation is needed, keep it traceable and validate the resulting dataset against the intended schema.
Record the schema name and version, the dataset’s intended role, and any documented field mappings or unit conversions.
Confirm that required columns are present and apply the stated policy to unrecognized extra columns. Distinguish an absent optional column from an absent required column.
For every field present, check its applicable type, unit, missingness, and allowed-value rules in every applicable record. This includes optional fields when they are supplied.
List failures with field names and record locations, and state whether the dataset passes. Do not turn a failed validation into a pass by silently coercing or deleting problematic values.
Worked Examples
Worked Example: Writing a Schema for a Sample Dataset
Suppose a hypothetical project receives one record per concentration sample. Define a schema for an incoming file that contains subject identifier, elapsed time, measured concentration, and an optional visit label.
Schema specification. ID is required text and must be nonmissing in every record. TIME is required numeric elapsed time in hours and must be nonmissing. DV is required numeric concentration in mg/L and must be nonmissing. VISIT is optional text; if supplied, each nonmissing value must be one of baseline, week4, or week8. For this contract, a supplied VISIT column may contain missing values, but it may not contain other labels. The file must declare or be linked to the stated units for TIME and DV.
These rules make two distinct decisions visible. A file without VISIT can satisfy the requiredness rule because VISIT is optional. A file that does include VISIT must still satisfy the allowed-value rule for each nonmissing entry. The schema says nothing about whether a concentration is scientifically plausible; that requires other checks.
Worked Example: Required Columns, Required Values, and Optional Fields
Consider a hypothetical three-record file with the schema just defined. Its columns are ID, TIME, DV, and VISIT. The records are: (P01, 0, 2.40, baseline), (P01, 4, blank, week4), and (P02, 8, 0.90, unknown). Assume the file declares hours for TIME and mg/L for DV.
Structure. All required columns—ID, TIME, and DV—are present. The optional VISIT column is also present. Therefore, there is no missing-column failure.
Required values. There are three required fields across three records, so there are \(3 \times 3 = 9\) required field-record values to check. One of these nine, DV in the second record, is blank. The required-value completeness is therefore
or 88.89%. This descriptive percentage does not make the file pass: a required value is missing.
Optional field. VISIT is optional, but it is present, so its nonmissing values must be checked. The first two values, baseline and week4, are allowed. The value unknown is not in the permitted set, so the file also fails the VISIT rule. The correct result is fail for two reasons: one required DV value is missing, and one supplied optional VISIT value is invalid. Removing VISIT from the schema’s required fields does not exempt its supplied values from validation.
Worked Example: A Passing File and an Absent Optional Column
A second hypothetical file contains ID, TIME, and DV, but no VISIT column. It has two records: (P03, 1.5, 1.20) and (P04, 6, 0.55). The file declares TIME in hours and DV in mg/L; both numeric fields are stored as numeric values, and both IDs are nonmissing text.
The three required columns are present. There are \(3 \times 2 = 6\) required field-record values; all six are present and have the specified types. The stated units match the schema. VISIT is optional and absent, which is permitted. There are no allowed-value violations because no VISIT values were supplied. Under the defined rules, this file passes schema validation.
If the same file instead declared TIME in minutes, the numbers alone would not verify the expected unit. It would fail the unit requirement as submitted, unless a documented conversion were applied and the converted output were validated. A schema validator should not assume that a value such as 6 is in hours merely because hours are expected.
Why the Pass Rule Is a Conjunction
A pass decision must account for both field presence and field validity. Let \(F\) be the set of fields in the dataset, \(R\) its applicable records, and \(C(f,r)\) the applicable schema conditions for field \(f\) in record \(r\). A dataset passes only when the required-presence conditions hold and every condition for every field that is present is satisfied. In logical form:
Here, \(\land\) means “and,” and \(\bigwedge\) means that every listed condition must hold. The final term includes optional fields whenever they are present. This follows directly from the purpose of a schema: a file must conform to the rules that apply to the contents it actually supplies, while also providing all required information. Checking only required fields would incorrectly allow a supplied optional field to violate its declared type or allowed-value rule.
Diagnostics, Alternatives, and Limits
A useful validation report identifies the schema version, file, field, record location, failed rule, and observed value where appropriate. Distinguish structural failures (such as a missing required column) from value failures (such as a blank required cell or an unallowed category). Record counts can help summarize completeness, but a percentage alone can hide which field failed; the file should not pass merely because most cells are valid.
A strict schema rejects unexpected names, types, units, or categories. A more permissive policy may allow extra columns or documented aliases, but those permissions should be specified rather than applied informally. Likewise, automatic type coercion can be convenient—for example, converting text “2.5” to a number—but it can also conceal malformed values or alter identifiers. If coercion is allowed, document exactly what conversion is permitted, retain the original input, log changes, and validate the transformed output. Silent coercion is not a reproducible validation policy.
Schema validation has important limits. It confirms conformance to declared rules; it does not prove that source values are correct, that a unit declaration is truthful, or that fields are scientifically meaningful. A numeric value can satisfy a numeric type rule but still be implausible. A row can satisfy every field-level rule while containing an inconsistency between fields. Numerical range and distribution checks, cross-field consistency checks, and source reconciliation address different questions and should not be mistaken for schema validation.
Common Mistakes
- Checking only required fields. Requiredness determines whether a field may be absent; it does not waive rules for an optional field that is supplied. Validate each present field.
- Confusing an absent column with a blank value. A required column can exist while some of its records have missing values. Check both conditions.
- Assuming numbers reveal units. A numeric TIME value does not show whether it is in hours or minutes. Require unit metadata or a documented contract.
- Treating a successful schema check as complete data quality assurance. Type and field checks do not establish plausibility or agreement between fields.
- Fixing failures without a record. Silent recoding, deletion, or conversion makes it difficult to reproduce what was analyzed. Use documented, traceable transformations and validate the resulting data.
Check Your Understanding
Answer each question using the schema-validation rules above. Solutions follow the questions.
- A required field is present as a column but is blank in one record. Is the structural check sufficient for a pass? Why?
- An optional categorical field is absent from a file. The required fields all pass. Is the absent optional field a failure?
- An optional field is supplied, and one of its values is not in the declared allowed set. Does optionality excuse the value?
- A required numeric time field contains 4.2, but the file does not declare units and no data-transfer contract establishes them. Can the validator confirm that the unit rule passes?
- What is the required-value completeness if a file has four records and two required fields, with one missing required value?
Solutions. (1) No. The column-presence check passes, but the required-value check fails for that record. (2) No. An optional field may be absent, provided no other applicable rule requires it. (3) No. If the optional field is supplied, its applicable allowed-value rule still applies, so the file fails. (4) No. The numeric value does not establish its unit; the validator needs unit metadata or a documented contract. (5) There are \(4 \times 2 = 8\) required field-record values, of which \(8-1=7\) are present, so completeness is \(7/8=0.875\), or 87.50%. The file still fails because at least one required value is missing.