Why the Stored Value Is Not Always the Observed Value
A regression data file may appear complete even when some measurements were never obtained. A blank cell, a text string such as “NA,” and a number such as \(-99\) can all represent missing information, depending on how the data were collected and imported. Conversely, a zero may be a genuine measurement, and a blank may mean that a measurement was not applicable in the first place. These cases must not be treated as interchangeable.
This distinction matters because data-cleaning choices determine which values enter a regression analysis and what the variables mean. Replacing an undocumented code with missing data without checking its meaning can discard valid observations. Treating a missing code as a real number can create a false measurement. Filling a value that does not apply with zero can change the question being analyzed.
The stored entry alone may not reveal which category it belongs to. A blank could mean “not measured,” “not applicable,” or “not entered.” A value of \(-9\) could be a valid quantity in one variable and a missing code in another. Meaning comes from the data dictionary, collection procedure, units, and context—not from a number’s appearance.
A Traceable Audit Before Recode
Audit imported data before fitting a regression model. First preserve the original file and inspect how each column was imported. Software may convert a blank to a missing value, retain it as an empty string, or read a special code as an ordinary number. Mixed entries can also cause a numeric column to be imported as text. Record the variable definition, expected type, units, valid categories or range, and any missing codes documented by the source.
Then inspect unusual values and their context. Check whether they match a documented code, violate a known constraint, or occur only when another field makes the measurement inapplicable. A value outside a plausible range is a reason to investigate, not proof that it is missing: it could be a data-entry error, a unit conversion problem, an unusual but real observation, or a valid sentinel code. If the source cannot resolve an invalid entry, a documented decision may be needed to treat it as unavailable; do not silently make that decision during import.
Keep the original values and check the imported types, blanks, text strings, and unusual numeric codes.
Use the data dictionary and collection context to establish the variable’s meaning, units, valid values, and documented missing codes.
Decide whether it is a valid measurement, a missing code, structurally inapplicable, or invalid and in need of investigation.
Convert confirmed missing codes to the analysis software’s missing-value representation. Retain valid values, and document how inapplicable or unresolved invalid entries are handled.
Compare the cleaned values with the original file, count each classification, and save the recoding rules and reasons.
A useful rule is that a recode should be supported by an explicit reason: for example, “the codebook defines \(-9\) as not recorded for this variable.” “It looks unusual” is not enough. The audit should also be variable-specific. The same numeric value can be a missing code in one column and an actual observation in another.
Worked Example: Auditing a Sentinel and an Impossible Age
Suppose an adult health file contains the following imported ages, in years: \(34, 29,\) a blank, \(-9, 41, 999, 0,\) and \(27\). The codebook says that \(-9\) means “age not recorded” and \(999\) means “unknown age.” The file contains adults enrolled in a program, so age \(0\) is inconsistent with the study population. What should be done?
The blank is an unrecorded value, and both \(-9\) and \(999\) are documented missing-value codes. These four entries should be represented as missing for age, with the codebook rule retained in the audit record. The values \(34,29,41,\) and \(27\) are plausible adult ages and remain observed values.
The entry \(0\) is not one of the documented missing codes. It is inconsistent with the adult population, so it should be flagged and checked against the source record. If it is corrected from the source, use the verified age. If it is confirmed to be erroneous and cannot be recovered, document its treatment as unavailable. Until this check is done, do not silently recode it as a missing value or accept it as an adult age.
If the source confirms that \(0\) is an unrecoverable error, the cleaned age column has four observed values and four missing values. The mean of the observed ages is
This mean describes the four records with usable observed ages; the audit does not by itself establish whether those records represent all adults in the program. If \(0\) were mistakenly retained as an age, the computed mean for the five values would instead be \(131/5=26.2\) years. That change is caused by treating an invalid entry as a measurement, not by a different legitimate summary of the same usable ages.
Structural Inapplicability Is Not a Missing Measurement
A variable may be defined only for part of the data set. For example, gestational week is defined for a pregnancy, but not for a person who is not pregnant. The nonpregnant person does not have an unobserved gestational week; under this variable definition, the measurement does not apply. By contrast, if a pregnant participant’s gestational week was not recorded, that is a missing observation of an applicable quantity.
How structural inapplicability is represented depends on the intended analysis and the variable definition. If the question concerns gestational week among pregnant participants, nonpregnant records are outside the variable’s applicable group. If the question instead concerns a measure defined for everyone, the variable may need to be defined differently. A structural absence should not be filled with zero unless zero is genuinely part of the revised variable’s meaning.
Worked Example: Separating Inapplicable and Unrecorded Gestational Weeks
A hypothetical file has six participants. Pregnancy status and gestational week are recorded as follows. The study defines gestational week only for participants who are pregnant.
| Participant | Pregnant? | Gestational week (weeks) |
|---|---|---|
| A | Yes | 20 |
| B | No | blank |
| C | Yes | 32 |
| D | No | blank |
| E | Yes | blank |
| F | Yes | 8 |
For participants B and D, gestational week is structurally inapplicable because they are not pregnant. Their blanks should not be counted as unrecorded gestational weeks among applicable cases, and they should not be changed to zero weeks. Participant E is pregnant, so gestational week is applicable; the blank means that this measurement was not recorded and is genuinely missing.
Among the six participants, four have an applicable gestational-week measurement, and one of those four has a missing value. Three have recorded weeks: \(20,32,\) and \(8\). Their arithmetic mean is \((20+32+8)/3=20\) weeks, a summary of the three recorded values only. It is not a gestational-week summary for the two nonpregnant participants, for whom the variable is undefined.
If the intended regression population is all six participants, the analyst must first define a meaningful outcome or predictor for all six; treating inapplicable gestational weeks as zero would answer a different question. The audit identifies this definition issue but does not choose the regression strategy.
Valid Zeros, Invalid Values, and a Justification
A numeric value of zero is not inherently a missing code. If a variable is cumulative exposure measured in pack-years, for example, a person who has never smoked can have a genuine value of zero under that definition. A blank or a documented sentinel for a current smoker whose exposure was not recorded is different: the exposure may be meaningful but unknown. Context and variable definition decide.
The arithmetic consequence of retaining a sentinel as though it were a real value can be seen directly. Suppose \(n\) observed values have mean \(\bar{x}\), and \(k\) missing records are incorrectly included with the same sentinel value \(c\). The resulting mean is
Subtracting the mean of the observed values gives
Thus, unless \(k=0\) or the code \(c\) happens to equal the observed mean, the reported mean changes. The same issue can affect regression inputs: the sentinel is treated by the software as a value, not as an instruction to ignore an unobserved measurement. This is why codebook-confirmed missing codes must be handled before analysis, while valid observations—including valid zeros—must be preserved.
Worked Example: A Zero Exposure and a Missing Exposure
A hypothetical data set records smoking status and lifetime exposure in pack-years. The variable is defined as the total smoking exposure; zero means no lifetime smoking. The codebook says \(-9\) means “not recorded.” Consider four records: never-smoker with \(0\), former smoker with \(12\), current smoker with \(-9\), and never-smoker with \(0\).
The two zeros are valid observed values because the variable definition assigns zero to no lifetime exposure. The value \(12\) is an observed exposure. The \(-9\) is a missing-value code according to the codebook and should be recoded as missing, not retained as negative pack-years.
There are three observed exposures, \(0,12,0\), with mean \((0+12+0)/3=4\) pack-years. If \(-9\) were mistakenly included as an exposure, the four-value mean would be \((0+12+0-9)/4=0.75\) pack-years. The difference illustrates why a confirmed sentinel must not be treated as a measurement and why legitimate zeros must not be removed merely because they seem unusual.
Common Mistakes and a Practical Record
- Automatically converting every blank to missing. A blank can mark structural inapplicability or a failed import. Check the field definition and source conventions.
- Assuming every unusual number is a sentinel. Confirm the code against the data dictionary or source. A surprising but valid observation should not be erased.
- Replacing inapplicable measurements with zero. Zero is a value with a meaning, not a universal substitute for “does not apply.”
- Converting impossible values to missing without review. First check for transcription, unit, and source-record errors. Document the decision if the value cannot be resolved.
- Overwriting the original column. Preserve raw entries and record the cleaned variable, reason, and rule used for each recode.
A compact audit record can include the variable name, original entry, classification, evidence for that classification, cleaned value, and action taken. For example, “age, \(-9\), documented missing code, recoded as missing” is more informative and reproducible than “cleaned unusual age.” Keep counts of valid observations, confirmed missing values, structurally inapplicable cases, and unresolved invalid entries separate. Those counts describe different situations and support later decisions about the analysis sample.
Check Your Understanding
For each question, explain what evidence you would use before deciding how to represent the value.
- A survey stores “999” for income. What should you check before treating it as a missing-value code?
- A variable records the number of prior pregnancies. Is zero automatically missing, structurally inapplicable, or a possible valid value? Explain what the variable definition must establish.
- A participant is not pregnant, and gestational week is blank. How does this differ from a pregnant participant whose gestational week is blank?
- A value falls outside the documented possible range, but no missing code is documented. What should happen before recoding it as missing?
- Why should the original imported values and the recoding rules be retained after cleaning?