Why Missing-Value Encodings Need Explicit Attention
A data file can contain values that look like observations but were entered to mean “not recorded.” Examples include a numeric value such as -99, a category such as “Unknown,” or a code used by a particular file format to represent missingness. If these values enter a MICE workflow as ordinary data, software may treat them as genuine measurements or categories. That can distort summaries and the conditional models used in imputation.
The task here is narrower than general data cleaning: identify the missing-value encodings documented for an input dataset, convert those encodings explicitly, and verify the conversion with counts. A value that seems unusual is not, by itself, evidence that it means missing. The interpretation must come from the data dictionary, source-system documentation, file-format documentation, or a confirmed clarification from the data provider. The variable dictionary established for a MICE workflow is a useful place to record the variable-specific meaning.
These categories should not be collapsed casually. A sentinel may mean “not applicable,” “not asked,” “refused,” or “below detection limit,” rather than simply “unknown.” Those states can have different analytical meanings. This tutorial concerns codes documented as missing values; other special states should be retained and interpreted according to the analysis plan, not automatically recoded to missing.
Conditions for a Safe Conversion
Before recoding, establish four facts: which variable is affected; the exact code or codes; the meaning of each code; and whether each code can also be a valid observation. The code must be specific to the variable and the source representation. For example, -99 may mark missing blood pressure in one column while being a legitimate value in another variable. A blanket replacement across every numeric column is therefore unsafe.
Check the permitted domain and units as a supporting diagnostic, not as a substitute for documentation. If age is recorded in years and the source specifies 999 as “not reported,” the value is outside a plausible age range and the documentation supports recoding. If a value lies within the variable’s valid domain, do not infer that it is missing merely because it is rare.
The input representation also matters. A native missing marker may be recognized by the import software, while an export may turn it into a character string. A string such as “.” or “NA” could be a missing marker in one file and literal text in another. Inspect the imported values and data types, and consult the source format’s rules before deciding what to convert. Preserving the original input and documenting its provenance makes it possible to trace the conversion; the analysis copy should not silently overwrite the raw values.
A Count-Based Verification Rule
Let \(N\) be the number of rows, \(M_0\) the number already represented as missing in a variable before conversion, and \(S\) the number of rows containing documented sentinel codes in that variable. If the sentinel codes are disjoint from the already-missing values, and the conversion changes only those sentinels into missing values, then:
Here \(M_1\) and \(O_1\) are the missing and observed counts after conversion. The row count remains \(N\). The first equality follows because each of the \(S\) sentinel-coded rows becomes missing, while the already-missing rows remain missing. The second follows because exactly those \(S\) rows leave the observed set. Adding the two post-conversion counts gives \(M_0+S+O_0-S=M_0+O_0=N\), so the total is conserved.
This reconciliation is necessary but not sufficient: a wrong conversion can still preserve the total row count. Verify both the counts and the actual values—especially the frequencies of each code before conversion and whether any valid values changed. For multiple sentinel codes, \(S\) is the number of rows containing any of the documented codes, not the sum of code counts if a row could be counted more than once.
Worked Recoding and Verification
Worked Example: Two Numeric Sentinels in a Clinical Measure
Suppose a hypothetical input file contains 12 records for a laboratory measure in mmol/L. Its source documentation states that -99 means “not measured” and 999 means “not reported.” The valid measurement range for this field is 0.0 to 30.0 mmol/L. Before conversion, 2 records are already explicitly missing. Among the other 10 records, the values are: 4.8, 5.1, -99, 6.0, 999, 5.5, 4.9, 5.2, 6.1, 5.7.
The codes are documented for this variable and both fall outside its permitted measurement range. The pre-conversion frequency is one -99 and one 999, so \(S=2\). Convert each documented code to explicit missing, leaving all other values unchanged.
Before: \(N=12\), \(M_0=2\), and \(O_0=10\). The total is \(2+10=12\).
After: \(M_1=M_0+S=2+2=4\) records are missing, and \(O_1=O_0-S=10-2=8\) records are observed. Check: \(4+8=12\). The observed values are 4.8, 5.1, 6.0, 5.5, 4.9, 5.2, 6.1, and 5.7 mmol/L. Neither the row count nor any valid measurement changes.
A complete record of the operation would state the variable name, both source codes and their meanings, the explicit missing representation used, and the counts before and after. Reporting only “cleaned the data” would not permit a reviewer to check what happened.
Worked Example: Categorical Codes With Different Meanings
In a hypothetical survey file of 10 participants, the variable “smoking status” is documented as containing “Current,” “Former,” and “Never” as valid categories. The data provider says “NR” means “no response” and is missing, while “Not applicable” means the question was intentionally skipped under the survey design. The imported values are Current, Former, NR, Never, Current, Not applicable, Never, NR, Former, and Current.
There are 10 rows and no explicitly missing values in this column before conversion. “NR” occurs twice. The instructions define “Not applicable” as a distinct survey state, not as an unrecorded response. Therefore, convert the two “NR” values to explicit missing and retain “Not applicable” as its own category, unless the analysis plan separately specifies how to handle that state.
Before: \(N=10\), \(M_0=0\), \(O_0=10\), and the frequency of NR is 2.
After: \(M_1=0+2=2\), \(O_1=10-2=8\), and \(2+8=10\). The eight observed entries comprise three Current, two Former, two Never, and one Not applicable. The distinction in the documentation prevents an overbroad conversion of both special-looking strings.
Worked Example: Checking a Software-Code Import
A hypothetical data export contains 15 records for a score that is documented to range from 0 to 100. The source system uses the text token “.M” for missing scores. On import, the column is character-valued because the file contains both numbers and “.M.” The imported column has 2 blank entries, 3 occurrences of “.M,” and 10 numeric strings between 0 and 100.
The import behavior must be checked before recoding. Since “.M” is documented by the source system as a missing marker, convert that exact token to the analysis software’s explicit missing value, and parse the documented numeric entries as numbers. Do not replace every period-containing string or every nonnumeric value: those rules could change other legitimate text if the column’s contents or format differ.
Before: \(N=15\), with 2 explicit blank missings, 3 “.M” codes, and 10 observed scores. Thus \(M_0=2\), \(S=3\), and \(O_0=13\), counting the three codes as not yet explicitly missing.
After: \(M_1=2+3=5\) and \(O_1=13-3=10\). Check: \(5+10=15\). Confirm that the 10 scores parse within 0–100 and that the resulting missing count is 5. If the importer had already recognized “.M” as missing, the post-import missing count would already include those three records; recoding them a second time would not increase the count. Inspect the imported values and counts to distinguish these cases.
Practical Procedure and Diagnostics
A defensible conversion can be organized as a short sequence. Count the codes before applying any replacement, then repeat the counts after conversion. Where possible, compare the non-sentinel values before and after as well. A row-total check detects dropped or duplicated records, while variable-level checks detect whether the intended code was converted and ordinary values were preserved.
Record the variable, exact token or value, meaning, valid domain, and relevant file-format behavior. Resolve ambiguous meanings with the data provider rather than guessing from the value alone.
Check the column’s type, frequencies of candidate codes, existing missing count, and a sample of values. Determine whether the importer has already recognized a software-specific code as missing.
Apply a variable-specific mapping to the exact codes designated as missing. Preserve other special states and all valid values, and retain the unchanged source data separately.
Compare pre- and post-conversion code frequencies, missing and observed counts, and row totals. Document the rule and the resulting counts so the conversion can be reviewed.
Investigate if the missing count does not rise by the number of sentinel-coded rows that were previously treated as observed; if the row total changes; if a documented code remains among ordinary values; or if valid observations have disappeared. A mismatch may indicate that the importer already converted a code, a replacement targeted the wrong column, a token was read differently than expected, or some rows were filtered or duplicated. Counts localize the problem but cannot establish whether the source documentation itself is correct.
Alternatives, Limitations, and a Common Mistake
The preferred approach depends on where the missingness is encoded. If the source file format defines a native missing marker and the import software reads it correctly, allowing the importer to preserve that explicit missing value may be appropriate; still verify the resulting counts. Explicit recoding is needed when a documented sentinel arrives as an ordinary number or string. Leaving a sentinel unchanged is not a safe alternative for an analysis that will treat the field as measured data. Conversely, a global rule such as “replace all negative numbers with missing” is not a safe substitute for documented, variable-specific mappings.
A count check cannot prove that a code was correctly classified. If -99 could be a valid value under a different unit convention, or “Unknown” is an actual response category, the conversion requires clarification. Recoding can also erase meaningful distinctions if multiple states are combined into one missing category. Record such distinctions before deciding which states belong in the analysis as missing.
Common mistake: “The value is impossible for this variable, so it must be missing; replace every such value.” This reasoning relies on plausibility alone and may conceal valid extreme observations, unit errors, or data-entry errors that need separate investigation. The correction is to use source documentation to establish the code’s meaning, check its scope and representation, convert only the specified values, and verify the change with frequencies and before-and-after counts.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A variable has 40 rows, 4 explicit missing values, and 6 documented sentinel values currently counted as observed. What are the expected missing and observed counts after conversion?
- A variable’s documentation says -9 means “not asked,” but does not say that this state is missing. Should -9 automatically be recoded to explicit missing? Explain.
- A categorical variable has “NR” documented as missing and “Unknown” documented as a valid response. What should be converted if the data contain both?
- After a recode, the row count is unchanged and the missing count increased by the expected amount. Is that enough to establish that the conversion is correct? Name another check.
Solutions. (1) \(M_1=4+6=10\) missing and \(O_1=(40-4)-6=30\) observed; \(10+30=40\). (2) No. “Not asked” may be analytically distinct from an unobserved response; the intended treatment must be established before converting. (3) Convert only NR to explicit missing and retain Unknown as a valid response category, consistent with the documentation. (4) No. Also verify the pre-conversion frequency of the exact code, confirm it no longer appears as an ordinary value, and check that valid values were preserved.