Structural Missingness Versus Data-Collection Failure

Why the Reason for a Missing Value Matters

A blank in a data table does not explain itself. A value may be absent because the field does not apply to that record, or because a value that should have been collected was lost, never entered, or not transferred correctly. Those situations have different meanings. Treating them as the same kind of absence can inflate missingness rates, obscure a collection problem, or imply that a value ought to exist when it cannot.

For example, a driving-distance field may not apply to a policyholder who did not enroll in a driving-monitoring program. For an enrolled policyholder, however, a blank distance may signal a missed upload or an incomplete record. The first absence follows from the data design; the second suggests a collection or recording issue that needs investigation.

Definition: A value is structurally missing (or structurally not applicable) when the data design makes the variable undefined for that record. A value is missing because of data-collection or recording failure when the variable applies, but its value was not captured, recorded, or made available in the analysis data. An explicit status such as “device outage” is informative about why a value is unavailable, but it is not itself a blank value.

The distinction depends on what the field means and who is eligible to have a value—not just on whether an R column displays NA. If a data pipeline converts several statuses to NA, the resulting R object no longer tells you which reason applied. Check the data dictionary, eligibility rules, source-system codes, and collection process before collapsing those statuses.

Define Applicability Before Counting

Let \(A_i=1\) if variable \(X\) is defined for record \(i\), and \(A_i=0\) if it is structurally not applicable. Among records with \(A_i=1\), let \(C_i=1\) indicate that a usable value was captured and is available for analysis, and \(C_i=0\) indicate that it is unavailable. This notation separates the question “does the variable apply?” from the question “was its value obtained?”

\text{Eligible records}=\sum_{i=1}^{N} A_i,\qquad \text{unavailable among eligible records}=\sum_{i=1}^{N} A_i(1-C_i).

The collection-related unavailability rate should use the eligible records as its denominator:

\text{Unavailability rate among eligible records} = \frac{\sum_{i=1}^{N} A_i(1-C_i)}{\sum_{i=1}^{N} A_i}.

This rate may include several distinct statuses, such as a blank and a documented device outage. If those reasons are recorded separately, report them separately as well as, when useful, in a combined “no usable value” rate. Do not call that combined rate a blank rate.

Why the Overall Blank Rate Can Mislead

Suppose every structurally inapplicable value is stored as a blank, and some eligible values are also blank. The overall blank proportion then combines both groups. The identity below follows by partitioning blank records into structurally inapplicable records and eligible records with a blank value. It assumes there are no other kinds of blank records outside those two groups.

\frac{\text{all blank records}}{N} = \frac{\text{structurally inapplicable blank records}}{N} + \frac{\text{eligible records with a blank value}}{N}.

This overall proportion can describe how many rows are blank in the file, but it is not the collection-failure rate. To assess collection among records for which a value is defined, use the eligible denominator. Also keep blanks separate from nonblank status codes: a status code may mean “no measurement available,” but it is not a blank entry.

Establish the field’s meaning.
Write down what the variable measures and the circumstances under which it has a defined value.

Identify eligible records.
Use the data design or business rules to determine which rows should have a value. Do not define eligibility merely by whether a value happens to be present.

Separate absence statuses.
Count structural non-applicability, blank eligible values, and explicit unavailable-value codes separately when the data permit.

Reconcile the counts.
Check that the status counts sum to the number of records and calculate rates with clearly stated denominators.

Document the decision.
Record the evidence used to classify each status, including relevant eligibility rules and source-system definitions.

Worked Examples

Worked Example: Separating Eligibility, Blanks, and Outages

Method. Partition the records by program enrollment, then calculate rates using the appropriate denominator. In a hypothetical portfolio of 180 policies, 120 policyholders enrolled in a driving-monitoring program and 60 did not. Distance is defined only for enrolled policyholders. Among the 120 enrolled policies, 110 have recorded distances, 6 have an explicit “device outage” code, and 4 have a blank distance. Assume the 60 non-enrolled policies also have blank distances because distance was not collected for them. Find the eligible blank rate, the eligible rate with no usable distance, the all-policy blank proportion, and the all-policy proportion with no recorded distance.

Step 1: Check the enrolled-policy count. The enrolled statuses total \(110+6+4=120\), agreeing with the stated number eligible for a distance value. The non-enrolled count is 60, so the total is \(120+60=180\).

Step 2: Calculate the eligible blank rate. Only 4 of the 120 eligible policies have a blank distance. Thus the eligible blank rate is \(4/120=0.03333\ldots\), or 3.33% to two decimal places.

Step 3: Include documented outages in the eligible no-usable-value rate. There are \(6+4=10\) enrolled policies without a usable distance. The rate is \(10/120=0.08333\ldots\), or 8.33% to two decimal places. This is not the blank rate: six of those ten records have outage codes rather than blanks.

Step 4: Calculate the all-policy blank proportion. Under the stated assumption, the 60 non-enrolled policies and the 4 eligible policies have blank fields. The blank count is \(60+4=64\), so the all-policy blank proportion is \(64/180=0.35555\ldots\), or 35.56% to two decimal places.

Step 5: Calculate the all-policy proportion with no recorded distance. This includes the 60 non-enrolled policies, 4 eligible blanks, and 6 eligible outage codes: \(60+4+6=70\). Thus the proportion with no recorded distance is \(70/180=0.38888\ldots\), or 38.89% to two decimal places. The count of 70 is not the blank count.

Check and common mistake avoided. The 180 records partition into 110 recorded distances, 6 outage codes, and 64 blanks: \(110+6+64=180\). Calling \(70/180\) the all-policy blank proportion incorrectly treats the six outage codes as blanks. It is instead the proportion with no recorded distance, including documented outages.

Worked Example: A Field Defined Only After a Repair

Method. Apply the field’s eligibility rule before assessing capture. In a hypothetical file of 300 appliance claims, invoice amount is defined only for claims with a completed repair. There are 180 completed repairs and 120 claims without a completed repair. Among the 180 completed repairs, 140 have invoice amounts, 25 have a “vendor system unavailable” code, and 15 have a blank invoice amount. Calculate the structural non-applicability proportion, the eligible blank rate, and the eligible rate with no usable invoice amount.

Step 1: Identify the structural group. For the 120 claims without a completed repair, a repair invoice amount is undefined under the stated field definition. These records are structurally not applicable for this variable. Their proportion of all claims is \(120/300=0.40\), or 40.00%.

Step 2: Verify the eligible statuses. The completed-repair counts sum to \(140+25+15=180\), matching the eligible count.

Step 3: Calculate the eligible blank rate. There are 15 blanks among 180 completed repairs, so \(15/180=0.08333\ldots\), or 8.33% to two decimal places.

Step 4: Calculate the eligible rate with no usable invoice amount. The 15 blanks and 25 vendor-system statuses total 40 eligible claims with no usable amount. The rate is \(40/180=0.22222\ldots\), or 22.22% to two decimal places.

Step 5: Check the whole file. The status counts are \(140+25+15+120=300\). The 120 structurally inapplicable claims are not evidence of failed invoice capture. Conversely, the 25 system-unavailable statuses and 15 eligible blanks deserve separate collection-status counts.

Common mistake avoided. Dividing the 40 eligible claims with no usable amount by all 300 claims gives 13.33%, but that rate mixes in claims for which no invoice amount is defined. It does not measure unavailability among claims eligible for an invoice amount.

Worked Example: Comparing Collection Status Across Groups

Method. First exclude the structurally ineligible records from the field-specific denominator; then compare eligible status rates across the groups. In a hypothetical analysis of 200 applicants, a telematics-trip measure applies only to applicants who selected an optional monitoring rider. Eighty did not select it. Among the 120 rider selections, 70 are from urban applicants and 50 from rural applicants. Urban statuses are 63 recorded values, 4 outage codes, and 3 blanks. Rural statuses are 42 recorded values, 5 outage codes, and 3 blanks. Calculate each group’s eligible no-usable-value rate and the combined eligible rate.

Step 1: Verify group counts. Urban statuses total \(63+4+3=70\); rural statuses total \(42+5+3=50\). Together they account for \(70+50=120\) rider selections. Adding the 80 non-selections gives all \(120+80=200\) applicants.

Step 2: Calculate the urban rate. Urban applicants have \(4+3=7\) eligible records with no usable value. The rate is \(7/70=0.10\), or 10.00% to two decimal places.

Step 3: Calculate the rural rate. Rural applicants have \(5+3=8\) eligible records with no usable value. The rate is \(8/50=0.16\), or 16.00% to two decimal places.

Step 4: Calculate the combined eligible rate. There are \(7+8=15\) eligible records with no usable value among 120 rider selections. The combined rate is \(15/120=0.125\), or 12.50% to two decimal places.

Step 5: Interpret with care. The rates describe unavailable trip measures among rider selections. The 80 non-selections are structurally inapplicable for this measure, not additional collection failures. The difference between the urban and rural rates identifies a pattern to investigate; by itself, it does not establish why collection statuses differ.

Common mistake avoided. Counting the 80 non-selections as collection failures would make the overall unavailable count 95 and the rate \(95/200=47.50\%\). That number mixes non-applicability with missing capture and answers a different question.

Cleaning Decisions and Data Checks

Classification is a data-quality decision, not a cosmetic relabeling. A useful audit preserves enough information to distinguish at least: a valid observed value, structural non-applicability, an eligible blank, and each meaningful explicit status code. If an import or data-preparation step has already combined these statuses into NA, look for a source flag or recover the information from the system that produced the file before deciding that all such records have the same explanation.

In R, is.na() can count values represented as NA, but it cannot determine why a value is absent. Likewise, a text code such as "device outage" is not an R missing value unless it has been converted to one. Maintain a separate status or eligibility field when possible, and check that the observed value and status are logically consistent. For example, a record labeled “not enrolled” should not ordinarily have a measured program distance; if it does, the enrollment rule, field definition, or record may need review.

Structural absence should not automatically be replaced with a numeric value such as zero. Zero means an observed quantity equals zero; it does not mean “this quantity has no definition for this record.” Nor should an explicit reason code be silently treated as a valid measurement. The appropriate representation depends on the variable’s definition and the planned analysis. The immediate cleaning task is to retain the distinctions and describe them accurately, not to assume that every absence can be handled identically.

Key takeaway: Ask two separate questions for every absent value: “Is this field defined for this record?” and, if it is, “Was a usable value captured?” Count and report structural non-applicability separately from collection-related unavailability, and name the denominator used for every rate.

Common Exam Traps

Using the whole data set as the denominator automatically. A field-specific collection rate generally uses the records for which the field is defined. State the target denominator explicitly.
Calling every unavailable value a blank. A blank and an outage code may both lack a usable measurement, but they are different recorded statuses. Report a combined rate only with a label that makes the combination clear.
Inferring structure from the pattern alone. If a subgroup has many blanks, that is not enough to prove the field is structurally undefined there. Confirm the field definition and eligibility rules.
Turning “not applicable” into zero. Unless zero is the actual measured value, this changes the meaning of the data.
Assuming a status code settles the cause. A code is evidence only to the extent that its definition and use in the source system are understood. Validate it against the collection process.

Check Your Understanding

For each problem, classify the statuses before calculating. Rates should be reported to two decimal places.

A hypothetical file has 240 accounts. A monthly usage field applies only to 150 active accounts; 90 inactive accounts have blank usage. Among active accounts, 132 values are recorded and 18 are blank. What are the active-account blank rate and the all-account blank proportion? (Answer: \(18/150=12.00\%\); \((90+18)/240=45.00\%\).)
A hypothetical repair file has 160 eligible jobs. It contains 118 recorded labor times, 12 “clock system offline” codes, and 30 blanks. What is the blank rate and what is the rate with no usable labor time? (Answer: \(30/160=18.75\%\) blank; \((12+30)/160=26.25\%\) with no usable time.)
A field is defined only for policies with a roadside-assistance add-on. Of 400 policies, 100 have the add-on; among them, 92 have a recorded call count and 8 are blank. The other 300 have no add-on and are blank. Calculate the eligible blank rate and the all-policy blank proportion. (Answer: \(8/100=8.00\%\); \((300+8)/400=77.00\%\).)
A hypothetical file has 90 eligible records in Group A, with 81 values recorded and 9 outage codes; and 60 eligible records in Group B, with 51 values recorded, 6 outage codes, and 3 blanks. Find the no-usable-value rate for each group and overall. (Answer: Group A \(9/90=10.00\%\); Group B \((6+3)/60=15.00\%\); overall \((9+9)/150=12.00\%\).)
A data set has 500 rows. A field applies to 320 rows. Of these, 280 values are recorded, 24 have a documented equipment-failure code, and 16 are blank. The remaining rows are structurally inapplicable and stored as blanks. Find the eligible blank rate, eligible rate with no usable value, all-row blank proportion, and all-row proportion with no recorded value. (Answer: \(16/320=5.00\%\); \((24+16)/320=12.50\%\); \((180+16)/500=39.20\%\); \((180+24+16)/500=44.00\%\).)

What You'll Learn