Diagnosing Treatment Measurement Error

Why Diagnose Treatment Measurement?

Causal analyses rely on a well-defined treatment, but the treatment in a data set may be a record of what was assigned, prescribed, received, or remembered rather than an error-free measure of what occurred. A dose may be rounded, a treatment status may be entered incorrectly, or the recording process may change between clinics. These problems can make the measured treatment differ from the treatment relevant to the causal question.

Diagnosis asks how the recorded treatment relates to a credible measure of the treatment of interest. It does not, by itself, correct the data or identify a causal effect. The aim is to characterize the likely error mechanism, assess whether it varies across people or settings, and make clear what evidence supports that assessment.

The distinction between \(A\) and \(A^*\) is substantive, not merely notational. For example, “prescribed dose” and “dose taken” are different treatment definitions. A mismatch between those definitions is not automatically measurement error: it may instead mean the recorded variable does not represent the treatment the analysis intends to study. Diagnosis should therefore begin by stating what treatment is relevant, over what time period, and what the available record actually measures.

Patterns of Error and What They Mean

A useful first distinction is between random variation and systematic error. These are descriptions of patterns, not guarantees about the cause of a discrepancy. A record can contain both.

For a continuous treatment, a simple reference model is \(A^*=A+U\), where \(U\) is the measurement error. Calling this “classical” error requires specific assumptions, commonly that the error has mean zero and is independent of the true treatment and relevant characteristics. Those assumptions should not be inferred just because two measurements differ. A nonzero average discrepancy suggests a possible offset; discrepancies that grow with dose suggest a possible scale problem or changing precision. For binary treatments, error is instead summarized by the ways true treated and untreated statuses are recorded.

A diagnostic comparison needs a reference. A validation measure is often treated as the best available measure of \(A\), but it may itself be imperfect. Its timing, definition, and method of ascertainment must be compatible with the treatment of interest. If the reference measures a different treatment window, apparent disagreement can reflect a mismatch in definitions rather than faulty recording.

Replicate Measures: Agreement and Drift

When two or more measurements are available for the same person or treatment episode, compare them as paired observations. For two continuous measurements \(A_1^*\) and \(A_2^*\), define the paired difference \(D=A_2^*-A_1^*\). The mean of the differences describes average directional change between measurement occasions; the spread describes how much individual differences vary. Neither quantity alone establishes which measurement is closer to the treatment of interest.

A short derivation clarifies the interpretation. Suppose both measurements equal the same underlying treatment \(A\) plus occasion-specific errors: \(A_1^*=A+U_1\) and \(A_2^*=A+U_2\). Then

The common underlying treatment cancels only if it is unchanged between occasions and both measures refer to the same treatment definition. If the errors have means \(E(U_1)=\mu_1\) and \(E(U_2)=\mu_2\), then \(E(D)=\mu_2-\mu_1\). Thus a nonzero mean difference indicates a directional difference between occasions, but does not reveal which occasion is accurate. If the errors are independent with variances \(\sigma_1^2\) and \(\sigma_2^2\), then \(\operatorname{Var}(D)=\sigma_1^2+\sigma_2^2\). Without independence, the variance also depends on their covariance. In particular, similar replicate values can result from shared errors, not necessarily from accurate measurement.

With replicate binary measures, a cross-tabulation shows how often the records agree and in which direction they disagree. Agreement is a reproducibility measure: it describes whether the same method gives the same classification again. It is not an accuracy measure unless at least one measurement is compared with a credible reference. Replicates are most informative when their errors are not simply shared—for example, when they are collected independently and the recording process has not been copied from the first entry.

Validation Comparisons for Binary Treatment

For a binary treatment, suppose a validation sample has both a routine record \(A^*\) and a reference classification \(A\). The reference must define the relevant treatment and time window. The comparison can be summarized by a \(2\times2\) table. Sensitivity is the proportion of reference-treated people recorded as treated; specificity is the proportion of reference-untreated people recorded as untreated. These are descriptive properties of the measurement process in the validation sample, not causal effect estimates.

Also inspect whether discrepancies differ across relevant groups, such as sites, calendar periods, or treatment levels. For example, a site-specific comparison can reveal a data-entry rule that is invisible in a pooled table. Such stratification is diagnostic: it asks whether measurement performance changes across groups, rather than asserting that a difference is causal. Small validation samples can produce unstable subgroup summaries, so counts and uncertainty should be considered alongside percentages.

Heaping and Substantive Checks

Heaping occurs when recorded continuous values accumulate at preferred numbers, such as multiples of five or ten, rather than being spread as expected from the treatment process. It can arise from rounding, limited instrument precision, recall, or a data-entry convention. A frequency table, histogram, or count of values at preferred digits can reveal the pattern. The key question is whether those values are plausible features of treatment itself or artifacts of measurement.

Substantive knowledge gives these numerical checks meaning. Ask what values the treatment protocol permits, how doses are administered, whether treatment can change during the recorded interval, and who enters the data. A value outside the protocol’s plausible range may signal a unit conversion or data-entry problem, but it could also reflect an exceptional clinical event. Preserve the distinction between an observed anomaly and a confirmed error.

A Practical Diagnostic Sequence

Common Mistakes and Why Diagnosis Matters

• Treating agreement as accuracy. Two records can agree because they share a source or systematic error. Agreement establishes reproducibility only relative to the measurement process being repeated.
• Calling every discrepancy random. A consistent upward shift, site-specific coding rule, or concentration at rounded values may indicate systematic error. Inspect direction and context, not just the amount of disagreement.
• Assuming the reference is perfect. A validation measure can be mistimed, incomplete, or based on a different treatment definition. Describe why it is credible and acknowledge remaining uncertainty.
• Confusing a protocol feature with measurement error. Doses at multiples of ten may be genuine if the protocol uses those increments. Compare observed patterns with how treatment is actually assigned and delivered.
• Turning a diagnostic statistic into a causal conclusion. Sensitivity, specificity, replicate differences, and heaping summaries describe measurement. None alone establishes the direction or size of bias in a causal effect.

Treatment measurement affects what exposure groups the analysis is comparing. If errors vary by site, time, or participant characteristics, the measured treatment may not represent the same underlying treatment consistently across the study. Good diagnosis therefore informs whether a measurement model is plausible and what additional validation data are needed. It does not remove uncertainty: the conclusions should remain proportional to the quality and scope of the diagnostic evidence.