A Workflow for Causal Analyses With Imperfect Data

Why Imperfect Data Require a Workflow

A causal analysis can be weakened at several points at once. Some outcomes may be unobserved, the recorded treatment may differ from the treatment received, and a confounder may be measured only through an imperfect proxy. A correction that addresses one problem does not automatically address the others. For example, weighting for outcome response cannot repair treatment misclassification, and correcting treatment labels does not remove confounding by a poorly measured covariate.

A disciplined workflow starts by defining the causal quantity of interest, then identifying how each variable was generated and recorded. The analysis must state which assumptions make the desired quantity identifiable, choose corrections that match those assumptions, and examine how conclusions change when uncertain assumptions are varied. The final report should distinguish uncertainty due to sampling from uncertainty that remains because the assumptions cannot be checked fully.

Definition: Let $A$ denote the true treatment, $Y$ the outcome, and $Y(a)$ the potential outcome that would be observed under treatment level $a$, for $a\in\{0,1\}$. For a binary outcome, the average causal risk difference is $$ \psi=E[Y(1)]-E[Y(0)]. $$ Let $W$ be the recorded treatment, $X$ a true confounder, $Z$ an imperfect measurement or proxy for $X$, and $R$ indicate whether $Y$ is observed: $R=1$ means observed and $R=0$ means missing. These variables have different roles: $A$ and $X$ describe the causal process, while $W$, $Z$, and $R$ describe limitations in the data.

Specify the Target and the Assumptions

For the risk difference above, the target population must be specified: the expectation is over the population to which the analysis is intended to apply. This tutorial focuses on estimating an effect in the study population, not on transporting it to another population. Define the outcome time point and the treatment strategies precisely enough that each potential outcome has a clear meaning.

A common identification argument uses consistency, no interference, conditional exchangeability, and positivity. Consistency says that a person's observed outcome under the treatment actually received equals the corresponding potential outcome. No interference says one person's treatment does not change another person's outcome. Conditional exchangeability says that, given the true confounders $X$ and any other required covariates $C$, treatment assignment is independent of the potential outcomes. Positivity requires a positive probability of each treatment level at every covariate pattern in the target population. Under these assumptions, the causal risks can be obtained by standardizing conditional outcome risks:

E[Y(a)]=E_{X,C}\{E[Y\mid A=a,X,C]\}.

With imperfect data, this expression is not yet an estimator. The analyst must also address how $Y$, $A$, and $X$ are observed. For instance, a missing outcome assumption might be that response is independent of $Y$, conditional on treatment and covariates. A treatment correction might require known or estimable sensitivity and specificity. A proxy for a confounder requires a measurement model or other information that links it to the true confounder. Without sufficient information or defensible assumptions, the causal effect may not be point-identified; a sensitivity analysis or bounds may be more honest than a single corrected estimate.

Define the estimand.
Specify the treatment contrast, outcome, time point, effect scale, and target population before inspecting which correction gives a preferred result.

Map the data problems.
Identify which outcomes are missing, how treatment was classified, which confounders are measured directly, and which are represented by proxies.

State identification assumptions.
Separate causal assumptions, such as exchangeability, from measurement and missingness assumptions, such as known classification accuracy or missing at random.

Choose compatible corrections.
Use validation data, response models, measurement models, or sensitivity parameters as appropriate. When several processes interact, account for them jointly rather than treating each correction as automatically independent.

Assess uncertainty and report limits.
Quantify sampling uncertainty, vary important unverified assumptions, and state what remains unidentified or dependent on modeling choices.

Worked Examples of Individual Data Problems

Worked Example: Missing Outcomes Under a Response Assumption

Suppose a hypothetical randomized study assigns 100 people to each of two treatments. Outcomes are observed for 80 people assigned $A=1$, of whom 24 have the event, and for 60 people assigned $A=0$, of whom 15 have the event. Assume consistency, no interference, and that, within each assigned arm, outcome observation is independent of the outcome. Also assume every person in each arm has a positive chance of response. This is a missing-at-random assumption conditional on treatment in this example.

The observed event risks are $24/80=0.30$ and $15/60=0.25$. Under the stated response assumption, the observed responders represent their respective treatment arms for estimating each arm's risk. Equivalently, inverse-response weighting gives estimated event totals $24/(80/100)=30$ and $15/(60/100)=25$, with estimated arm totals $80/(80/100)=100$ and $60/(60/100)=100$. The estimated risks are therefore $30/100=0.30$ and $25/100=0.25$, giving

\widehat{\psi}=0.30-0.25=0.05.

The estimated risk difference is 0.05, or 5 percentage points, in favor of a higher event risk under $A=1$. It is not evidence that the missingness assumption is true. If people with events were less likely to respond even after conditioning on treatment, the observed risks could be biased. The randomized assignment supports treatment exchangeability, but it does not by itself make outcome nonresponse ignorable.

Worked Example: Correcting a Misclassified Binary Treatment

Consider a hypothetical randomized study with 2,000 participants, 1,000 truly assigned to each treatment. Suppose recorded treatment $W$ has sensitivity 0.90 and specificity 0.90 for the true assignment $A$. Assume these accuracy values apply to the study, and that classification is independent of the outcome conditional on true treatment. The hypothetical event counts are 400 among the 1,000 truly treated and 200 among the 1,000 controls.

Among 2,000 participants total, the expected counts with $W=1$ are 900 truly treated and 100 truly untreated. Thus there are 1,000 people with $W=1$, including $900(0.40)+100(0.20)=380$ events. The recorded-treatment risk is $380/1{,}000=0.38$. The expected counts with $W=0$ are 100 truly treated and 900 truly untreated, again 1,000 people; their event count is $100(0.40)+900(0.20)=220$, giving a recorded-treatment risk of $220/1{,}000=0.22$. The uncorrected risk difference is $0.38-0.22=0.16$, smaller than the true-assignment contrast in this example.

The correction can be shown by inverting the classification matrix. The columns correspond to $A=1,0$, and the rows to $W=1,0$:

\begin{pmatrix} N_{W=1}\\ N_{W=0} \end{pmatrix} = \begin{pmatrix} 0.90&0.10\\ 0.10&0.90 \end{pmatrix} \begin{pmatrix} N_{A=1}\\ N_{A=0} \end{pmatrix}.

Apply the same relationship separately to event counts. In this example the observed event counts are 380 and 220. Solving $0.90T+0.10C=380$ and $0.10T+0.90C=220$ gives $T=400$ and $C=200$, where $T$ and $C$ are the corrected event counts for true treatment and control. Inverting the group-size counts gives 1,000 participants in each true arm. The corrected risks are $400/1{,}000=0.40$ and $200/1{,}000=0.20$, so the corrected risk difference is $0.20$, or 20 percentage points. This calculation relies on the stated accuracy and nondifferential-classification assumptions; it is not a universal correction when accuracy varies with outcome or covariates.

Worked Example: A Noisy Confounder Proxy Can Leave Bias

In a hypothetical observational study, let $X$ be a binary confounder with prevalence 0.50. Treatment is received with probability 0.80 when $X=1$ and 0.20 when $X=0$. The outcome risks under control are 0.50 for $X=1$ and 0.10 for $X=0$; under treatment they are 0.60 and 0.20, respectively. Assume consistency, no interference, positivity, and exchangeability conditional on the true $X$. The causal risk under treatment is $0.5(0.60)+0.5(0.20)=0.40$; under control it is $0.5(0.50)+0.5(0.10)=0.30$. Thus adjustment for true $X$ gives a causal risk difference of $0.10$.

Suppose instead that only a proxy $Z$ is available, with sensitivity and specificity both 0.80 for $X$, and that proxy errors are independent of treatment and outcome conditional on $X$. In the $Z=1$ stratum, the population masses with $X=1$ and $X=0$ are $0.40$ and $0.10$. The treated denominator is $0.40(0.80)+0.10(0.20)=0.34$, and its event mass is $0.40(0.80)(0.60)+0.10(0.20)(0.20)=0.196$. The treated risk is $0.196/0.34=0.5765$. The control denominator is $0.40(0.20)+0.10(0.80)=0.16$, and its event mass is $0.40(0.20)(0.50)+0.10(0.80)(0.10)=0.048$, giving risk $0.3000$.

For $Z=0$, the masses with $X=1$ and $X=0$ are $0.10$ and $0.40$. The treated denominator is $0.10(0.80)+0.40(0.20)=0.16$; the event mass is $0.10(0.80)(0.60)+0.40(0.20)(0.20)=0.064$, giving risk $0.4000$. The control denominator is $0.10(0.20)+0.40(0.80)=0.34$; the event mass is $0.10(0.20)(0.50)+0.40(0.80)(0.10)=0.042$, giving risk $0.1235$. Giving the two equally prevalent proxy strata equal weight yields an adjusted contrast of

\frac{(0.5765-0.3000)+(0.4000-0.1235)}{2}=0.2765.

This is far from the true-$X$ adjusted contrast of 0.10. The proxy is informative about $X$, but it does not fully control the association between treatment and the true confounder. A proxy-adjusted estimate is not automatically a corrected causal estimate; a measurement model, validation information, or a sensitivity analysis for residual confounding is needed.

Combining the Problems in One Analysis

Consider a hypothetical observational cohort in which the event outcome is missing for some participants, treatment is recorded as $W$ rather than known true treatment $A$, and a clinically important confounder $X$ is measured only by proxy $Z$. Each problem can affect the others. For example, the probability of outcome observation may differ by true treatment or confounder even after conditioning on the recorded variables. A correction for missingness based only on $W$ and $Z$ can therefore fail if the response model omits important predictors that are latent.

A defensible analysis would first define the target risk difference and describe which treatment, confounder, and outcome values are observed. It would then use validation data, if available, to estimate treatment-classification accuracy and the relationship between $Z$ and $X$. It would model outcome response using observed predictors, while stating whether the working assumption is missing at random conditional on those predictors. If that assumption is doubtful, it would vary the outcome distribution among nonresponders using a prespecified sensitivity parameter or report bounds. A joint likelihood, imputation, or other coherent analysis can represent the latent treatment and confounder together with outcome response; separate corrections should not be stacked unless their assumptions and dependencies justify doing so.

The following distinctions help prevent overstatement:

Sampling uncertainty concerns variation from observing a finite sample. It can be quantified with a confidence interval or a suitable resampling procedure.
Measurement uncertainty concerns uncertain treatment accuracy or proxy calibration. If those quantities are estimated from a validation sample, their estimation uncertainty should be carried through the causal analysis.
Identification uncertainty concerns assumptions not determined by the observed data, such as departures from missing at random or residual confounding after proxy adjustment. A narrow confidence interval under one model does not remove this uncertainty.

Worked Example: A Missingness Sensitivity Analysis

Return to the hypothetical randomized study with 100 people per arm: 24 events among 80 observed outcomes under $A=1$, and 15 among 60 observed outcomes under $A=0$. The missing-at-random estimate was a risk difference of 0.05. Now let $\delta_1$ and $\delta_0$ denote additive differences between the event risk among missing participants and the observed event risk in each arm. For this illustration, restrict the resulting risks to the interval from 0 to 1.

The estimated event risk in the full $A=1$ arm is $0.80(0.30)+0.20(0.30+\delta_1)=0.30+0.20\delta_1$. In the $A=0$ arm it is $0.60(0.25)+0.40(0.25+\delta_0)=0.25+0.40\delta_0$. Therefore the risk difference under these sensitivity assumptions is

\widehat{\psi}(\delta_1,\delta_0)=0.05+0.20\delta_1-0.40\delta_0.

If missing participants in the $A=1$ arm have an event risk 0.10 higher than responders and the $A=0$ missing risk equals its observed risk, then $(\delta_1,\delta_0)=(0.10,0)$, and the estimated difference is $0.05+0.20(0.10)=0.07$. If $\delta_1=0$, the estimate reaches zero when $0.05-0.40\delta_0=0$, or $\delta_0=0.125$. Thus, under this simple sensitivity model, an event risk 12.5 percentage points higher among missing controls than observed controls would bring the estimated contrast to zero. These are sensitivity calculations, not data-based estimates of the missing outcomes.

Common Mistakes and Reporting

Applying a correction as if it fixed every problem. Response weighting addresses outcome observation under its response assumptions; it does not correct a misclassified treatment or an uncontrolled confounder.
Adjusting for a proxy as if it were the true confounder. A noisy proxy may leave residual confounding, as the worked example shows. Explain what validation or measurement assumptions support any proxy-based correction.
Treating estimated accuracy as known. Sensitivity and specificity from a small validation sample are uncertain. Propagate that uncertainty rather than using the point estimates as exact constants.
Calling a sensitivity scenario a corrected truth. A delta analysis describes how the estimate changes under specified departures from missing at random; it does not reveal which departure actually occurred.
Reporting only a model-based confidence interval. State the assumptions behind the model and show how important conclusions change across plausible alternatives. Sampling precision conditional on a model is not proof that the model is correct.

A clear report names the estimand and effect scale, describes the missingness and measurement processes, states the assumptions supporting identification, and explains each correction. It presents the principal estimate with its sampling uncertainty, then summarizes sensitivity analyses for assumptions that the observed data cannot verify. If plausible analyses lead to materially different conclusions, that variation is part of the result rather than a nuisance to conceal.

Key takeaway: Imperfect-data causal analysis is a sequence of linked decisions: define the target, map the data limitations, state assumptions, choose compatible corrections, and assess sensitivity. A correction is credible only to the extent that its assumptions and uncertainty are made explicit.

Check Your Understanding

Use the workflow to answer these questions.

In a study with treatment misclassification and missing outcomes, why does correcting the recorded treatment alone not address possible bias from nonresponse?
What response assumption supports estimating an arm-specific risk from responders alone, and what important concern does randomization leave unresolved?
Why can adjusting for a proxy of a confounder leave a causal contrast biased even when the proxy is informative?
In the missingness sensitivity example, what does the value $\delta_0=0.125$ represent, and what does it not establish?
Name one source of sampling uncertainty and one source of identification uncertainty in an analysis that uses validation data and a missing-at-random assumption.