Auditing and Reporting an Identification Argument

Why an Identification Audit Matters

A reported treatment effect can look precise while relying on assumptions that are easy to overlook. An identification audit makes the argument visible: it names the information actually observed, defines the target quantity, states the assumptions that connect the two, and describes what uncertainty remains. This is especially important in a longitudinal trial, where outcomes scheduled after baseline may be missing for some participants.

The audit is not a claim that assumptions have been proved by fitting a model. It is a structured account of the route from observed data to the estimand. The earlier material on observable implications and partial identification distinguishes restrictions that the data can assess from assumptions that concern unobserved outcomes. Here the goal is to report that distinction clearly for a trial estimand.

A Basic Longitudinal Trial Setup

Consider a trial with treatment assignment \(A\in\{0,1\}\), baseline covariates \(X\), and a final scheduled outcome \(Y\). Let \(R=1\) indicate that the final outcome is observed and \(R=0\) that it is missing. Baseline covariates and treatment assignment are recorded for everyone. The observed data for each participant can therefore be written as \((X,A,R,RY)\), where \(RY\) denotes the outcome when observed and carries no outcome value when \(R=0\).

Let \(Y^a\) be the final outcome that would be seen under assignment \(a\). One common target is the intention-to-treat mean difference in the trial population:

This target compares outcomes under assignment, not necessarily outcomes under treatment received. To interpret it causally, the audit must state the relevant trial conditions. Random assignment makes \(A\) independent of the potential outcomes in the trial population; consistency connects the observed outcome under assigned treatment to the corresponding potential outcome; and no interference means one participant's assignment does not alter another participant's outcome. If these conditions fail, randomization alone does not justify the intended causal interpretation.

Missing outcomes create a separate identification issue. Randomization addresses the relation between assignment and potential outcomes; it does not, by itself, reveal the outcomes of participants with \(R=0\). For a simple identification argument, suppose the following conditions are invoked:

• Conditional missing at random: \(Y\) and \(R\) are conditionally independent given \(A,X\), written \(Y\perp R\mid A,X\).
• Response positivity: within covariate and treatment strata used for the target population, \(P(R=1\mid A=a,X=x)>0\).
• Treatment positivity: each treatment assignment has positive probability in the target population. In a randomized trial this is supported by the randomization design.
• Consistency and randomization: the assigned-arm observed outcomes represent the corresponding potential outcomes, subject to the trial conditions stated above.

Conditional missing at random is an identifying assumption, not a consequence of randomization. It says that among participants with the same treatment assignment and measured baseline covariates, the distribution of the final outcome is the same for responders and nonresponders. Since \(Y\) is unobserved when \(R=0\), this equality cannot be checked directly from the trial data.

From Observed Data to the Target

Under the assumptions above, the mean outcome in arm \(a\) can be obtained by first estimating the outcome mean among responders within each \(X\) stratum, then averaging those stratum means over the covariate distribution of the target trial population. The response indicator and covariates are observed, so the response probabilities and covariate distribution are part of the observed-data law. Conditional missing at random supplies the step that permits the responder outcome mean to stand in for the full stratum mean.

For continuous \(X\), the sum is replaced by an integral. In a randomized trial, the observed arm-specific conditional outcome distribution corresponds to the potential-outcome distribution under assignment \(a\), by randomization and consistency. Conditional missing at random then gives \(E[Y\mid A=a,X=x]=E[Y\mid A=a,X=x,R=1]\). Averaging over the target distribution of \(X\) yields the displayed expression. The contrast of the two arm-specific standardized means identifies \(\Delta\).

The positivity condition matters because a conditional responder mean cannot be learned from a stratum with no responders. Nor can the observed covariate strata represent the target population if some target strata are absent from the relevant treatment arm. A model may produce predictions in such empty regions, but those predictions depend on extrapolation rather than direct identification from observed support.

Audit Covariate Adjustment Explicitly

With covariates, a pooled responder mean generally does not implement the standardization formula. The target population's covariate distribution should be stated, and the arm-specific responder means should be averaged over that distribution. This avoids silently changing the target to the covariate distribution among responders, which can differ between arms when response rates depend on \(X\).

Report What Is Assumed and What Remains Uncertain

A complete audit separates four kinds of statements. First, describe what is observed, including which variables are recorded for everyone and which outcomes are observed only for some participants. Second, define the estimand and its population, treatment contrast, outcome, and time point. Third, state each assumption that bridges the observed-data law to that estimand. Fourth, report what remains uncertain: finite-sample error, modeling choices, weak support or positivity, measurement error, and sensitivity to untestable assumptions.

The distinction between identification and estimation is essential. If a target is identified under stated assumptions, its population value is determined by the observed-data distribution under those assumptions. A sample estimate can still be imprecise, and a fitted model can still be misspecified. Conversely, a narrow standard error does not resolve uncertainty about whether the identifying assumptions are credible.

Observed-data diagnostics can reveal problems without validating the missingness assumption. Useful checks include tabulating response rates by arm and baseline covariates, examining whether important target strata have responders, and checking model fit for observed outcomes and response indicators. Sparse strata, extreme fitted response probabilities, or marked arm differences in response can signal instability or extrapolation. Good fit among observed records cannot establish how outcomes would have been distributed among nonresponders.

When the key assumption cannot be empirically verified, report how conclusions change under alternatives. A simple sensitivity parameter can describe the difference between the missing and observed outcome means within each arm. This does not identify the missing outcomes from the data; it makes the additional assumption explicit and shows its consequences.

Common Audit Failures and Reporting Practice

A frequent mistake is to write, “The trial was randomized, so the treatment effect is identified,” even when final outcomes are missing. That reasoning confuses treatment assignment with outcome observation. Randomization supports comparability of treatment groups; it does not supply the unobserved outcomes. The correction is to state the missing-outcome assumption separately and show where it enters the identification argument.

A second mistake is to call a model-based result “identified” without naming its extrapolation assumptions. If some covariate-treatment strata contain no observed outcomes, a regression model may predict them, but the predictions depend on model form and cannot be checked against those absent outcomes. Report the support problem and distinguish model-based extrapolation from identification based on observed support.

For a concise report, give the audit in a fixed order:

The audit is useful even when the result is not point identified. It can show that the observed data restrict the target only to a range, or that a point estimate depends on a stated extrapolation. The appropriate conclusion is not that the analysis has failed, but that the strength of the claim should match the information and assumptions available. A transparent identification argument makes that match assessable.