Why Missing Covariates Matter in External Controls
An external-control comparison brings together people observed under different data-collection systems. A trial may collect a baseline covariate using a required protocol assessment, while a nontrial source records it only when clinically indicated. Consequently, the percentage of missing values can differ by source, and the missingness can reflect different processes. Ignoring that difference may change which patients contribute to an analysis or how their covariates are represented.
The issue is not resolved merely by choosing a familiar missing-data procedure. Complete-case analysis, imputation, and missingness indicators answer different practical needs and rely on different assumptions. The relevant question is whether the resulting analysis still represents the chosen estimand, with uncertainty that accounts for the missing values and the data source.
Missingness Assumptions and Their Limits
A missingness assumption describes how the observation indicator relates to observed and unobserved data. It is not a claim about why a particular value is absent in a clinical record; it is a condition under which an analysis can use the observed data to learn about the target quantity.
- Missing completely at random (MCAR): missingness is independent of the covariate values and other data, possibly considered within a defined source. Under MCAR, complete cases are a random subset with respect to those data.
- Missing at random (MAR): conditional on observed information included in the analysis, the probability that \(X\) is missing does not depend on its unobserved value. For example, missingness may depend on source, observed age, and outcome, while not depending further on the missing value of \(X\).
- Missing not at random (MNAR): even after conditioning on the observed information, missingness depends on the unobserved value. For example, an unusually high laboratory value may be more likely to be absent even among patients with the same observed characteristics.
These definitions are relative to the information included in the analysis. If missingness depends on whether a patient is in the trial or external cohort, omitting \(S\) from an imputation or missingness model can make an otherwise plausible MAR description inadequate. Conversely, including source does not make MAR true if the unobserved covariate value still predicts its own missingness after conditioning on the available information.
A key practical condition is observation positivity: within relevant combinations of source and observed characteristics, there must be a nonzero chance of observing the covariate. If a source never records a covariate for a particular patient group, observed data from that group cannot directly identify the covariate distribution there. An imputation model can still produce values, but those values depend on extrapolation and model assumptions, not information supplied by observed measurements in that group.
Worked Example: How Complete Cases Can Change the Cohort Mix
Suppose two hypothetical cohorts each contain 100 patients: 50 with high baseline severity and 50 with low severity. In the trial, severity is recorded for 45 of 50 high-severity patients and 45 of 50 low-severity patients. In the external cohort, it is recorded for 20 of 50 high-severity patients and 40 of 50 low-severity patients. Compare the severity distribution among complete cases.
For the trial, the observed high-severity fraction is \(45/(45+45)=45/90=0.500\), or 50.0%. For the external cohort, it is \(20/(20+40)=20/60=0.333\), or 33.3%.
The full cohorts each have 50.0% high-severity patients, but the complete-case subsets do not. In the external cohort, high-severity patients are less likely to have severity recorded: \(20/50=40.0\%\), compared with \(40/50=80.0\%\) for low-severity patients. A comparison restricted to complete cases therefore uses an external subset with a different severity mix. This calculation does not establish the direction or size of any treatment-effect bias; that also depends on how severity relates to the outcome and the chosen estimand.
Complete-Case Analysis
Complete-case analysis discards any participant missing one or more covariates required by the analysis, then analyzes the remaining participants. It is straightforward and may be reasonable when the loss is small and the selection into the complete-case subset does not distort the quantity being estimated. But MCAR is a strong sufficient condition, not a universal requirement. In some models, complete-case estimates can remain valid under less restrictive conditions; validity depends on the estimand, model, and missingness process.
For an external-control analysis, checking the missing fraction overall is not enough. Complete-case selection can differ between \(S=1\) and \(S=0\), and within each source it can depend on observed or unobserved patient characteristics. Even if a model estimates a conditional relationship correctly in the retained sample, the retained covariate distribution may not represent the target population. Thus, a complete-case result may describe the selected patients rather than the population defined by the estimand.
Complete-case analysis can be attractive when the number of missing records is very small, the retained sample remains adequate, and source-specific comparisons show no important evidence of selection on relevant observed factors. It is less defensible when missingness is substantial, strongly imbalanced by source, or concentrated in clinically important subgroups. Reporting only the complete-case result hides those distinctions.
Imputation: Filling in Missing Values While Preserving Uncertainty
Imputation replaces missing values with values drawn or predicted from a model for the covariate conditional on observed information. In multiple imputation, the process is repeated to create \(m\) completed data sets, each reflecting uncertainty about the missing values. The same prespecified analysis is applied to each completed data set, and the estimates and their uncertainty are combined. A single deterministic fill-in, such as replacing every missing value by the mean, is not multiple imputation: it treats uncertain values as known and can understate uncertainty.
Under MAR, an imputation model should condition on variables that help explain missingness or predict the missing covariate, including relevant observed outcomes and design information. In this setting, source should generally be represented, either by fitting source-specific models or by a suitably flexible model that allows covariate distributions and relationships to differ by source. Treatment or strategy, outcome, and important interactions may also be needed for compatibility with the intended analysis. These choices are modeling decisions, not a guarantee that MAR holds.
A Bayesian analysis can instead specify a joint model for observed and missing variables and integrate the missing values over their posterior distribution. This propagates uncertainty in a principled way when the joint model and assumptions are appropriate. Whether using Bayesian computation or multiple imputation, the model must not silently borrow a trial covariate distribution for external patients when source-specific recording or patient mix makes that distribution implausible.
The term \(B\) reflects how much the estimate changes across plausible completed data sets; the factor \(1+1/m\) accounts for using a finite number of imputations. Standard multiple-imputation inference can use \(T\) with an appropriate degrees-of-freedom adjustment. A Bayesian joint-model analysis instead summarizes posterior draws under its specified model; do not label a pooled multiple-imputation interval as a Bayesian credible interval unless it was obtained from a posterior distribution.
Worked Example: Pooling Estimates Across Imputations
In a hypothetical study, five imputed data sets yield estimates of a treatment contrast in systolic blood pressure, in mmHg, of \(Q_1=2\), \(Q_2=3\), \(Q_3=1\), \(Q_4=4\), and \(Q_5=2\). Suppose each completed-data analysis has estimated variance \(U_k=16\) \((\mathrm{mmHg})^2\). Calculate the pooled estimate and standard error.
The pooled estimate is \(\bar{Q}=(2+3+1+4+2)/5=12/5=2.40\) mmHg. Since each within-imputation variance is 16, \(\bar{U}=16\) \((\mathrm{mmHg})^2\).
The deviations from 2.40 are \(-0.40, 0.60, -1.40, 1.60,\) and \(-0.40\). Therefore, \(B=(0.16+0.36+1.96+2.56+0.16)/4=5.20/4=1.30\) \((\mathrm{mmHg})^2\). The total variance is \(T=16+(1+1/5)(1.30)=16+1.56=17.56\) \((\mathrm{mmHg})^2\), so the pooled standard error is \(\sqrt{17.56}=4.19\) mmHg.
For illustration, a large-sample normal approximation gives \(2.40\pm1.96(4.19)\), or approximately \(-5.81\) to \(10.61\) mmHg. This is an approximate 95% multiple-imputation interval under the specified analysis and imputation assumptions, not a statement that there is a 95% probability the fixed treatment contrast lies in that interval. In practice, the interval should use the appropriate finite-imputation degrees-of-freedom procedure when warranted.
Missingness Indicators: Useful Description, Limited Correction
A missingness-indicator approach adds a variable that records whether a covariate is missing. It commonly fills the covariate with a fixed value, such as zero or a mean, and includes both the filled value and the indicator in a downstream model. The indicator can describe differences associated with the fact that a value was recorded, and it can be useful as a diagnostic or a pragmatic model component. However, it does not reveal the missing covariate values.
The limitation is especially important when trial and nontrial missingness arise from different measurement practices. A source-specific missingness indicator may distinguish recording patterns, but it cannot by itself adjust for the unobserved severity or prognosis of patients whose values are absent. Its validity depends on the analysis model and missingness process; it is not a general substitute for imputation or a justified missing-data model.
Worked Example: Why an Indicator Does Not Recover Severity
Consider hypothetical data on 100 patients. Severity \(X\) is observed for 50: among these, 20 have high severity and 12 experience the outcome, while 30 have low severity and 6 experience the outcome. For the other 50 patients, severity is missing; 25 experience the outcome. Compare two possible distributions of severity among those with missing values.
If 25 of the 50 missing-severity patients are high severity, the high-severity outcome rate among all high-severity patients is \((12+13)/(20+25)=25/45=55.6\%\), where 13 is a possible allocation of the 25 outcomes to the 25 high-severity patients. The low-severity rate is \((6+12)/(30+25)=18/55=32.7\%\). If instead 45 of the 50 are high severity and 5 are low severity, allocating 23 outcomes to the high group and 2 to the low group gives rates of \((12+23)/(20+45)=53.8\%\) and \((6+2)/(30+5)=22.9\%\), respectively.
Both allocations agree with the observed data and with the total of 25 outcomes among patients whose severity is missing, yet they imply different severity distributions and different severity-outcome relationships. A missingness indicator identifies those 50 patients as a group but cannot distinguish these allocations. Additional assumptions or information are required to infer their missing severity values.
Diagnostics, Comparisons, and Failure Modes
No observed-data diagnostic can prove MAR rather than MNAR. Diagnostics can, however, reveal whether an analysis relies on implausible extrapolation or ignores clearly different missingness patterns. Before selecting a primary approach, examine missingness by source and by covariate, and summarize the characteristics and outcomes that are observed for patients with and without each covariate. Check whether there are source-specific groups with almost no observed values. These checks do not establish the mechanism, but they help identify where assumptions matter most.
| Approach | Potential advantage | Main concern in external controls |
|---|---|---|
| Complete-case analysis | Simple; avoids specifying an imputation model | Can select a different population in each source and lose information |
| Multiple imputation or joint modeling | Uses partial records and represents uncertainty under a model | Depends on conditional-model adequacy, MAR or other stated assumptions, and source-appropriate extrapolation |
| Missingness indicator with fixed-value fill | Easy to implement and can describe recording status | Does not generally recover the missing covariate distribution or remove bias |
For imputation, inspect observed-versus-imputed distributions separately by source, including ranges and clinically plausible values. If the imputed external-control values look like the trial distribution despite substantial source differences in observed data, investigate whether the model has pooled sources too aggressively. Check convergence and Monte Carlo stability for simulation-based methods, and use enough imputations that pooled results are not dominated by simulation noise. Compare conclusions under defensible alternative models, such as source-specific imputation or different assumptions about values missing in the external cohort.
When MNAR is plausible, a MAR analysis alone cannot settle the question. Sensitivity analysis can vary assumptions about the distribution of missing values, for example by shifting imputed values within one source by prespecified clinically meaningful amounts, then repeating the analysis. The purpose is not to identify the true unobserved values from the data, but to show how conclusions change across transparent assumptions. A sharp change in conclusions signals that missingness assumptions are consequential.
A Practical Decision Sequence
Tabulate missingness for each covariate by source and relevant observed characteristics; note groups with little or no covariate information.
Clarify which population the chosen estimand concerns, what information predicts missingness, and whether MAR or MNAR is being considered.
Use complete cases only with a defensible selection argument; otherwise consider an imputation or joint model that represents source and relevant observed information.
Inspect source-specific imputed distributions and repeat the analysis under plausible alternative missingness assumptions.
The central pitfall is to treat a computationally convenient procedure as if it removed the need for assumptions. A complete-case result may describe selected patients; a missingness indicator may encode data-collection practice without recovering clinical information; and an imputation model may extrapolate across sources without adequate support. State which data were missing, how their absence differed by source, what assumptions the method requires, and how sensitive the conclusion is to those assumptions.
Check Your Understanding
Answer each question, then compare with the solutions.
- In a trial and an external cohort, the missing fraction for a baseline covariate differs substantially. Does that difference alone prove that the missingness is MNAR?
- What does MAR mean when source \(S\) is included among the observed variables?
- Why might complete-case analysis change the population represented in an external-control comparison?
- What does the between-imputation variance \(B\) represent in multiple-imputation pooling?
- What can a missingness indicator tell you, and what can it not generally recover?
Solutions. (1) No. It shows source-specific missingness rates, but does not establish whether missingness depends on unobserved covariate values after conditioning on observed information. (2) Conditional on source and the other included observed variables, missingness does not depend on the unobserved covariate value; this is an assumption, not a consequence of including \(S\). (3) If inclusion as a complete case depends on covariates, outcomes, or source-specific recording practices, the retained patients can have a different covariate distribution from the target population. (4) \(B\) measures variation in the estimates across completed data sets and therefore reflects uncertainty due to missing values. (5) The indicator records whether a value was observed and may capture patterns associated with recording; it does not generally identify the values or distribution of the missing covariate.