Qualification Is About Fitness for a Specific Question
A real-world data source can contain millions of records and still be unsuitable for a particular external-control analysis. The issue is not whether the source is generally “high quality,” but whether its records can support the specific comparison being proposed. A claims database may record medication dispensing reliably but lack a clinical measure needed to determine eligibility. An electronic health record (EHR) may contain detailed clinical measurements but miss care delivered outside its network. A registry may define a disease population carefully but enroll only a selected subset of patients.
The earlier material on target-trial emulation motivates the key starting point: specify the target population, treatment strategies, outcome, and observation period before judging the data. Qualification asks whether the source captures those elements adequately enough for the intended external-control role. It does not, by itself, establish that patients receiving different treatments are causally comparable.
The assessment should be explicit about the required population, treatment, endpoint, and follow-up. For each, state what must be recorded, what counts as adequate capture, and what evidence would demonstrate it. Requirements depend on the estimand and the consequences of error; there is no universal percentage threshold that makes a source fit.
Four Dimensions to Examine
Population capture concerns whether the source identifies the people who meet the target population definition. Coverage and representativeness are related but not identical. Coverage asks what fraction of the target population appears in the source. Representativeness asks whether inclusion is associated with characteristics that affect treatment or outcome. A source can cover many patients yet selectively miss groups with different prognoses.
Treatment capture asks whether the recorded information identifies the treatment actually received, with timing and detail appropriate to the treatment strategy. A prescription, a dispensing record, an administration record, and a patient’s actual use are not interchangeable. The appropriate evidence depends on the treatment: a dispensing claim may support medication access, for instance, but does not prove adherence.
Outcome capture asks whether the source can identify the endpoint as defined for the analysis. A code or recorded value is a measurement, not automatically a validated endpoint. Validation may compare the source’s classification with an independent reference, such as adjudicated records. Sensitivity is the fraction of true cases correctly identified; specificity is the fraction of true noncases correctly identified. Positive predictive value is the fraction of source-identified cases that are true cases and depends on outcome prevalence in the population being assessed.
Follow-up capture asks whether the data can establish that patients remain observable over the required period and whether outcomes can be recorded during that period. Continuous enrollment, repeated clinical encounters, or linkage to a death registry may provide different kinds of evidence. A missing record is not necessarily evidence that an event did not occur; it may indicate that care took place elsewhere or that the person was no longer observable.
Worked Example: Coverage Can Still Distort the Population
Suppose a hypothetical target population contains 1,000 patients: 600 lower-risk patients and 400 higher-risk patients. A registry captures 90% of the lower-risk group but only 50% of the higher-risk group. The event risks are 10% and 30%, respectively. Assess how this selective coverage changes the event rate among captured patients.
The target population’s expected event count is \(600(0.10)+400(0.30)=60+120=180\). Its event risk is therefore \(180/1{,}000=0.18\), or 18.0%.
The registry captures \(600(0.90)=540\) lower-risk patients and \(400(0.50)=200\) higher-risk patients. The expected event count among those captured is \(540(0.10)+200(0.30)=54+60=114\). The captured group’s expected event risk is \(114/740=0.1541\), or 15.4%.
Overall coverage is \(740/1{,}000=74.0\%\), but the captured group’s expected event risk is 2.59 percentage points lower than the target population’s risk: \(18.0\%-15.4\%=2.59\) percentage points, subject to rounding. The source has substantial coverage but is not compositionally representative on this prognostic grouping. The calculation assumes the stated group-specific risks and capture fractions apply to the target population; it illustrates a coverage problem, not a causal treatment effect.
This example shows why a single overall coverage rate is not enough. Qualification should examine capture across characteristics that matter for the target population and outcome. The analysis must also distinguish what the data directly establish from what is assumed or inferred.
Check Whether Recorded Treatments Match the Strategy
Treatment information should be evaluated against the actual strategy definition. Ask what event generates the record, whether the record indicates prescribing, dispensing, administration, or use, and whether treatment changes can be observed. If a source records only one medication fill, it may not establish sustained use. If treatment is delivered in hospital but the source captures only outpatient claims, administrations may be missed.
A validation sample can quantify classification errors. Sensitivity and specificity describe performance against a reference classification. Predictive values also reflect how common the treatment is in the population where the source will be used. These measures are useful only when the validation sample and reference standard are appropriate to the intended setting.
Worked Example: Interpreting Treatment-Classification Accuracy
In a hypothetical validation study, an independent review identifies 100 truly treated and 100 truly untreated patients. The source marks 90 of the treated patients as treated and incorrectly marks 5 of the untreated patients as treated. Assume the treatment prevalence in the intended external-control population is 40%. Calculate sensitivity, specificity, and the expected positive predictive value at that prevalence.
Sensitivity is the fraction of truly treated patients identified as treated: \(90/100=0.900\), or 90.0%. Specificity is the fraction of truly untreated patients identified as untreated: \(95/100=0.950\), or 95.0%.
For the intended population, the positive predictive value is \(P(\text{truly treated}\mid\text{source says treated})\). Using the assumed prevalence, sensitivity, and specificity:
Thus, under these assumptions, about 92.3% of source-classified treated patients are truly treated. This is not the same as saying that 90% of source treatment records are correct: sensitivity has a different denominator. The calculation assumes the validation estimates transport to the intended population and that treatment prevalence there is 40%. If either assumption fails, the predictive value may differ.
Validate the Endpoint and the Observation Window
Outcome validation should match the endpoint definition and the population in which it will be used. A code that identifies an event during inpatient care may not identify the same event in outpatient records. Validation based on a different case mix, coding system, or time period may not apply. Report the reference standard, the sampling design, and the denominator for each accuracy measure. If validation samples are selected according to source classification, predictive values from that sample may not reflect the target population’s predictive values.
Worked Example: Reading an Outcome-Validation Table
In a hypothetical simple random sample of 200 patients, an endpoint is classified both by the source and by independent clinical review. The source identifies 100 patients as having the endpoint. Of those, 90 are confirmed by review and 10 are not. Among the 100 source-negative patients, review finds 20 endpoint cases and 80 noncases. Calculate sensitivity, specificity, positive predictive value, and negative predictive value.
There are \(90+20=110\) true endpoint cases and \(10+80=90\) true noncases. Sensitivity is \(90/110=0.8182\), or 81.8%. Specificity is \(80/90=0.8889\), or 88.9%. Positive predictive value is \(90/100=90.0\%\), and negative predictive value is \(80/100=80.0\%\).
The source classification is not error-free: it misses 20 of 110 reviewed cases and labels 10 reviewed noncases as cases. The positive and negative predictive values here apply to this simple random validation sample. They should not automatically be carried to a target population with a different endpoint prevalence.
Follow-up qualification is a separate question from endpoint accuracy. A source may classify an observed event well but fail to observe patients consistently throughout the required period. Examine whether care outside the source is visible, whether data updates are sufficiently timely, and whether a recorded absence of an event can reasonably be distinguished from lack of observation. The details of defining follow-up and censoring require their own analysis; for qualification, the essential point is to establish what the source can and cannot observe.
Worked Example: Quantifying Follow-Up Availability
A hypothetical source identifies 250 eligible external-control patients. At the 12-month point, records establish that 210 remain observable in the source; for 40, coverage is no longer established. Calculate the documented 12-month availability and interpret it.
Documented availability is \(210/250=0.840\), or 84.0%. Thus, 84.0% of identified patients have evidence of observability through the 12-month point under the source’s availability definition. The remaining 16.0% cannot be treated as confirmed event-free merely because no event appears in the source. This proportion alone does not show whether loss of observability is related to prognosis, nor does it prove that events are captured completely among those still observable.
Evidence, Diagnostics, and Alternatives
Useful qualification evidence includes a clear description of how records are generated, counts with explicit denominators, validation against a suitable reference, and comparisons of capture across clinically relevant groups. Where possible, assess whether findings hold under plausible alternative definitions of treatment or endpoint. Warning signs include unexplained gaps in key fields, abrupt changes in recording practices, inconsistent patient counts across source components, and validation results that differ sharply across subgroups.
No data-source type is uniformly superior. Registries may have disease-specific definitions and structured outcomes, but enrollment can be selective and follow-up may be incomplete. EHRs may provide clinical detail, yet care outside the contributing network can be missing. Claims may offer broad records of billed services and dispensing, but clinical state and actual medication use may be less directly observed. Linked sources can combine strengths, while adding linkage errors or leaving some patients unlinked.
| Source type | Potential qualification strength | Question to investigate |
|---|---|---|
| Registry | Structured disease-specific enrollment and measures | Who is eligible to enroll, and who is actually enrolled? |
| EHR | Clinical notes, measurements, and care decisions | How much relevant care occurs outside contributing systems? |
| Claims | Recorded billed services and some dispensing activity | Do recorded codes and claims represent the clinical treatment or endpoint required? |
| Linked sources | Complementary information across data systems | Who links successfully, and what information remains unavailable? |
The main alternative is not simply “use another database.” It may be to use a different source, restrict the proposed analysis to a population the source can support, or decide that the source cannot support the required endpoint or follow-up. These choices have different consequences for the estimand and should not be hidden by a general claim that the database is large or reputable.
Why Qualification Matters
External-control analyses use observed data to represent outcomes under a comparison strategy. If treatment or outcome records are systematically wrong, or if the source selectively captures patients, the resulting evidence may not represent the intended comparison. In Bayesian analyses, adding more records can make a posterior distribution more concentrated without removing systematic error. A large sample is not a substitute for reliable measurement or relevant population coverage.
Qualification is therefore necessary but not sufficient. A source may accurately capture all four dimensions and still differ from the trial population in ways that matter for the treatment–outcome comparison. Conversely, a source may be informative for one endpoint or population but unsuitable for another. Record the evidence and the unresolved limitations so that the analysis does not treat assumptions as verified facts.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A source captures 95% of a target population overall, but capture is much lower among patients with severe disease. Why may the overall percentage be inadequate evidence of population suitability?
- A dispensing record is present for a patient. What does it establish, and what does it not necessarily establish about treatment?
- In a population where an outcome is rare, what happens to positive predictive value if sensitivity and specificity remain fixed? Explain briefly.
- A patient has no recorded endpoint during a year, but the source does not establish that the patient remained observable. What conclusion is justified?
Solutions. (1) Overall coverage can conceal selective capture of a prognostic subgroup; the captured population may differ in severity and outcome risk from the target population. (2) It supports that a dispensing record exists, but does not by itself establish that the medication was taken or used as intended. (3) Positive predictive value generally decreases as prevalence decreases, because false-positive classifications make up a larger share of all positive classifications. (4) The source does not establish that the patient remained endpoint-free; the event may be unobserved because follow-up in that source is not confirmed.