Why External Data Need a Structured Assessment
External clinical data may help describe outcomes under a comparator treatment, provide context for a rare disease, or contribute information to a regulatory analysis. But a large dataset is not automatically informative for a particular trial. It can be accurately recorded yet poorly matched to the patients, treatment, endpoint, or care setting relevant to the regulatory question. Conversely, a closely matched source may be too incomplete or unreliable to support its intended use.
Assessment therefore has two related but distinct parts. Relevance asks whether the source represents the clinical question and target population. Data quality asks whether the records and measurements are sufficiently accurate, complete, traceable, and consistent for the proposed analysis. Neither is captured adequately by a single sample-size figure or a general statement that the data are “real world.”
Start from the regulatory question and specify the intended use before inspecting source data. State who the target patients are, what outcome and time point matter, which treatment strategy is being represented, and the relevant calendar period and care context. The earlier material on establishing acceptability of external controls addresses whether an external control is acceptable in principle; the present assessment examines the concrete features of particular data sources that can support or undermine that case.
Dimensions of Relevance and Quality
A useful assessment examines each dimension explicitly, documents evidence, and describes the likely consequence of any mismatch. It does not assume that all differences are equally important or can be offset by a strong result on another dimension.
| Dimension | Questions to ask | Potential consequence of mismatch |
|---|---|---|
| Population | Do eligibility, disease severity, prior therapy, demographics, and care setting match the target patients? Are important characteristics measured in both sources? | Different prognosis or treatment response; limited comparability or lack of overlap. |
| Endpoint | Are the outcome definition, assessment method, time origin, and follow-up window aligned? | Outcome misclassification or comparison of different clinical events. |
| Treatment | Does the source represent the same intervention, dose, treatment line, supportive care, and treatment era? | The source may represent a different treatment strategy or care pathway. |
| Calendar time | Were patients treated under similar standards of care, diagnostic practices, and follow-up systems? | Secular changes can alter prognosis, outcome capture, or available therapies. |
| Data quality | Are records traceable, variables consistently defined, and outcomes and follow-up sufficiently complete? | Errors, missingness, or selective recording can distort estimates. |
The comparison must be specific to the intended use. For example, an endpoint recorded in an electronic health record may be adequate for identifying a hospitalization but less reliable for a subtle symptom change requiring a standardized patient-reported instrument. Similarly, matching age distributions does not establish population comparability if disease stage or prior treatment differs substantially. If important factors are absent or defined differently across sources, apparent similarity on recorded variables may be misleading.
A principled assessment also considers whether target patients have counterparts in the external source. If the source contains almost no patients with a target-relevant severity level, a numerical adjustment cannot create direct information for that subgroup. The issue is not merely whether the source is “similar on average,” but whether its observed patients and measurements can speak to the target population and outcome.
Worked Assessments
Worked Example: A Difference in Baseline Population
Suppose a hypothetical trial seeks to understand outcomes in patients with advanced disease. The target trial population is 70% high severity and 30% moderate severity. An external registry has 30% high-severity and 70% moderate-severity patients. Its observed event risks are 40% among high-severity patients and 20% among moderate-severity patients. For this illustration, assume these risks are measured consistently in both sources and the registry has enough patients in each stratum to estimate them.
The registry’s overall event risk is
A risk standardized to the target population’s severity mix is
The 8 percentage-point difference, 34% minus 26%, arises from the different severity composition under the stated assumptions. It shows why the registry’s crude outcome rate may not describe the target population. This calculation does not prove that severity is the only relevant difference: standardization is informative only for the measured strata and the assumed comparable within-stratum risks. Other differences, such as prior treatment or care setting, still require evaluation.
Worked Example: Endpoint Definitions Do Not Align
A hypothetical trial measures disease progression by centrally reviewed imaging at 24 weeks. An external database records whether a clinician entered “progression” in a medical record during the same period. Among 100 database patients, 30 have that entry. A review of the records confirms that only 24 of those 30 satisfy the trial’s imaging-based definition; six do not. A further 10 patients have no usable imaging or clinical documentation at 24 weeks.
Among patients with enough information to classify the outcome, there are \(100-10=90\) patients. The proportion meeting the trial definition is
or 26.7%. The raw database proportion is \(30/100=0.3000\), or 30.0%, but it counts six clinician entries that do not meet the trial definition and treats the ten unclassified patients as if their outcomes were known. The calculated 26.7% is a descriptive proportion among classifiable patients, not necessarily the full-cohort 24-week risk. The 10% unclassified fraction must be investigated; if outcome availability is related to progression, restricting to classifiable patients may also introduce bias. The source’s endpoint therefore requires validation and a clear account of unavailable outcomes before it can be compared with the trial.
Worked Example: Calendar-Time Change in Standard Care
A hypothetical external cohort was treated several years before a trial. In the earlier period, 50 of 200 patients received a newly available supportive therapy, and 40 of those patients experienced an event. The other 150 did not receive it, and 60 experienced an event. The corresponding later-period trial setting routinely uses the supportive therapy, so the treatment mix differs.
The earlier cohort’s overall event risk is
Among the earlier-period patients who received the supportive therapy, the observed risk is \(40/50=0.800\). Among those who did not, it is \(60/150=0.400\). If, purely for a descriptive comparison, the later setting’s treatment mix is 100% supportive therapy and the earlier group’s observed treated risk were applicable, the standardized risk would be \(1.00(0.800)=0.800\), not the earlier cohort’s crude 0.500.
This large difference is not evidence that the supportive therapy caused the difference: treatment was not randomly assigned, and patient characteristics or other changes may explain some or all of the contrast. The calculation identifies a serious calendar-time and treatment-context issue. It motivates investigating treatment allocation, prognosis, and other contemporaneous changes rather than treating the older overall rate as directly comparable.
How to Judge the Evidence and Its Limits
The preceding examples illustrate a disciplined sequence. First, define the target use and comparison. Next, examine whether source definitions and patients match that target. Then quantify important differences where the data permit, while stating the assumptions behind each calculation. Finally, determine whether the mismatch is addressable with available information or instead limits the source’s usefulness.
Describe the target patients, treatment strategy, endpoint, time point, and care context relevant to the regulatory question.
Record how patients entered the dataset, which variables and outcomes were collected, how they were defined, and when and where care occurred.
Tabulate key differences, check overlap and missingness, and verify uncertain definitions against source documentation or record review where feasible.
Explain how each important discrepancy could affect the intended inference and whether the available evidence can address it.
Quality checks should examine provenance, data collection, linkage, coding, time stamps, and outcome ascertainment—not just whether a dataset is large. For example, an event count may be incomplete if patients leave the contributing health system, while a recorded zero may mean either “no event” or “no record found.” Missingness should therefore be described by variable, reason where known, and time pattern. Review whether outcome availability differs by observed patient characteristics or clinical course; if it does, a complete-case summary may not represent the full source population.
A useful diagnostic is to compare the data dictionary and operational definitions with the protocol’s definitions, then test the agreement in a sample of records if validation is feasible. Check impossible values, duplicate records, inconsistent dates, implausible follow-up, and changes in coding practices over time. Such checks can detect errors, but passing them does not demonstrate clinical relevance. Likewise, matching recorded baseline summaries does not establish that unmeasured prognostic factors or care pathways are comparable.
There is no universal relevance score that converts these dimensions into a regulatory conclusion. A weighted score can conceal a critical failure—for instance, an excellent population match cannot compensate for an outcome that measures a different clinical event. Qualitative judgments should be supported by concrete evidence, and quantitative summaries should be accompanied by their definitions and assumptions. Where a source lacks information needed to assess a material difference, describe that uncertainty rather than treating the absence of documentation as evidence of equivalence.
This assessment is also not itself a method for incorporating external observations into a Bayesian analysis. Whether and how information is borrowed, discounted, or combined is a separate design and analysis decision. The source evaluation provides evidence about what the data represent and how trustworthy they are; it should not be replaced by a model’s ability to produce a precise estimate. A model cannot repair an endpoint that was not measured, make an unrepresented target subgroup appear in the data, or establish that an old treatment context matches current practice.
Common Pitfall
Wrong reasoning: “The external database contains thousands of patients and its average outcome is close to the trial’s, so it is a high-quality and relevant source.” The sample size and numerical similarity do not establish either claim. A large source can systematically misclassify outcomes, omit patients with poor follow-up, or represent a different severity mix and treatment era.
Correction: Separate relevance from quality, and assess both against the prespecified regulatory use. Verify what was measured and how; compare target and source populations, endpoint definitions, treatment context, and calendar period; quantify important discrepancies where possible; and state unresolved limitations. Similar results can be reassuring only when the underlying measurements and populations are meaningfully comparable.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A source and target trial have the same mean age, but the source has many more patients with prior treatment. Which relevance dimension needs further assessment, and why?
- A registry records “no hospitalization” when no hospitalization appears in its database. What data-quality concern does this raise?
- In a source, event risks are 30% in a subgroup that makes up 20% of source patients and 10% in a subgroup that makes up 80%. What is the crude overall risk? If the target population is 50% in each subgroup, what is the standardized risk using the same subgroup risks?
- Why does a close match in overall event rates not establish that an external endpoint is suitable?
- Name two reasons calendar time may undermine the relevance of an older external cohort.
Solutions. (1) Prior treatment and related population differences need assessment because they may relate to prognosis or treatment response; matching mean age alone is insufficient. (2) The absence of a database record may reflect incomplete capture or care outside the system, not a confirmed absence of hospitalization. (3) The crude risk is \(0.20(0.30)+0.80(0.10)=0.06+0.08=0.14\), or 14%. The target-standardized risk is \(0.50(0.30)+0.50(0.10)=0.15+0.05=0.20\), or 20%. (4) Similar rates can arise even when definitions, ascertainment, or follow-up differ; the outcome must be shown to represent the same clinical event at the relevant time. (5) Standards of care, available treatments, diagnostic practices, patient selection, or outcome-capture systems may change over time.