Why Source Selection Matters
An external-control analysis can be carefully implemented and still be uninformative if its source does not represent the patients, outcomes, and care processes relevant to the trial. Selection is therefore not a database popularity contest. It is a structured comparison of candidate sources against the question the trial is designed to answer.
The key question is not simply whether a source contains many records. A large source may have weak outcome measurement or unclear treatment histories; a smaller source may document these well but represent a different clinical population. The choice affects which external patients can be considered, what their records mean, and how credible any borrowing from them can be. Source selection should be documented before results from the external controls are used to support a treatment conclusion.
A suitable source must meet essential requirements before it can be considered for borrowing. After those requirements are met, sources can be compared on relative strengths. A high score on one dimension should not compensate for a fundamental defect on another: excellent follow-up cannot rescue an endpoint that does not measure the trial outcome, for example.
Five Dimensions to Compare
The comparison should be specific to the trial question. The following dimensions provide a practical structure; their relative importance depends on the endpoint and the protocol.
| Dimension | Selection question | Evidence to examine |
|---|---|---|
| Population relevance | Could the source represent patients relevant to the trial’s target population? | Clinical setting, disease characteristics, care context, and availability of information needed to judge relevance. |
| Endpoint validity | Does the recorded outcome measure the clinical endpoint of interest? | Operational definition, measurement process, and validation evidence for the records used to identify outcomes. |
| Treatment documentation | Can the control-care strategy be identified reliably? | Whether treatment, treatment changes, and relevant dates are recorded sufficiently for the trial question. |
| Follow-up completeness | Can outcomes be ascertained over the period relevant to the trial? | Availability of outcome information over time, reasons observation may end, and how much follow-up is missing. |
| Protocol compatibility | Can the source support the main requirements of the trial protocol? | Whether the source’s available population, measurements, treatment information, and observation period can support the intended comparison. |
These dimensions are related but not interchangeable. For example, a record may contain a treatment label without enough detail to establish the care strategy of interest. Likewise, a high proportion of recorded outcomes does not show that the endpoint is valid. The selection rationale should state what evidence supports each judgment, what remains uncertain, and which deficiencies are disqualifying.
Use Gates Before Scores
A transparent comparison can use two stages. First, set essential requirements that a source must satisfy to remain eligible for consideration. These might include a minimum level of endpoint validity or sufficient documentation to identify the relevant treatment. Requirements should be justified by the protocol and endpoint, not chosen after seeing which source wins. A source that fails a genuine requirement should not be rescued by a high total score.
Second, compare sources that pass the requirements on graded criteria. A simple weighted score can make priorities visible. It is a decision aid, not a statistical estimate of bias, a probability of validity, or a guarantee of exchangeability.
The score is a weighted average because each rating is first converted to the same 0-to-1 scale and then multiplied by its weight. Since the weights sum to 1, a source rated \(R\) on every criterion has a score of 1. This calculation is useful for organizing judgments, but the ratings themselves remain evidence-based assessments rather than objective measurements unless a criterion has a directly measured value.
If rankings change substantially under plausible alternative weights, the choice is sensitive to subjective priorities. Report that sensitivity rather than presenting one score as an unquestionable answer. A scorecard should also preserve the underlying evidence: two sources with the same rating may have reached it for different reasons and have different residual risks.
Worked Example: Ranking Two Sources With an Explicit Scorecard
Suppose a hypothetical trial team compares two candidate sources using ratings from 0 to 4. Before scoring, the team confirms that both sources pass its essential minimum requirements for endpoint measurement and treatment identification. The team assigns weights of 30% to population relevance, 25% to endpoint validity, 20% to treatment documentation, 15% to follow-up completeness, and 10% to protocol compatibility.
| Candidate source | Population | Endpoint | Treatment | Follow-up | Compatibility |
|---|---|---|---|---|---|
| A | 4 | 3 | 4 | 2 | 3 |
| B | 3 | 4 | 3 | 4 | 4 |
For source A, the score is:
For source B, the score is:
Under these stated weights and ratings, B ranks higher, with a score of 0.875 compared with 0.8375 for A. This is a narrow preference, not proof that B is unbiased or suitable in every respect. The team should retain the dimension-by-dimension comparison and check whether reasonable changes to the weights reverse the ranking. If so, it should explain why its final choice gives priority to one dimension, rather than hiding the trade-off in a single total.
Assess Measurement and Observation Evidence
A source’s descriptions of its data are not enough to establish endpoint validity. For an outcome identified through a code or algorithm, validation evidence should relate to the intended endpoint and to the records that will actually be used. One common quantity is positive predictive value: among records classified as having the outcome, the proportion for which the outcome is confirmed by a suitable reference assessment. This does not, by itself, measure sensitivity, which concerns the proportion of all true outcomes that the method detects.
A validation estimate also has sampling uncertainty. The simple standard error below assumes a binomial sample in which validation outcomes are independent and the sample represents the coded-positive records to which the estimate will be applied. If those assumptions are not plausible, this calculation may not describe the uncertainty adequately.
Worked Example: Comparing Endpoint Validation Results
In a hypothetical validation exercise, reviewers assess 100 records classified as having a trial-relevant event in each of two candidate sources. They confirm the event in 88 records from source A and 80 records from source B. Assume for this illustration that each set is a representative, independent sample of coded-positive records, and that the review is an appropriate reference assessment.
The estimated positive predictive values are \(88/100=0.88\), or 88%, for A and \(80/100=0.80\), or 80%, for B. For a sample proportion \(\hat{p}\), the estimated binomial standard error is \(SE=\sqrt{\hat{p}(1-\hat{p})/n}\). For A:
Using the approximate interval \(\hat{p}\mathbin{\pm}1.96SE\), A’s interval is \(0.88\mathbin{\pm}1.96(0.0325)\), approximately 0.816 to 0.944. For B:
B’s corresponding approximate interval is \(0.80\mathbin{\pm}1.96(0.0400)\), or 0.722 to 0.878. These calculations suggest higher positive predictive value in A’s validation sample, but the intervals overlap. They do not establish that A is better on every source-selection dimension, and they say nothing by themselves about missed events among records not classified as positive. The team should consider the validation design and its relevance to the intended endpoint before using these estimates to rank sources.
Follow-up completeness also requires interpretation in context. A source can have many observed outcomes and still miss outcomes systematically, for example if patients who leave a care system are no longer observable. A completion percentage is a useful description of available information, not evidence that missing outcomes are unrelated to patient characteristics or clinical status.
Worked Example: Applying Essential Requirements to Follow-Up and Treatment Records
A hypothetical protocol team specifies, before reviewing the candidates, that a source must have at least 90% of patients with the relevant treatment identifiable and at least 85% with outcome status ascertainable at 12 months. These are illustrative protocol-specific screening requirements, not universal standards. Source C has treatment identifiable for 228 of 240 patients and 12-month outcome status for 216 of 240. Source D has treatment identifiable for 170 of 200 patients and 12-month outcome status for 190 of 200.
For C, treatment documentation is \(228/240=0.950\), or 95.0%, and 12-month outcome ascertainment is \(216/240=0.900\), or 90.0%. It passes both stated requirements. For D, treatment documentation is \(170/200=0.850\), or 85.0%, while 12-month outcome ascertainment is \(190/200=0.950\), or 95.0%. D passes the follow-up requirement but fails the treatment-documentation requirement. Under the stated rules, C remains eligible and D does not, despite D’s higher outcome-ascertainment percentage.
This conclusion is conditional on the specified thresholds and on the denominator representing the patients relevant to the intended comparison. It does not show that all C patients have unbiased outcomes or that C is clinically representative. The source team should also investigate what “ascertainable” means operationally and whether patients without observed status differ in ways that matter.
Diagnostics, Alternatives, and Limits
Before finalizing a choice, look for warning signs that a source’s apparent strengths are overstated or that the ranking depends on an arbitrary decision. Useful checks include:
- Weak or mismatched validation: the endpoint was validated in a different setting or using a definition unlike the trial endpoint.
- Vague treatment records: a broad treatment label is being treated as proof of the control strategy required by the trial question.
- Unexplained loss of observation: completeness is reported as a percentage, but the source cannot describe how outcome observation ends or who is no longer observed.
- Ratings without evidence: scores are assigned from general impressions rather than documented source characteristics.
- Ranking instability: small, reasonable changes in weights reverse the preferred source.
- Compensation for a fatal defect: a strong population rating is allowed to offset a failure to measure the endpoint of interest.
A qualitative comparison is often preferable when the evidence is sparse or the criteria are not meaningfully commensurate. A scorecard is useful when it makes judgments and priorities more transparent, but a numerical total can create false precision. Conversely, a purely qualitative recommendation can conceal inconsistent standards unless the evidence and reasons are recorded clearly.
Using one source can simplify interpretation and avoid inconsistencies across systems, but may leave important weaknesses unresolved. Combining sources may broaden the available evidence, yet differences in measurement and care context can make the combined data harder to interpret; combining records does not automatically improve relevance or validity. If no candidate meets essential requirements, declining to borrow is a defensible result. The cost of a larger external sample does not justify using information that cannot support the trial question.
Finally, source selection is not a demonstration that trial and external patients are exchangeable, nor does a favorable score remove confounding or measurement error. It identifies a source whose characteristics and limitations can be assessed against the protocol. The analysis must still account for the consequences of remaining differences; source selection alone cannot establish a causal treatment effect.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A candidate source has excellent follow-up but does not measure the trial endpoint with a validated method. Should its high follow-up rating compensate? Explain.
- A validation sample contains 75 confirmed outcomes among 100 records classified as positive. Calculate the estimated positive predictive value and its approximate standard error.
- Two sources have scores of 0.84 and 0.86. What should the team check before treating the higher score as a clear preference?
- A source has 95% outcome ascertainment but fails a prespecified minimum requirement for identifying the relevant treatment. What follows under the two-stage selection approach?
- Does selecting a clinically relevant, well-documented source establish that its patients are exchangeable with trial participants? Why or why not?
Solutions. (1) No. If endpoint validity is an essential requirement and the source fails it, the high follow-up rating cannot compensate; follow-up matters only for outcomes that measure the intended endpoint. (2) The estimate is \(75/100=0.75\), or 75%. Under the binomial approximation, \(SE=\sqrt{0.75(0.25)/100}=\sqrt{0.001875}\approx0.0433\). (3) Check the evidence behind the ratings, whether both sources pass essential requirements, and whether reasonable changes in weights or ratings reverse the ranking; a 0.02 difference is not automatically decisive. (4) The source fails an essential requirement and should not remain eligible under those prespecified rules, even though its outcome ascertainment is high. (5) No. Source selection evaluates relevance and data quality but cannot establish exchangeability or eliminate confounding and measurement error.