Tutorials › Bayesian Clinical Trials › Aligning Eligibility Criteria

Home  /  Tutorials  /  External and Real-World Controls

External and Real-World Controls · Tutorial 353 of 696

Aligning Eligibility Criteria

Turn protocol eligibility criteria into reproducible record-based rules, then assess how uncertain or erroneous classification can alter the external-control population.

Advanced 10 min read

What You'll Learn

  • Separate each protocol criterion into a record-based rule, a review rule, and an explicit unknown category.
  • Distinguish evidence that a patient fails a criterion from missing or ambiguous evidence.
  • Design reproducible clinical adjudication for criteria that codes and structured fields cannot resolve.
  • Calculate how imperfect eligibility classification changes the composition of the selected comparator.
  • Identify diagnostics and sensitivity analyses for eligibility-related selection.

Why Eligibility Alignment Matters

An external control is informative only to the extent that its patients could have entered the trial under the trial’s eligibility rules. A diagnosis code may identify the right disease family while failing to distinguish the required subtype; a laboratory field may be absent even when the patient would have passed the trial threshold. If the operational rules for nontrial records differ from the protocol, the comparator can represent a different clinical population. Borrowing more information from that comparator does not repair the mismatch.

Eligibility alignment is therefore a translation and measurement task. The protocol states clinical conditions; the external data contain codes, measurements, notes, and gaps in documentation. Each criterion needs a defensible rule for interpreting that evidence, with uncertainty retained rather than silently converted into a pass or fail. The result should be an auditable classification of patients as eligible, ineligible, or unresolved under the operationalized rules.

Definition: Eligibility alignment is the process of translating every relevant trial inclusion and exclusion criterion into explicit rules for assessing nontrial records, and evaluating whether those rules identify a comparator population consistent with the trial’s intended eligibility population. A patient is operationally eligible only if the available evidence satisfies all required criteria and does not establish an exclusion.

Translate Each Criterion Before Selecting Patients

Begin with the protocol wording, not with whichever fields happen to be easiest to extract. For every criterion, record its clinical meaning, the evidence needed to assess it, the permissible evidence sources, the decision rule, and what happens when evidence is absent or conflicting. Fix the protocol version and the operational definitions before examining comparator outcomes where practicable. Otherwise, decisions may be influenced, even unintentionally, by the observed outcome pattern.

Some criteria are directly operationalizable from reliable structured data. Others require a clinical interpretation that codes alone cannot support. A histologic subtype, disease stage, prior-treatment history, or functional-status threshold may depend on narrative documentation or a combination of records. For such criteria, specify a review rubric: what evidence counts, how contradictions are resolved, whether reviewers can assign “uncertain,” and how disagreements are handled. A reviewer should not infer a condition from silence unless the protocol’s meaning and the data source justify that inference.

The key distinction is between evidence of failure and failure to find evidence. If the protocol requires a laboratory value above a threshold, a documented value below the threshold can establish failure; no recorded value generally establishes that the criterion cannot be assessed from that field. Treating every unrecorded value as a failure can exclude eligible patients, while treating it as a pass can include patients whose eligibility is unsupported. Keep “unknown” visible and report its frequency.

1
Decompose the protocol.
List each inclusion and exclusion criterion separately, preserving thresholds, qualifiers, and any required clinical meaning.
2
Map evidence to rules.
Specify the data fields or note evidence that can satisfy, contradict, or fail to resolve each criterion.
3
Classify without collapsing uncertainty.
Assign pass, fail, or unknown for each criterion, then define how those statuses determine overall eligibility.
4
Review and validate.
Use trained reviewers for ambiguous criteria, check agreement, and examine a sample of records that automated rules exclude.
5
Describe the resulting population.
Report criterion-level exclusions and unknowns, not only the final number selected.

Worked Examples of Operationalization

Worked Example: A Clinical Criterion Requiring Review

Suppose a hypothetical trial requires a specified tumor subtype confirmed by pathology review. In an external record set, 120 patients have a code for the broader tumor family. A code for the broad family is not sufficient evidence for the required subtype, so the code is used only to find potentially relevant records. A reviewer applies a written rubric to pathology reports.

The review classifies 82 records as confirmed subtype, 21 as a different subtype, and 17 as indeterminate because the report is absent or inconclusive. Under a rule requiring confirmation, the 82 confirmed records pass this criterion, the 21 documented alternatives fail it, and the 17 indeterminate records remain unresolved. Thus, the confirmed pass proportion among the 120 screened records is \(82/120=0.6833\), or 68.3%; the unresolved proportion is \(17/120=0.1417\), or 14.2%.

These calculations describe the screening and review process, not the prevalence of the required subtype in all patients with that disease. In particular, the 17 indeterminate records cannot be assumed to be either eligible or ineligible. Their frequency is itself important: if they are excluded from the primary comparator, the selected group may differ from the group that would have been eligible had complete pathology documentation been available.

Worked Example: A Laboratory Threshold and Unrecorded Values

A hypothetical protocol requires hemoglobin of at least 9.0 g/dL at the eligibility assessment. In 30 otherwise potentially relevant external records, 24 have a usable value in the prespecified assessment window: 20 values are at least 9.0 g/dL and 4 are below 9.0 g/dL. Six records have no usable value.

For this criterion, \(20/24=0.8333\), or 83.3%, of patients with an observed usable value pass, and \(4/24=0.1667\), or 16.7%, fail. Relative to all 30 records, the observed counts are 20 pass, 4 fail, and 6 unknown. It would be incorrect to report \(20/30=66.7\%\) as the pass rate among assessed patients: that denominator treats missing measurements as if they were observed failures.

If a primary rule requires a documented qualifying value, only the 20 observed passes satisfy this criterion. The six unknown cases should be reported separately, and a sensitivity analysis can examine how conclusions change under plausible classifications of them. This does not establish that their true values pass; it makes the dependence on incomplete measurement explicit.

How Misclassification Changes the Comparator

When eligibility is imperfectly measured, the selected comparator can contain truly ineligible patients or omit truly eligible ones. A simple calculation shows how false inclusions change its composition. Let \(E=1\) denote true eligibility and \(E^*=1\) denote operationally classified eligibility. Suppose the prevalence of true eligibility among candidate records is \(q=P(E=1)\), the sensitivity is \(Se=P(E^*=1\mid E=1)\), and the specificity is \(Sp=P(E^*=0\mid E=0)\). Among records classified eligible, the proportion that is truly eligible is:

$$ P(E=1\mid E^*=1) =\frac{qSe}{qSe+(1-q)(1-Sp)}. $$

The numerator is the proportion of all candidates who are true eligible patients and are correctly included. The denominator adds those patients to the truly ineligible candidates incorrectly included. This expression requires the stated sensitivity and specificity to apply to the candidate population being considered; it does not by itself account for different error rates across clinical subgroups.

Worked Example: False Inclusions Shift the Comparator Event Rate

Consider a hypothetical candidate population in which 70% are truly eligible. Suppose the operational rule has sensitivity \(Se=0.90\) and specificity \(Sp=0.80\). Among 1,000 candidates, 700 are truly eligible and 300 are truly ineligible. The rule includes \(700(0.90)=630\) true eligible patients and \(300(1-0.80)=60\) ineligible patients. It therefore selects 690 patients, of whom \(630/690=0.9130\), or 91.3%, are truly eligible.

Now suppose the hypothetical control event risk is 0.20 among true eligible patients and 0.40 among ineligible patients. The event risk in the selected comparator is the weighted average:

$$ \frac{630(0.20)+60(0.40)}{690} =\frac{126+24}{690} =0.2174. $$

The selected comparator’s event risk is 21.7%, rather than the 20.0% risk among true eligible patients: false inclusions raise it by 1.74 percentage points in this example. If the ineligible patients instead had a lower event risk, the mixture could move in the opposite direction. The direction and size of the resulting treatment comparison distortion depend on how the ineligible patients’ outcomes differ and on the treatment-group outcomes; they cannot be inferred from misclassification rates alone.

Justification: The Selected Outcome Is a Mixture

The outcome-mixture calculation follows directly from conditioning on true eligibility among selected patients. Let \(Y\) be a binary outcome and let \(r_1=P(Y=1\mid E=1,E^*=1)\) and \(r_0=P(Y=1\mid E=0,E^*=1)\). The law of total probability gives:

$$ P(Y=1\mid E^*=1) =P(Y=1\mid E=1,E^*=1)P(E=1\mid E^*=1) +P(Y=1\mid E=0,E^*=1)P(E=0\mid E^*=1). $$

Writing \(w=P(E=1\mid E^*=1)\), this is \(wr_1+(1-w)r_0\). Its difference from the eligible-group risk \(r_1\) is:

$$ \{wr_1+(1-w)r_0\}-r_1=(1-w)(r_0-r_1). $$

Thus, the shift is determined by both the ineligible fraction \(1-w\) and the outcome-risk difference \(r_0-r_1\). The derivation assumes the risks refer to the same selected population and outcome definition, and does not imply that eligibility status alone causes outcome differences. In practice, this calculation is a diagnostic illustration; estimating the quantities requires suitable validation information and comparable outcome ascertainment.

Diagnostics, Alternatives, and Limits

Useful diagnostics make the eligibility decision visible. Report the number screened, the number passing, failing, and unresolved for each criterion, and how many patients are lost at each sequential rule. For clinically reviewed criteria, report the rubric, reviewer training, and agreement on a sample independently assessed by more than one reviewer. Agreement is not proof that the classification is correct, so where feasible also compare classifications with a stronger reference source, such as detailed chart review. Check records excluded by automated rules, because a rule that looks specific among included records can still miss many eligible patients.

The main alternatives each answer a different question. A strict structured-data rule is reproducible, but may exclude patients whose records lack the required fields. Broad inclusion preserves more records, but risks admitting patients who do not meet the protocol. Clinical adjudication can interpret nuanced evidence, but requires resources and can introduce reviewer variation. A useful primary analysis can use a prespecified, defensible rule while sensitivity analyses vary uncertain classifications or use a narrower, better-documented subset. These are not interchangeable ways to create certainty: they show how conclusions depend on assumptions about eligibility.

The simple sensitivity-and-specificity calculation assumes a binary true status, a binary operational decision, and error rates that can be meaningfully applied to the candidate population. Real criteria are often multidimensional, and errors may depend on disease severity, care setting, or documentation practices. A patient can pass most criteria and remain unresolved on one decisive criterion. Further, missing evidence may be associated with health status or healthcare use, so the unknown group need not resemble either confirmed group. Avoid presenting a single pooled misclassification rate as adequate if its assumptions are implausible.

Eligibility alignment also cannot establish that the external and trial populations are comparable on every prognostic feature, nor can it fix differences in outcome measurement or treatment assignment. It defines a more defensible comparator population; it does not by itself guarantee an unbiased treatment contrast. This distinction matters when interpreting Bayesian borrowing: a model can represent uncertainty in an estimate, but uncertainty about whether the records represent trial-eligible patients is a design and measurement concern that should be addressed explicitly.

Common Pitfall

Common mistake: “No record of the exclusion means the patient passes the criterion.” This reasoning confuses absence of evidence with evidence that a criterion is satisfied. Correction: define an explicit unknown status when the record cannot establish pass or fail, quantify how often it occurs, and assess how alternative defensible classifications affect the comparator.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

  1. A protocol requires a particular disease subtype, but the external database records only a broad disease code. What can the code establish, and what additional step is needed?
  2. Among 200 candidate records, 150 have an observed threshold measurement; 120 pass and 30 fail. The other 50 have no usable measurement. What are the pass proportion among assessed records and the appropriate status for the 50 unmeasured records?
  3. If a selected comparator contains 90% truly eligible patients, and the event risk is 0.10 in eligible patients and 0.30 in ineligible patients, what is the selected comparator’s event risk under the mixture model?
  4. Why does high agreement between two clinical reviewers not, by itself, establish that eligibility classification is valid?
  5. Name one limitation of applying a single sensitivity and specificity to every patient in an external record set.

Solutions. 1. The broad code can identify candidates but cannot establish the required subtype; review a suitable source such as the pathology documentation using a prespecified rubric. 2. The assessed-record pass proportion is \(120/150=0.80\), or 80%. The 50 unmeasured records are unknown for that criterion, not automatically failures or passes. 3. The risk is \(0.90(0.10)+0.10(0.30)=0.12\), or 12%. 4. Reviewers can agree while sharing the same mistaken interpretation or incomplete evidence; agreement measures consistency, not truth. 5. Error rates may differ by disease severity, care setting, or documentation practice, so a pooled rate may not represent important subgroups.

Previous

Next

Back to Tutorials
Clinical Biostats