Why Selection Comes Before Discounting
A historical control trial can look useful because it is large or reports a favorable control outcome. Neither feature establishes that its data are relevant to the current trial. The first question is whether the historical control arm represents a defensible source of information about the current control condition. Only after that screening should an analyst consider how much information to borrow.
This distinction matters because selecting trials after seeing their outcomes can make borrowing depend on the very data used to estimate the control response. A prespecified screen instead asks whether each trial matches the current trial’s intended comparison and measurement conditions. The decision should be based on protocol documents and data availability, not on whether its results make the current study’s conclusion more attractive.
In a power prior, the historical likelihood contributions are discounted, schematically, by raising each trial’s likelihood to a power \(a_j\). If \(\mathcal{E}\) is the set of trials selected as eligible, the contributions have the form
Here \(D_j\) denotes the historical data from trial \(j\), \(L_j\) its likelihood under the outcome model, and \(a_j\) its discount. The details of the likelihood and discounting depend on the outcome model. The screening principle is common to these models: decide which likelihoods are substantively relevant before assigning their discounts. A small \(a_j\) is not a substitute for an eligibility decision.
What the Screen Should Compare
The screen begins with a precise description of the current trial’s control condition and target outcome. Record the population, control regimen and background care, primary endpoint, and assessment time or follow-up horizon. Then compare each candidate against those features using trial protocols, amendments, and other prespecified documentation. If a relevant fact cannot be established, record it as unresolved rather than silently treating the trial as a match.
| Dimension | Questions for the screen | Reason it matters |
|---|---|---|
| Protocol and control condition | Was the control arm assigned the same type of regimen and background care? Were key co-interventions or eligibility rules materially different? | Different care pathways can produce different control outcomes even in similar populations. |
| Population | Do eligibility criteria, disease state, severity, and other clinically defining features align with the current target population? | A trial in a different patient group may estimate a different control response. |
| Endpoint | Does the trial measure the same clinical outcome, with a definition that supports the same target quantity? | Different outcome definitions need not represent the same estimand. |
| Follow-up | Is the outcome assessed at a clinically comparable time, or over a comparable follow-up period? | Outcome rates can depend on when or over what period they are measured. |
A protocol comparison is not a judgment that every difference makes a trial unusable. It is a structured way to decide which differences are acceptable under the scientific question and which prevent a meaningful comparison. Criteria should be concrete enough that two reviewers could apply them consistently. For example, “similar disease severity” is less reproducible than a criterion naming the permitted severity categories or a defined eligibility range.
Prespecification should include how to handle protocol amendments, subgroup records, missing documentation, and disagreements between reviewers. It should also state whether a mismatch leads to exclusion or to an unresolved decision requiring adjudication. There is no universal numerical cutoff for every clinical feature. Any thresholds used should be justified by clinical knowledge and the target question, not chosen because they retain a preferred set of studies.
Why a Discount Cannot Repair a Mismatch
A power-prior discount reduces the influence of a historical likelihood; it does not change what quantity that likelihood describes. The point can be seen in a simple binary-outcome setting. Suppose the target is the control probability \(p_C\) of an event by a specified follow-up time, but a historical trial actually estimates a different probability \(p_H\). If the historical count is \(Y_H\) out of \(n_H\), its binomial log-likelihood for a shared probability \(p\) is
The expected score from this historical likelihood, evaluated at the current target \(p_C\), is
The second line uses \(E(Y_H)=n_Hp_H\). If \(p_H\ne p_C\), this expected score is not zero: the historical likelihood points toward its own event probability rather than the current target. Raising the likelihood to a positive power \(a_j\) multiplies its score by \(a_j\), but does not make the mismatch disappear. This is why screening for a shared target and measurement context must come before discounting.
This calculation is a justification for careful eligibility rules, not a test for whether two trials are compatible. In practice, the true historical and current probabilities are not both known. The relevant evidence comes from the protocols, clinical context, and outcome definitions. A lack of detectable differences in observed outcomes would not, by itself, prove that the underlying conditions were comparable.
A Practical Screening Process
Write down the control regimen, target population, endpoint, and follow-up time or period. Keep the description specific enough to compare against trial records.
State which protocol, population, endpoint, and follow-up features are required, acceptable with justification, or grounds for exclusion.
Record the supporting source and classify each dimension as aligned, misaligned, or unresolved. Do not infer alignment from a study’s outcome rate.
Include only candidates meeting the prespecified rules. Record reasons for exclusions and the resolution of unresolved cases before assigning any discount.
Eligibility identifies the candidate set; it does not determine the discount or guarantee that borrowing is harmless.
This process makes selection auditable. A useful record includes the rule, the evidence source, the reviewer’s assessment, and the final decision for each candidate. If reviewers disagree, resolve the disagreement according to the prespecified procedure rather than letting the observed outcome decide the case.
Worked Screening Examples
Worked Example: Protocol and Population Alignment
A current trial enrolls adults with moderate disease, requires stable background therapy, and assigns the control group to placebo plus that therapy. Historical Trial A enrolled adults with moderate disease, used the same background-therapy requirement, and had the same placebo control. Its records show that all 164 control participants met the adult eligibility criterion. Trial B enrolled adolescents and used a different background regimen. Apply a prespecified rule requiring adult participants with stable background therapy and the same type of control condition.
For Trial A, the observed count meeting the age criterion is 164 of 164, so the proportion is \(164/164=1.00\), or 100%. Its documented population and control condition meet the stated rule; it passes these protocol and population checks. This does not settle endpoint or follow-up alignment, which must be checked separately.
Trial B fails the age criterion because its participants are adolescents, and it also fails the background-regimen criterion. It is ineligible under this rule regardless of its sample size or observed response rate. The conclusion follows from the prespecified clinical design criteria, not from an arithmetic comparison of outcomes.
Worked Example: Follow-Up Does Not Match
The current trial’s primary control outcome is assessed at week 12. Before reviewing results, the team specifies that a candidate must assess the same outcome between weeks 11 and 13. Historical Trial C assessed it at week 8. The difference from the current assessment time is \(12-8=4\) weeks, which exceeds the allowed one-week difference. Therefore Trial C fails the follow-up rule.
The arithmetic identifies the protocol mismatch; it does not establish a universal four-week cutoff. The one-week window was stipulated for this hypothetical target and would need clinical justification in an actual analysis. Even if Trial C reports the same endpoint name, the week-8 result is not automatically an estimate of the week-12 outcome. It should not be declared eligible merely because a small discount might later be assigned.
Worked Example: A Subgroup Is Not Automatically a Match
The current trial targets patients in a defined disease-severity category. Historical Trial D enrolled 120 control participants: 72 in that category and 48 in a more severe category. Its report gives only the combined control outcome, not the separate outcome for the target category. The target-category share is \(72/120=0.60\), or 60%.
The 60% figure describes the composition of the arm; it does not show that its combined outcome estimates the control response for the current target population. Under a prespecified rule requiring outcome data for the target category, Trial D is unresolved if subgroup outcome data might be obtained, and ineligible if they cannot be obtained. It would be a mistake to treat 60% as an eligibility score or to assume the combined result is interchangeable with a target-category result.
Alternatives, Diagnostics, and Limitations
Two broad strategies are often contrasted. A restricted strategy applies firm eligibility criteria and borrows only from trials that pass. An inclusive strategy considers a broader set of trials and relies more heavily on modeling or discounting to limit their influence. The restricted strategy is transparent but may leave little historical information. The inclusive strategy can retain information but makes stronger demands on the model and on the justification for including dissimilar studies. Neither strategy makes a mismatch disappear.
Some remaining differences may be handled by appropriate outcome models or covariate adjustment when the necessary data and assumptions are available. That possibility does not remove the need to define a target population, endpoint, and follow-up first. If a candidate differs in a way that changes the target quantity, the analyst should not present adjustment or a small discount as an automatic remedy.
Warning signs include eligibility rules written only after candidate outcomes are inspected; criteria so vague that reviewers cannot apply them consistently; decisions based on sample size or favorable response rates; and missing records treated as evidence of alignment. Another warning sign is an eligible set that changes substantially under small, clinically plausible changes to the rules. Such sensitivity should be reported and investigated; it is not a reason to choose whichever rule yields the preferred inference.
The screen also has limits. Protocol alignment cannot prove equality of unmeasured clinical circumstances, and a trial that passes may still differ from the current trial in ways relevant to the outcome. Conversely, a study that fails one broad rule may contain a usable subset, but that subset must be defined and supported by the available data under prespecified criteria. Selection is a reasoned eligibility judgment, not a guarantee of exchangeability.
Common Mistake
Correction: A discount changes the strength of a likelihood contribution, not the population, endpoint, or follow-up that generated it. First decide whether the trial estimates a relevant target under prespecified criteria. Then assign discounts among eligible trials and assess the consequences of those choices.
Check Your Understanding
Answer each question using the prespecified-screening principle. Brief solutions follow the questions.
- Why should eligibility decisions be made before examining candidate control outcome rates?
- A current endpoint is assessed at week 16. A historical trial assesses it at week 16, but its control group received a different background regimen that violates the stated eligibility rule. Is it eligible?
- A historical trial has an overall population that only partly matches the target population. What information would make it possible to assess a target-population subgroup under a subgroup-based rule?
- Explain why a power-prior discount cannot make a week-8 outcome estimate identical to a week-12 target.
- What should the screening record contain when a protocol document does not establish whether a candidate met an eligibility criterion?
Solutions. (1) Outcome-driven selection can make inclusion depend on the result being borrowed; prespecified decisions reduce this risk. (2) No. Matching follow-up alone is insufficient, and the protocol mismatch violates the stated rule. (3) It would require appropriately defined and available subgroup outcome data, plus a prespecified rule allowing that subgroup to be considered; the overall result is not enough. (4) Discounting scales the historical likelihood’s contribution but does not change the time point or estimand that generated it. (5) Record the criterion as unresolved, identify the missing evidence, and follow the prespecified procedure for obtaining evidence or adjudicating eligibility.