Why Audit Feasibility Before Borrowing
Selecting a plausible data source is only an initial decision. Before it can contribute external controls, the source must contain enough patients who could qualify, enough usable information to characterize them and ascertain outcomes, and follow-up that supports the planned comparison. A feasibility audit makes those constraints visible before substantial analysis or borrowing is attempted.
The audit is descriptive and decision-oriented. It estimates the size and information content of the candidate pool; it does not establish that the source and trial participants are exchangeable, or that a treatment effect is unbiased. A large database can yield few usable controls after restrictions and missingness. Conversely, a smaller source may be promising if it provides nearly complete, relevant follow-up for a sufficiently large group.
The audit should begin with a specified use: the intended control population, the outcome and its measurement horizon, and the approximate amount of information needed. These are audit inputs, not conclusions. Eligibility definitions will need to be aligned with the target trial; this audit can count records under a provisional screening definition without treating that preliminary count as a final eligible cohort.
Construct a Count Funnel
A count funnel documents how many records remain at each stage and why records are lost. State the starting population and the denominator for every percentage. A percentage of the original source and a percentage of records remaining at a later stage answer different questions.
If the stages are nested, the final count can be written as the original count multiplied by the conditional retention proportions at each stage. This identity is useful because it shows how several individually moderate losses can leave a small usable sample.
Each fraction is conditional on reaching the previous stage, so the product telescopes to the final count. In practice, keep the stage counts as well as the fractions: a rounded percentage can hide a consequential loss when the starting source is large or small. Record duplicate records, records with no usable outcome data, and other exclusions separately when those categories can be identified. Do not imply that every exclusion is independent; the funnel is an accounting device, not a model of independent losses.
Worked Example: Estimating the Usable Control Count
A hypothetical registry contains 1,200 patient records. Under a provisional screen, 864 meet the broad clinical criteria. Of those, 786 have the baseline fields required for the planned analysis, and 660 have an outcome recorded within the audit’s specified assessment period. The planning team wants at least 600 potentially usable records.
The sequential retention proportions are \(864/1200=0.720\), \(786/864=0.910\), and \(660/786=0.840\), to three significant figures. The overall proportion is \(660/1200=0.550\), or 55.0%.
The product gives the same result:
The source clears the stated count target by 60 records. That is a preliminary feasibility result, not a guarantee that 660 people will form the final analysis set: the provisional screen may change, and further data checks may remove records. The audit should therefore report the margin over the target, the exact stage counts, and which screening assumptions remain unverified.
Summarize Missingness With the Right Denominators
A single statement such as “10% of data are missing” is rarely adequate. Report missingness for each important variable and, where meaningful, for the outcome, with the number eligible to have that information as the denominator. A variable measured only in a subset should not be described as though every source record was expected to contain it. Also distinguish a missing value from a value that is structurally unavailable or not applicable.
For each key field, useful summaries include the number and proportion missing, the number with observed values, and whether missingness is concentrated in particular calendar periods, sites, or patient groups. These patterns can signal that the apparent usable count overstates the information available for a relevant subset. The summaries do not by themselves establish why values are missing or justify a particular statistical correction.
A complete-case count is a practical feasibility quantity, but it is not automatically the final analysis sample size. Some missing data may be addressed using suitable methods, subject to their assumptions and the analysis plan; such methods cannot recover information that was never captured without relying on additional assumptions. For a feasibility decision, show both what is observed and what would be required to make a larger count usable. Avoid silently substituting an imputation plan for a demonstrated data capability.
Worked Example: Finding a Missingness Bottleneck
In a hypothetical source, 500 provisionally eligible patients have outcome records. A key baseline severity measure is recorded for 410, and a required comorbidity field is recorded for 450. Both fields are needed for the proposed adjustment plan. Among the 500 patients, 390 have both fields observed.
Severity is missing for \(500-410=90\) patients, or \(90/500=18.0\%\). Comorbidity is missing for \(500-450=50\), or \(50/500=10.0\%\). The number complete on both fields is 390, so \(500-390=110\), or \(22.0\%\), are missing at least one of them. The two missingness counts cannot simply be added: some patients may be missing both fields.
The complete-on-both count is \(390/500=78.0\%\). If the planning threshold is 400 patients with both fields, the source falls short by 10 on the observed-data criterion. The audit should report the overlap and investigate whether missingness varies across sites or time. It cannot conclude from these percentages alone that missingness is harmless or that adjustment will be valid.
Use Outcome Rates as a Diagnostic, Not a Verdict
For a binary outcome, report the event count, denominator, event rate, and assessment horizon. For example, “48 of 300 patients had the event by 12 months” is interpretable; “the event rate was 16%” without its denominator or horizon is not. If the comparison includes another cohort or a planning benchmark, the outcome definition, ascertainment method, and time horizon must be sufficiently alike for the crude rates to be informative.
Sampling uncertainty matters. Under a simple binomial model in which each patient has the same event probability \(p\) and outcomes are independent, the estimated rate is \(\hat{p}=x/n\), and its approximate standard error is \(\sqrt{\hat{p}(1-\hat{p})/n}\). This approximation is a summary of random variation under those assumptions, not a remedy for differences in case mix, measurement, or follow-up.
A notable crude difference can flag a mismatch, a data-quality problem, or a real population difference that needs investigation. A small difference does not prove comparability, especially when the sample is imprecise. Neither finding by itself determines whether or how much information should be borrowed.
Worked Example: Interpreting a Crude Event-Rate Difference
Suppose a hypothetical external source has 60 events among 300 patients by 12 months. A separate, comparable-in-definition reference group has 48 events among 300 patients by the same horizon. Treat these rates as descriptive audit summaries, not as a randomized treatment comparison.
The source rate is \(60/300=0.200\), or 20.0%; the reference rate is \(48/300=0.160\), or 16.0%. The observed difference is \(0.200-0.160=0.040\), or 4.00 percentage points.
Using independent binomial approximations, the standard error of the difference is:
An approximate 95% interval for the descriptive difference is \(0.040 \pm 1.96(0.0313)\), which is about \(-0.0213\) to \(0.101\), or \(-2.13\) to 10.1 percentage points. The interval is wide: these data do not show that the rates are close, and they do not establish that they differ. The calculation also assumes independent observations and comparable outcome ascertainment. A feasibility audit should prompt investigation of those conditions rather than turn this interval into a test of causal validity.
Assess the Follow-Up Distribution
A mean or median follow-up time alone does not show whether enough patients have outcome information at the horizon needed for the comparison. Report a distribution—such as the median and interquartile range—and the number or proportion with observation extending to clinically relevant landmarks. If follow-up varies substantially, include useful quantiles or a plot in the audit materials. State how follow-up duration was measured and what constitutes an observed outcome at each landmark.
The proportion observed through a horizon is a feasibility summary, not the event-free proportion. Patients who stop contributing data before that horizon cannot simply be counted as non-events. If outcome ascertainment ends early for many records, the source may provide limited information for a long-term endpoint even when the overall patient count looks adequate. Differences in follow-up patterns between the source and the intended trial should be investigated; a matching median alone cannot resolve them.
Worked Example: Checking Coverage of a 12-Month Horizon
A hypothetical source has 200 provisionally eligible patients. Their median recorded follow-up is 13.2 months, with an interquartile range of 9.4 to 18.1 months. The audit finds that 132 patients have outcome observation extending through 12 months, while 184 have observation extending through 6 months.
Coverage through 12 months is \(132/200=0.660\), or 66.0%; 68 patients, or 34.0%, do not have observation extending through that horizon. Coverage through 6 months is \(184/200=0.920\), or 92.0%.
Although the median follow-up exceeds 12 months, only two-thirds of the cohort has documented observation through 12 months. The median alone would obscure this limitation. Whether 132 is adequate depends on the intended use and the amount of information required; the audit should not treat the other 68 patients as event-free at 12 months.
Make the Audit Decision Transparent
Feasibility criteria should follow the intended analysis rather than an invented universal cutoff. A source may need a larger usable count for a rare outcome, more complete measurements for adjustment, or longer observation for a delayed endpoint. State the criteria before interpreting the summaries where possible, and show which criteria are met, unmet, or uncertain.
| Audit dimension | Useful summary | What it can flag |
|---|---|---|
| Potential control count | Stage counts, exclusions, and final count | Whether the source has enough candidates after screening |
| Data completeness | Missing counts and proportions by key field and relevant subgroup | Whether crucial information is available consistently |
| Outcome information | Event count, denominator, rate, and horizon | Whether the endpoint is observed often and consistently enough to be informative |
| Follow-up | Distribution and coverage at the planned horizon | Whether observation duration supports the intended endpoint |
A structured audit is more informative than choosing a source from its database size or reputation alone. It also differs from proceeding directly to a model-based analysis: modeling cannot make absent outcomes appear, guarantee sufficient follow-up, or establish that key variables were measured comparably. The audit can identify a promising source, a source needing targeted verification, or a source that should not advance. The later design and analysis must still address alignment, confounding, and other threats to comparability.
Diagnostics, Pitfalls, and Limits
- Using the wrong denominator. A missingness percentage among all database records can disguise substantial missingness among the patients who could actually enter the comparator. Report the relevant denominator at every stage.
- Adding missingness percentages. If two fields are missing for overlapping patients, adding the percentages overstates the number with any missing data. Count the overlap or report the number complete on all required fields.
- Treating a large source as a large control group. A large starting database may shrink sharply after screening, outcome ascertainment, and data-quality checks. Follow the counts through the funnel.
- Reading similar crude rates as proof of comparability. Similar rates can result from imprecision, different case mix, or offsetting measurement differences. Treat them as a diagnostic signal, not a causal conclusion.
- Using the median follow-up as the whole story. A median above the target horizon does not mean all or most patients have complete observation to that horizon. Report landmark coverage and the distribution.
- Assuming a discount fixes weak feasibility. Down-weighting borrowed information does not create missing baseline data or outcome follow-up. First establish that the source can provide usable information at all.
The audit is limited by what can be extracted and verified at this stage. Counts may change after detailed screening, and descriptive summaries cannot establish the absence of unmeasured differences. Document definitions, extraction dates, and unresolved uncertainties so the feasibility decision can be revisited when more detailed information becomes available.
Check Your Understanding
Answer each question, then compare with the short solutions below.
- A source has 900 records; 720 pass provisional screening, 648 have the required baseline field, and 518 have outcome data. What is the overall proportion with outcome data among the original records?
- Two required baseline fields are missing for 12% and 9% of 400 patients. Can you conclude that 21% are missing at least one field? What additional count is needed?
- A binary outcome occurs in 25 of 250 patients by 6 months. State the event rate and identify two details needed before comparing it with another source.
- A cohort’s median follow-up is 15 months, but 55 of 200 patients have observation through 12 months. What is the 12-month coverage, and why is the median insufficient?
Solutions. (1) \(518/900=0.576\), so 57.6% of the original records have outcome data. (2) No. The missingness groups may overlap; count patients missing at least one field, or count those with both fields observed. (3) \(25/250=0.100\), or 10.0%. At minimum, check that the outcome definition and assessment horizon match, and report the denominator and ascertainment basis. (4) \(55/200=0.275\), or 27.5%. The median does not show the fraction observed through a particular horizon, and most patients in this example lack documented observation through 12 months.