Why Recruitment Patterns Deserve Careful Comparison
An interim review can coincide with changes in enrollment, screening, consent, or withdrawal. A change in these processes may be operationally important: for example, a rapid rise in enrollment could warrant checking staffing, eligibility procedures, or communication practices. But a before-and-after difference is not, by itself, proof that anyone saw interim results or that those results caused a change. Recruitment also varies with site opening dates, season, referral volume, staffing, and chance.
The task is therefore to define a meaningful comparison, quantify the size and uncertainty of any shift, and examine whether the pattern is consistent across sites and indicators. This complements the earlier material on diagnosing leakage from operational signals: recruitment measures can help identify a pattern for review, but they do not establish its cause.
Define the Periods, Events, and Denominators
Before calculating a comparison, specify the event date that anchors the review period. Depending on the operational question, this might be the date interim information became available to a particular role, the meeting date, or the date a recruitment decision was implemented. These dates need not coincide. Define the pre- and post-review windows before inspecting the results where feasible, and state how observations during a transition period are handled.
The denominator must match the process being measured. A screening-failure proportion uses people screened as its denominator. A consent proportion usually uses eligible people approached for consent, with a clear rule for repeat approaches. An enrollment rate may use calendar time or site-time as exposure. A withdrawal proportion may use enrolled participants with a specified follow-up opportunity; a withdrawal rate may instead use participant-time at risk. These quantities answer different questions and should not be substituted for one another.
| Process measure | Possible numerator | Relevant denominator | Interpretation |
|---|---|---|---|
| Enrollment rate | Participants enrolled | Open site-months or calendar time | Enrollment pace over exposure time |
| Screening-failure proportion | People failing screening | People screened | Share of screened people who fail |
| Consent proportion | Eligible people who consent | Eligible people approached | Share of approached eligible people who consent |
| Withdrawal proportion | Participants withdrawing | Enrolled participants with defined follow-up | Share withdrawing over the specified follow-up |
For enrollment rates, site-months should count only time when a site was open and able to recruit. For proportions, eligibility and follow-up definitions should remain consistent across periods. If a definition or data-collection procedure changes, the apparent shift may reflect measurement rather than recruitment. Record whether incomplete records, repeat screenings, and participants who withdraw before a particular milestone are included.
Comparing Rates and Proportions
For a simple comparison, let \(Y_{pre}\) and \(Y_{post}\) be event counts in the two periods, and let \(E_{pre}\) and \(E_{post}\) be their exposure times. The estimated rates are \(Y_{pre}/E_{pre}\) and \(Y_{post}/E_{post}\). Their ratio describes relative change; their difference describes absolute change in events per unit of exposure. Both can be useful, because a large relative increase from a very low starting rate may still represent a small absolute change.
A basic count model assumes that, within each period, counts follow a Poisson distribution with a stable rate per unit of exposure, and that the two counts are independent. Under this model, the estimated log rate ratio has approximate standard error \(\sqrt{1/Y_{pre}+1/Y_{post}}\). The approximation is most useful when both counts are not very small. A confidence interval describes uncertainty under the model; it does not account automatically for site clustering, seasonality, or a change in operational conditions.
For two independent proportions, let \(\hat{p}_{pre}=x_{pre}/n_{pre}\) and \(\hat{p}_{post}=x_{post}/n_{post}\), with \(x\) denoting the number of events and \(n\) the corresponding denominator. A large-sample confidence interval for the difference \(\hat{p}_{post}-\hat{p}_{pre}\) uses the unpooled standard error \(SE=\sqrt{\hat{p}_{post}(1-\hat{p}_{post})/n_{post}+\hat{p}_{pre}(1-\hat{p}_{pre})/n_{pre}}\). This approximation can be poor with sparse counts or proportions near zero or one; exact or suitable regression methods may then be preferable. Independence also matters: repeated records from the same people or shared site-level processes can make a simple independent-sample calculation misleading.
Worked Example: Enrollment Rate Before and After a Review
Suppose, hypothetically, a trial has 24 enrollments during 12 open site-months before a defined interim-review date and 39 enrollments during 12 open site-months afterward. Assume for this initial calculation that counts are independent Poisson observations with a constant rate within each period, and that exposure has been measured consistently.
The estimated rates are \(24/12=2.00\) enrollments per site-month before and \(39/12=3.25\) afterward. The post-to-pre rate ratio is:
The approximate standard error of the log rate ratio is:
Since \(\log(1.625)\approx0.4855\), a 95% interval on the log scale is \(0.4855\pm1.96(0.2594)\), or approximately \((-0.0229,0.9939)\). Exponentiating the endpoints gives a rate-ratio interval of approximately \((0.977,2.702)\). The estimated rate is 62.5% higher after the review, but this interval includes 1. Under these assumptions, the comparison does not provide clear evidence of a rate difference at the 5% level. It neither proves the rates are equal nor identifies a cause for the observed difference.
The simple model would be questionable if the sites had different opening dates that were not reflected in site-months, if enrollment varied substantially by season, or if a small number of high-volume sites drove the total. Those features should be checked before treating the aggregate estimate as a reliable summary.
Worked Example: Comparing Consent Proportions
In a hypothetical trial, 30 of 100 eligible people approached before a review consent, compared with 18 of 90 approached afterward. Assume each person contributes once, the eligibility and approach rules are unchanged, and observations are independent for this approximate comparison.
The consent proportions are \(0.300\) before and \(0.200\) after. For the post-minus-pre difference, the estimate is \(-0.100\), or a decrease of 10.0 percentage points. The unpooled standard error is:
The approximate Wald test statistic for a zero difference is \(z=-0.100/0.06227\approx-1.606\), giving a two-sided normal-approximation p-value of about \(0.108\). The 95% confidence interval is:
The data are compatible with a decline as large as about 22.2 percentage points and with a small increase of about 2.2 percentage points, under the approximation. Because the interval includes zero and the p-value exceeds 0.05, this comparison does not provide strong statistical evidence of a change at that threshold. The p-value is not the probability that consent rates are equal, and the result does not establish that there was no operational change.
Why Site Mix Can Mislead
A trial’s overall recruitment measure can change even when each site’s process is stable. This occurs when the proportions of observations contributed by sites change and the sites have different baseline rates. A pooled comparison can therefore mix two phenomena: changes within sites and changes in which sites contribute participants. Stratification or standardization helps separate these descriptions, though neither alone establishes a cause.
Worked Example: An Aggregate Change Without Within-Site Change
Consider hypothetical consent data from two sites. Before the review, Site A has 20 consents among 50 eligible people approached, and Site B has 10 among 50. After the review, Site A has 40 among 100 and Site B has 5 among 25. The within-site proportions are unchanged: Site A is 40% in both periods, and Site B is 20% in both.
Before the review, the pooled proportion is \((20+10)/(50+50)=30/100=0.300\). Afterward, it is \((40+5)/(100+25)=45/125=0.360\). Thus the pooled proportion rises by 6.0 percentage points, despite no within-site change. The post-period contains a larger share of approaches from the higher-consent Site A.
With equal site weights, the standardized proportion is \(0.5(0.400)+0.5(0.200)=0.300\) in each period. This calculation shows that the pooled shift is explained by the changed site composition in this example. In real data, inspect site-specific denominators and uncertainty as well: small site counts can make rates unstable, and a site-specific difference may be ordinary variation.
The same reasoning applies to screening failures and withdrawals. A change in the mix of sites, referral sources, or participant follow-up can alter an overall proportion without a change in the underlying site-specific process. Conversely, a stable overall proportion can hide opposing changes at different sites. Report the aggregate and inspect relevant strata rather than relying on one number alone.
Diagnostics, Alternatives, and Limits
A useful diagnostic sequence is to plot the measure over time, mark the relevant review dates, and inspect the underlying counts and denominators. A single before-and-after contrast can hide a gradual trend, a temporary interruption, or a change that began before the review. Check whether the observed pattern is concentrated in one site, whether the same direction appears in related indicators, and whether documentation or definitions changed at the same time.
Check dates, eligibility rules, duplicate records, missing denominators, site opening periods, and the operational definitions used in each period.
Show counts, denominators, rates or proportions, and a time plot. Include absolute and relative contrasts where each is informative.
Compare within-site patterns and consider seasonality, gradual trends, staffing, referral changes, and other concurrent events.
Treat an unusual, well-verified pattern as a reason for a documented operational review, not as a conclusion about cause or intent.
For sparse data, exact methods for proportions or count rates can avoid unreliable large-sample approximations. With repeated monthly observations, a Poisson or binomial regression can model exposure and include period, time trend, site, and relevant operational covariates. Such models require their own assumptions; for example, a simple Poisson model assumes the conditional variance matches the conditional mean. If counts vary more than that assumption allows, standard errors and tests may be too optimistic unless overdispersion is addressed. A regression can adjust measured features, but it cannot remove bias from unmeasured changes or turn an observational before-and-after comparison into a randomized experiment.
The main alternative is not always a more complicated model. A transparent descriptive table with carefully chosen denominators and site-specific summaries may be more trustworthy when there are few observations or few sites. Conversely, a single pooled proportion can be inadequate when site composition changes. The method should match the data structure and the question, not simply maximize model complexity.
A common mistake is to reason, “Enrollment increased immediately after the review, so staff must have learned the treatment effect.” The increase may be a useful signal, but timing alone does not show that staff had access to interim information or that access caused the change. Referral volume, site activation, seasonality, chance, or a change in recording could also explain it. Establishing a credible operational explanation requires checking access and communication records and investigating the recruitment process; the recruitment comparison itself supplies neither proof nor exoneration.
Check Your Understanding
Answer each question using the definitions and calculations in this tutorial.
- A screening-failure comparison uses all people enrolled as the denominator. What is wrong with this choice?
- Consent proportions are 24/80 before and 18/90 after. Calculate the two proportions and the post-minus-pre difference.
- Why can an overall consent proportion increase while every site’s consent proportion remains unchanged?
- A post-review enrollment rate ratio is estimated as 1.4, with a 95% interval from 0.9 to 2.2. State an appropriate interpretation.
- Give two reasons a before-and-after recruitment shift does not establish that interim information caused it.
Solutions. (1) Screening failure is defined among people screened, not those enrolled; using enrolled participants changes the question and excludes failures by construction. (2) Before, \(24/80=0.300\); after, \(18/90=0.200\); the post-minus-pre difference is \(0.200-0.300=-0.100\), a decrease of 10.0 percentage points. (3) The site mix may shift toward sites with higher baseline consent proportions, changing the pooled value without within-site changes. (4) The estimated post-review rate is 1.4 times the pre-review rate, but the interval includes 1, so the data do not provide clear evidence of a difference at the 5% level under the model; the interval also reflects considerable uncertainty. (5) Examples include seasonality, referral or staffing changes, site activation, chance, and changes in data definitions. A temporal association alone does not establish access to information or causation.