Why Site-Level Differences Need Careful Interpretation
In a multicentre adaptive trial, sites may differ in how quickly they recruit, how often treatment-management events occur, or how completely outcomes are recorded. A marked difference can be a useful integrity signal: it may prompt a closer look at process, data quality, or the conditions under which the trial is operating. But a site comparison is not, by itself, proof of information leakage, misconduct, or biased treatment delivery.
Sites often serve different patient populations and carry different workloads. A site with more complex patients may reasonably have more protocol deviations or missing assessments. A busy site may recruit fewer eligible patients per month than a less busy one, even if its recruitment process is sound. Chance also creates variation, especially when a site has few patients. The aim is therefore not to rank sites by raw counts, but to compare observed patterns with expectations that reflect the relevant opportunities and circumstances.
Define the Comparison Before Calculating
First specify the process, the unit being counted, and the denominator that represents an opportunity for the process to occur. For recruitment, the denominator might be eligible patients approached, rather than calendar months alone. For a treatment-management event, it might be the number of patient visits at which that event could occur. For outcome recording, it might be the patients due for an assessment. A count without an opportunity denominator can make a high-volume site look unusual simply because it serves more patients.
Next identify characteristics that could reasonably alter the expected pattern. These can include patient risk or complexity, eligibility and referral pathways, follow-up burden, staffing, and workload during the period being examined. The comparison should use information collected independently of the outcome being assessed where possible. Adjusting for a variable that was itself affected by the suspected process can hide the signal rather than clarify it.
A simple starting point divides a site's opportunities into groups with comparable characteristics. For group \(j\), let \(n_j\) be the number of opportunities and \(p_j\) the expected probability of the event in that group, based on a suitable reference. The expected count is \(E=\sum_j n_jp_j\). If each opportunity has an independent Bernoulli outcome with the stated probability, its variance is \(p_j(1-p_j)\), so the variance of the total count is \(V=\sum_j n_jp_j(1-p_j)\). A standardized residual compares the observed count \(O\) with this expectation in units of its standard deviation.
This calculation is a screening summary, not a universal test of integrity. Its interpretation relies on the probabilities being relevant to the site, the opportunities being counted consistently, and the outcomes being approximately independent conditional on the groups used. When probabilities are estimated from data, their uncertainty should also be considered; treating estimated values as known can make the comparison appear more precise than it is. A normal approximation for \(Z\) is most plausible when expected counts are not very small and no one opportunity dominates the total. With sparse counts, exact or simulation-based reference distributions may be more appropriate.
Worked Comparisons
Worked Example: Recruitment Below a Case-Mix Expectation
Hypothetical data. A site has 80 eligible patients referred during a defined period: 50 are in a lower-complexity group and 30 in a higher-complexity group. From a suitable reference, the expected enrollment probabilities are 0.60 and 0.40, respectively. The site enrolled 31 patients. Assume referrals are counted consistently and, conditional on complexity group, each patient's enrollment outcome is independent with the stated probability.
The case-mix-adjusted expected count is \(E=50(0.60)+30(0.40)=30+12=42\) enrollments. The variance under the stated independent Bernoulli model is \(V=50(0.60)(0.40)+30(0.40)(0.60)=12+7.2=19.2\). Thus the standard deviation is \(\sqrt{19.2}=4.382\), and
The site enrolled 11 fewer patients than expected under this particular case-mix benchmark; the difference is about 2.51 model-based standard deviations below expectation. This is a reason to check the referral pathway, eligibility screening, staffing, and the appropriateness of the reference probabilities. It does not establish that enrollment was improperly restricted: unmeasured differences, an incorrect benchmark, or chance could also contribute. If workload varied substantially, the benchmark should also reflect the number and timing of recruitment opportunities.
Worked Example: Treatment-Management Events and Workload
Hypothetical data. A site records whether a specified protocol-management deviation occurred at each of 140 eligible opportunities. The opportunities fall into three groups defined in advance by patient complexity and workload: 60 routine, usual-workload opportunities with expected probability 0.04; 40 complex, usual-workload opportunities with probability 0.10; and 40 complex, high-workload opportunities with probability 0.16. There are 22 deviations. Assume each opportunity is counted once, the groups are correctly classified, and outcomes are independent conditional on group.
The expected count is \(E=60(0.04)+40(0.10)+40(0.16)=2.4+4+6.4=12.8\). The variance is \(V=60(0.04)(0.96)+40(0.10)(0.90)+40(0.16)(0.84)=2.304+3.6+5.376=11.28\). The standard deviation is \(\sqrt{11.28}=3.359\), giving
The observed count is higher than the benchmark after accounting for the specified complexity and workload groups. A follow-up check could examine whether the definition of a deviation was applied consistently, whether workload was measured for the relevant period, and whether the events cluster by staff member or time. The result does not show that workload caused the deviations, or that any deviation arose from an integrity problem. If events within a patient or staff member are correlated, the independence assumption understates uncertainty and the calculation should not be treated as calibrated.
Worked Example: Missing Outcomes After Accounting for Follow-Up Burden
Hypothetical data. A site has 90 patients due for a primary-outcome assessment. Based on a reference process, expected probabilities of a missing assessment are 0.05 for 40 patients with low follow-up burden, 0.10 for 30 with moderate burden, and 0.20 for 20 with high burden. The site has 18 missing assessments. Assume every patient due is included in the denominator, the burden categories are recorded consistently, and missingness is independent conditional on category.
The expected missing count is \(E=40(0.05)+30(0.10)+20(0.20)=2+3+4=9\). The variance is \(V=40(0.05)(0.95)+30(0.10)(0.90)+20(0.20)(0.80)=1.9+2.7+3.2=7.8\). Therefore, \(\sqrt{V}=\sqrt{7.8}=2.793\), and
There are nine more missing assessments than expected under the stated reference pattern. That is a substantial discrepancy for a screening comparison, but it is not evidence that outcome records were deliberately withheld. Check whether assessment windows, data entry practices, staffing, and the reference probabilities match the site's circumstances. Also consider whether missingness is concentrated in particular treatment groups or calendar periods; an overall count can conceal important patterns. Such follow-up is descriptive unless the design and analysis support a formal inferential conclusion.
What the Standardized Residual Does—and Does Not—Show
The residual \(O-E\) shows the direction and size of the difference in the original units: patients, events, or assessments. The standardized residual \(Z\) scales that difference by the model-based standard deviation, making comparisons across sites with different denominators more interpretable. Neither quantity explains why a difference occurred. A large absolute value can result from a real process difference, an inadequate adjustment, correlated observations, data errors, or chance.
The justification for the variance calculation follows directly from the assumptions. Let \(X_k\) equal 1 if opportunity \(k\) has the event and 0 otherwise. Then \(E(X_k)=p_k\) and \(\operatorname{Var}(X_k)=p_k(1-p_k)\). Since the count is \(O=\sum_k X_k\), linearity of expectation gives \(E(O)=\sum_k p_k\). If the \(X_k\) are independent, the variance of a sum is the sum of variances, so \(\operatorname{Var}(O)=\sum_k p_k(1-p_k)\). Grouping opportunities with common probabilities gives the formulas above. If outcomes are dependent, covariance terms must be added; ignoring positive dependence can make a site look more exceptional than it is.
A useful diagnostic sequence is to verify denominators and event definitions, inspect how the site and reference populations differ, and check patterns over time and across relevant subgroups. Review whether one staff member, shift, or period accounts for most of the difference. Look for data-quality problems such as delayed entry or inconsistent coding. If expected counts are small, avoid relying on a normal approximation. If many sites and outcomes are screened, some large residuals can arise by chance; a flag should be interpreted in the context of the number and purpose of comparisons, not as an isolated verdict.
Choosing a Comparison and Avoiding Overinterpretation
| Approach | Useful when | Main limitation |
|---|---|---|
| Crude site rate or count | As a quick descriptive check when denominators and populations are similar | Can confound site differences with case mix, volume, or workload |
| Stratified expected counts | Relevant groups and reference probabilities are available and understandable | Can become unstable with sparse groups; results depend on the reference |
| Regression or hierarchical model | Several covariates, unequal volumes, or site-to-site variation need to be represented | Requires defensible model assumptions; shrinkage and estimated uncertainty need explanation |
| Process and record review | A statistical flag needs interpretation against actual workflow and documentation | Requires reliable records and does not replace a sound comparison |
Crude rates are transparent but can be misleading when sites differ materially. Stratification makes the adjustment visible, but many narrow groups can leave too few observations to compare. Regression can use several predictors and represent between-site variation, but a more complex model is not automatically more credible: poorly measured workload or an incorrect functional form can produce misleading expectations. Process review supplies context; it should not be used to select a preferred statistical story after seeing the result.
A common mistake is to reason, “This site is more than two standard deviations from expectation, so the site has compromised trial integrity.” The calculation does not support that conclusion. It says that, under a specified reference model and its assumptions, the observed count is relatively far from the expected count. The correction is to report the observed and expected values, state the assumptions and uncertainty, check the model and operational context, and treat the result as a signal requiring evaluation—not as proof of cause, intent, or a breach.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A site has more protocol deviations than another site but also has twice as many eligible management opportunities. What should be compared, and what additional factors might matter?
- For 50 comparable opportunities with event probability 0.10, calculate the expected count and standard deviation under an independent Bernoulli model.
- Why can a site with a large standardized residual still have no integrity problem?
- What can go wrong if outcomes within a site are positively correlated but the calculation assumes independence?
Solutions. (1) Compare event rates or observed counts with expected counts using opportunity denominators, then account for relevant case mix and workload; raw counts alone are not comparable. (2) \(E=50(0.10)=5\), and \(V=50(0.10)(0.90)=4.5\), so the standard deviation is \(\sqrt{4.5}=2.121\) events. (3) The reference probabilities may be wrong, relevant case mix or workload may be unmeasured, data may be coded differently, or chance may explain the result; a residual does not establish cause or intent. (4) Positive dependence increases the variance of the total through positive covariance terms. Omitting those terms understates uncertainty and can exaggerate how unusual the observed count appears.