Why audit the package as a connected record?
An adaptive-trial regulatory package is not simply a set of individually polished documents. The protocol, statistical analysis plan, simulation report, committee charter, and study report describe different parts of one decision process. If they disagree about when interim data are available, who may review them, or what action follows a criterion, a reviewer cannot reliably reconstruct how the trial was meant to operate or what actually happened.
An audit asks whether the documents are complete enough to establish the planned rules, whether the evidence supports claims made about those rules, and whether the recorded conduct can be reconciled with the plan. It is a structured review of traceability and consistency, not a substitute for statistical review, inspection, or a conclusion about intent. A discrepancy warrants investigation; by itself, it does not establish that a trial decision was biased or that a result is invalid.
The audit assumes that the reviewer has identified the applicable, approved document versions and can distinguish them from drafts. If that cannot be established, version control is itself a finding: apparent inconsistencies may reflect different versions, and the reviewer cannot yet decide which text governed the trial. Earlier material on change control and amendments is useful here: a later version should have an identifiable approval status, effective date, and relationship to the version it replaces.
A practical cross-document checklist
Start with a single adaptation and trace it through the package rather than reading five documents independently and hoping discrepancies stand out. Record the exact location and version of each statement. The audit should establish what was prespecified, what the simulation evaluated, who had authority to decide, what evidence was recorded, and what the study report says occurred.
| Document | Audit question | Evidence to locate |
|---|---|---|
| Protocol | What adaptation is permitted and under what conditions? | Rule, timing, eligible data, allowed actions, and constraints |
| Analysis plan | How are the data and decision criteria operationally defined? | Endpoint, analysis population, cutoff, calculations, and handling of missing data |
| Simulation report | What properties were evaluated for the planned design? | Scenarios, assumptions, implementation of the rule, operating-characteristic estimates, and uncertainty |
| Committee charter | Who reviews the data and how is a valid decision made? | Membership, access, quorum, voting or recommendation process, and communication route |
| Study report | What happened, and can it be matched to the plan and records? | Interim timing, recommendation, decision, deviations, implementation, and analyses |
For each rule, make a compact traceability row: rule identifier; prespecified trigger; data cutoff and population; source document and version; simulation evidence; decision authority; record of the decision; and study-report description. Note whether each item is present and whether its meaning agrees across documents. A vague statement such as “the committee may adapt the study” is not equivalent to a fully specified adaptation rule: it does not identify the permitted action, the information used, or the decision conditions.
A useful audit distinguishes three levels of finding. A critical gap prevents reconstruction of a key decision or leaves its authority unclear. A material inconsistency gives conflicting instructions or descriptions that could change implementation or interpretation. An editorial difference changes wording but not meaning, such as an abbreviation defined differently without affecting the rule. Classifying findings this way helps reviewers focus on consequences rather than treating every typo as equally important.
Worked Example: Reconciling an Interim Data Cutoff
Suppose the controlled protocol states that the first interim review occurs when 120 participants have been randomized. The analysis plan says “120 evaluable participants,” while the simulation report says its interim was triggered at 120 randomized participants. At the review, the trial record shows 124 randomized participants, of whom 116 meet the analysis plan’s stated evaluability criterion.
Step 1: Identify the conflict. The protocol and simulation report use a randomized-participant count; the analysis plan uses an evaluability condition. These are not interchangeable if some randomized participants are not evaluable.
Step 2: Compare the stated trigger with the record. The record has reached 120 randomized participants because \(124 \geq 120\). It has not reached 120 evaluable participants because \(116 < 120\). Thus, the interim is due under one wording and not yet due under the other.
Step 3: State the audit finding precisely. The audit cannot establish a unique prespecified cutoff from these documents. It should request the governing approved rule, the operational definition of evaluability, and the dated record showing which data entered the review. The simulation evidence also needs reconciliation if its trigger differs from the operational rule.
This finding does not, by itself, show that the committee saw inappropriate data. It identifies a decision-relevant ambiguity that must be resolved before the timing and scope of the review can be confirmed.
Check the rule, its evidence, and its authority
The adaptation rule should be sufficiently operational that two qualified reviewers given the same specified information would identify the same eligible action, apart from any discretion explicitly assigned to the decision-maker. Check that the package defines the decision variable, cutoff, analysis population, timing, missing-data handling where relevant, and the actions permitted for each decision region. Confirm that any discretion is bounded and assigned to an identified role. Earlier material on prespecified rules and decision criteria provides the underlying standard: a rationale for an adaptation does not replace a rule that can be implemented.
Then check whether the simulation report evaluated the design that the protocol and analysis plan actually specify. At minimum, compare the simulated decision rule, timing, population, assumptions, and candidate actions with the governing documents. A report may contain many scenarios and still fail to support a claim about the planned design if the implemented rule differs. When the report makes a numerical claim, reconcile the claim with its tables, counts, and stated simulation uncertainty. Avoid extending conclusions beyond the scenarios and assumptions that were evaluated.
Worked Example: Checking a Simulation Claim Against Its Counts
A hypothetical simulation report states that a design’s estimated type I error rate was 0.048, based on 10,000 simulated trials under a specified null scenario. The corresponding table shows 520 rejections. The claimed rate and the table cannot both describe that same set of simulations.
The table’s rate is
A rate of 0.048 with 10,000 simulations would correspond to \(0.048 \times 10{,}000=480\) rejections. The difference is 40 rejections, or \(40/10{,}000=0.00400\) in the estimated rate. The approximate Monte Carlo standard error at a rate near 0.05 is
The discrepancy is therefore about \(0.00400/0.00218=1.83\) approximate Monte Carlo standard errors. That calculation describes simulation variability; it does not explain away an arithmetic disagreement between the report’s claim and its own table. The audit should ask which count, calculation, or report version is correct, and require the stated rate and supporting table to agree before relying on the claim.
Review authority separately from statistical specification. A charter should identify who may access interim information, whether the committee makes a recommendation or a binding decision, what constitutes a quorum, and how the decision is communicated to operational personnel. Compare these terms with the minutes and the study report. A committee’s recommendation and a sponsor’s implementation decision are not necessarily the same event; the package should make the distinction visible.
Worked Example: Testing a Decision Against the Charter
Suppose a hypothetical charter says that five voting members are appointed, at least four must participate for a quorum, and the committee recommends whether the trial continues. The minutes record three members participating in a vote to stop recruitment. The study report says, “The committee decided to stop the trial.”
The participation count is \(3\), while the charter’s quorum requirement is \(4\). Since \(3 < 4\), the recorded meeting did not meet the charter’s stated quorum. In addition, the charter assigns a recommendation to the committee, whereas the report describes the committee as making the final decision.
These are two distinct inconsistencies: the documented process does not meet the stated participation condition, and the report may overstate the committee’s authority. The audit should locate any applicable charter amendment, additional attendance record, sponsor decision record, or other governing document. If none resolves the discrepancy, report that the package does not establish a charter-compliant recommendation or identify the final authorized decision-maker. Do not infer from the discrepancy alone that the trial’s final action was necessarily improper.
Reconstruct what happened and classify the findings
The study report should be checked against contemporaneous records, not treated as the sole evidence of events. For each adaptation, compare the planned trigger and cutoff with the data cutoff record, committee materials and minutes, sponsor or delegated decision record, implementation date, and report narrative. Confirm that any deviation is described with its reason, timing, authorization, and consequences for analyses or interpretation, as applicable. If the report says a rule was followed but the records do not establish the trigger or decision path, the audit should say the claim is unsupported by the available record rather than asserting that the rule was not followed.
A short audit sequence keeps the review reproducible:
List the controlled versions, approval status, and effective dates. Resolve drafts or amendments before treating a text as governing.
Record its trigger, data, timing, allowable action, decision role, and any stated constraints in exact operational terms.
Compare implementation in the simulations, review arrangements in the charter, and records of the actual decision.
Match the study report’s account to the approved plan and contemporaneous records; identify any unresolved difference explicitly.
State the evidence, the consequence for interpretation or reproducibility, and what clarification or correction is needed. Do not claim more than the record supports.
Audit conclusions should separate fact from implication. “The plan specifies 120 evaluable participants, but the simulation uses 120 randomized participants” is a factual discrepancy. “The reported operating characteristics may not describe the implemented rule” is a consequence that warrants verification. “The adaptation invalidated the trial” is a much broader conclusion and cannot be justified by that discrepancy alone without further statistical and regulatory assessment.
Common pitfalls and limits
- Assuming one document automatically overrides the others. A protocol, approved amendment, analysis plan, and charter have different functions. Establish which version governed the relevant decision and whether an authorized change occurred; do not select the most convenient wording.
- Treating a rationale as a decision rule. A rationale explains why an adaptation may be useful. It does not specify the cutoff, calculation, conditions, or authority needed to apply it.
- Reading simulation results as a guarantee. Simulations estimate behavior under specified assumptions and scenarios. A favorable result does not establish performance in unexamined conditions or prove that the operational implementation matched the simulated design.
- Equating missing documentation with evidence of misconduct. Missing minutes or inconsistent dates limit what can be reconstructed. They justify a targeted query and a qualified finding, not an unsupported conclusion about intent.
- Fixating on wording while ignoring decision consequences. Prioritize inconsistencies that change eligibility, timing, data access, permitted actions, authority, or the meaning of reported results. Record editorial issues, but do not confuse them with substantive gaps.
An audit cannot recreate information that was never recorded, establish the truth of every source record, or replace review of whether the design’s statistical methods are appropriate. It can make the evidentiary boundary clear: what the documents establish, what they contradict, and what remains uncertain. This is especially important when a package makes claims about operational integrity or adaptation performance.
Check Your Understanding
For each question, identify the audit issue and state what evidence or calculation would resolve it.
- The protocol specifies review after 100 participants complete follow-up; the simulation report triggers after 100 are randomized. Why does this matter?
- A report claims 6.0% of 5,000 simulated trials met a criterion, while its table lists 275. Calculate the table’s proportion and identify the discrepancy.
- A charter requires four members for a quorum. The minutes list four attendees, but one is described as a non-voting observer. What should an auditor check?
- The study report says the prespecified adaptation rule was followed, but no dated data-cutoff record is available. What conclusion is justified?
- Why is it not enough to confirm that the simulation report contains favorable results?
Solutions. (1) Completion and randomization can yield different participant sets and interim timing; determine the governing trigger, operational definition of completion, and simulation implementation. (2) The table gives \(275/5{,}000=0.0550\), or 5.50%; this differs from the claimed 6.0%, which would correspond to \(0.060 \times 5{,}000=300\) trials. Reconcile the count, calculation, or report version. (3) Check the charter’s definition of voting membership and quorum, the observer’s status, and any attendance or charter amendment record; four attendees do not necessarily mean four eligible members. (4) State that the available record does not independently establish the cutoff or verify the report’s claim; request contemporaneous records. Do not infer that the rule was violated solely from the missing record. (5) The result supports only the scenarios and implementation actually simulated. Compare the simulation’s rule, timing, assumptions, and actions with the governing documents, and check that numerical claims agree with the report’s own counts.