Writing a Locked Cross-Cohort Validation Protocol

Why Lock the Protocol Before Validation

A validation cohort can provide evidence about whether a defined claim holds beyond the data used to develop it. That evidence is harder to interpret if the analysis changes after investigators see which endpoints, exclusions, or models produce favorable results. A locked protocol makes the planned claim and decision process explicit before validation outcomes are examined. It does not guarantee that a study is unbiased or successful; it makes it possible to compare the analysis that was planned with the one that was carried out.

The earlier material on defining the validation claim and target population supplies the scientific question to be tested. This tutorial focuses on converting that question into instructions that are precise enough to follow, audit, and report. Locking should occur before access to validation-cohort outcomes or outcome-linked omics results that could influence decisions. Administrative or feasibility information may be needed to prepare a protocol, but investigators should distinguish such information from results capable of revealing whether the validation claim appears to hold.

Locking is not the same as promising never to change anything. Unexpected data problems can require a change, and a protocol may have to be amended. The safeguard is to retain the original plan, document the amendment and its timing, and avoid presenting a post-result change as if it had been prespecified.

What the Protocol Must Make Unambiguous

A protocol should let another analyst determine which participants contribute, which quantity is estimated, how it is estimated, and what result would count as success. Vague instructions such as “adjust for relevant covariates” or “exclude poor-quality samples” leave consequential decisions open until the data are visible. State rules in operational terms, including how an unavailable variable, a failed assay, or an ambiguous outcome is handled.

• Claim and population. State the validation claim and the population to which it applies. Specify the cohort and any prespecified eligibility conditions.
• Hypothesis and estimand. Name the primary quantity of interest, its scale and direction, and the null and alternative hypotheses. An estimand should identify the population quantity, not merely name a statistical test.
• Endpoint. Define exactly what is measured, how it is coded, and the time point or window. If there are multiple endpoints, identify which is primary and which are secondary or exploratory.
• Analysis. Give the model or calculation, covariates and their coding, the uncertainty measure, and the method for handling missing values. State assumptions needed for the planned inference.
• Exclusions and data handling. Specify eligibility and quality-control rules, when they are applied, and how overlapping reasons for exclusion are counted. Explain how an unavailable endpoint or predictor affects analysis inclusion.
• Success criteria. State a decision rule in advance: for example, the required direction and a threshold for an effect estimate, interval, or classification measure. If several conditions must hold, say whether all are required.
• Reporting and deviations. Specify which planned results will be reported, including unfavorable or inconclusive results. Record departures from the plan and explain their timing and rationale.

The assumptions belong beside the analysis they support. For example, a regression-based inference may rely on independent participants, an appropriately specified mean model, and a valid standard error. A classification proportion requires clearly defined case and noncase groups and consistent application of the endpoint and classification rules. If assumptions are doubtful, the protocol should specify a diagnostic and a principled response rather than leave investigators to choose a favorable alternative after seeing results.

Worked Examples

Justifying the Decision Rule and Auditing the Plan

A success criterion should follow from the scientific claim, not from a threshold selected because it makes the validation result look favorable. The effect scale must match the intended interpretation: a standardized mean difference is not an unstandardized unit difference, and a proportion of correctly classified cases is not a measure of association. When uncertainty is part of the claim, specify how it enters the criterion. For example, requiring a confidence bound to exceed a minimum effect is not interchangeable with requiring only that the point estimate exceed it.

The logic of a threshold-based test illustrates why the distinction matters. For \(H_0:\beta\leq\delta\) versus \(H_a:\beta>\delta\), with an approximately normal estimator and valid standard error, the test statistic at the boundary is \(z=(\hat{\beta}-\delta)/SE\). At a one-sided level \(\alpha\), rejection requires \(z>z_{1-\alpha}\), or equivalently \(\hat{\beta}-z_{1-\alpha}SE>\delta\). Thus a large-looking estimate may still fail if its uncertainty is substantial. This rule controls the probability of rejection at or above the null boundary at no more than the chosen level under the stated approximation and assumptions; it does not give the probability that the null hypothesis is true.

Before locking, audit the document for decision points that could be made differently after results are seen. Verify that endpoint windows, covariate coding, sample exclusions, missing-data rules, and success conditions have one unambiguous interpretation. Test the analysis code on simulated or otherwise non-validation data where possible, and confirm that it implements the written plan. Keep a dated protocol and analysis-plan version, and retain a record of when analysts gained access to outcomes. These steps are quality controls, not substitutes for a sound design.

Alternatives, Diagnostics, and Limitations

A locked protocol is related to, but distinct from, several other practices. A registration can publicly timestamp key decisions, while a detailed statistical analysis plan can specify implementation details such as model coding and output tables. These documents can support one another, but a registration alone may be too brief to resolve operational ambiguity. In contrast, an adaptive analysis deliberately permits specified decisions to depend on emerging data; it is not equivalent to an analysis whose rules were fixed in advance. If adaptation is intended, its conditions and decision process need to be written before the relevant results are available.

Useful diagnostics include checking that exclusions follow the declared order, that reported denominators reconcile with cohort counts, and that the endpoint and analysis variables were coded as specified. Compare the executed analysis with the locked version and log every deviation. A discrepancy is a signal to investigate, not a reason to silently revise the protocol. If an assumption fails or a required variable is unavailable, report the issue and its consequences. A prespecified fallback may be used if its trigger and method were specified in advance; otherwise, present any changed analysis as an amendment or sensitivity analysis, with the original planned result still clearly identified where it can be computed.

The protocol cannot repair participant overlap, poor measurement, an unsuitable target population, or inadequate information in the cohort. Nor can a locked plan guarantee that the chosen endpoint captures the intended biological concept. The earlier material on distinguishing validation from replication remains relevant: a protocol is a framework for evaluating a stated claim, not evidence that the claim is true. Transparent documentation makes the evidence easier to assess, including when the result is inconclusive.

Common Pitfalls

• Changing the endpoint after seeing results. Choosing the most favorable time point or measurement turns a post-result choice into an apparent planned test. Define the primary endpoint and time window in advance; label later alternatives as exploratory or amended.
• Using vague exclusion language. “Remove low-quality samples” does not identify a reproducible rule. Specify the quality metric, cutoff, timing, and treatment of borderline or missing values.
• Equating a positive estimate with success. A point estimate above a threshold may be too uncertain to support the stated claim. Prespecify whether success depends on an interval, a test, a point estimate, or a combination.
• Reporting only the successful condition. If the rule requires two endpoints or performance measures, report both, even when one fails. A joint rule cannot be rewritten after results are known.
• Calling a post-access revision prespecified. A reasonable amendment may still be useful, but its timing matters. Preserve the original plan and distinguish confirmatory results from analyses motivated by observed validation data.