Prespecifying The Analysis Strategy

Why Prespecify the Analysis?

A target-trial protocol is not complete when it names a population, treatment strategies, follow-up period, and causal contrast. It must also say how the analysis will estimate that contrast. Without an advance plan, analysts can make seemingly reasonable choices after seeing the results: changing the outcome scale, dropping a difficult subgroup, selecting a model because it gives a smaller standard error, or reporting only the analysis with the preferred conclusion. Such flexibility makes results harder to interpret and can make nominal uncertainty summaries misleading.

Prespecification does not mean that one estimator is universally best, nor does it guarantee that the chosen analysis will be feasible or unbiased. It means recording the primary choices, their rationale, and rules for foreseeable problems before examining outcome results. The protocol should distinguish the scientific target from the statistical machinery used to estimate it.

Specify the Target Before the Estimator

The analysis plan should first restate the target population and causal contrast in operational terms. State who contributes to the target, which treatment strategies are compared, what outcome is measured, and at what follow-up time. Then name the effect measure: for example, a mean difference in outcome units, a risk difference in percentage points, or a risk ratio. These are different estimands, not interchangeable ways to report one number.

For a binary treatment \(A\), let \(Y^1\) and \(Y^0\) denote the outcomes that would be observed under the two specified strategies. For a population \(T\), a mean-difference estimand can be written as:

An estimator is a rule for using the observed data to estimate \(\psi_T\). The estimand does not change just because a different estimator is chosen. Conversely, changing the target population, outcome definition, follow-up horizon, or contrast can change the estimand even if the same software and model are used.

In observational data, a causal interpretation generally requires consistency, conditional exchangeability given the measured baseline covariates, and positivity for the treatment strategies within the target population. In words, observed outcomes must correspond to the specified strategies; the measured covariates must suffice to control confounding; and both strategies must be possible across the covariate patterns represented in the target. The analysis plan should state which assumptions support its causal interpretation. A fitted model or a favorable diagnostic does not, by itself, establish these assumptions.

Write Down the Primary Analysis Decisions

A useful plan is specific enough that another analyst could identify the primary analysis without asking which result the investigators prefer. At minimum, document the following decisions.

• Analysis population: State how eligibility and time zero determine who enters the analysis, and whether the target is a finite cohort or a broader population.
• Primary contrast and measure: Name the treatment strategies, outcome, follow-up horizon, target population, and effect scale.
• Estimator: Identify the primary estimation approach and the baseline variables or model components it uses. Give a rationale tied to the design and estimand.
• Censoring and unavailable outcomes: State what counts as censoring, how the primary analysis will address it, and what assumptions that approach requires.
• Treatment deviations: State whether the primary contrast concerns assignment or adherence to the specified strategies, and how deviations affect inclusion or analysis.
• Uncertainty and reporting: Name the standard-error or interval procedure, confidence level, any hypothesis test and significance level, and the units used in reporting.
• Diagnostics and contingencies: Specify checks relevant to the method and what will happen if a prespecified condition is not met.

These items should agree. For example, a protocol should not define an assignment-strategy contrast and then silently exclude people who later stop their assigned treatment. Nor should a primary estimator target a matched subset while the stated estimand is the effect in all eligible people, unless the analysis includes a justified procedure for connecting those targets.

Deviations, Censoring, and Uncertainty

Treatment deviations and censoring are distinct. A deviation occurs when a person’s treatment history departs from the specified strategy. Censoring occurs when outcome follow-up ends before the planned horizon or the outcome is otherwise not observed as planned. The protocol should say how each is handled, rather than treating every departure from the ideal data pattern as the same problem.

The planned handling depends on the contrast. An assignment-strategy contrast asks about outcomes under assignment to strategies, regardless of later adherence; a strategy-adherence contrast asks about following the strategies as specified. These questions require different analysis rules. The primary rule should be stated before outcomes are inspected. If censoring is expected, identify the planned approach and its identifying assumptions, including whether measured information is sufficient to account for differences in who remains observed. The exact method must suit the data and estimand; a protocol should not imply that a statistical adjustment automatically removes bias from unmeasured causes of censoring or outcome.

Uncertainty summaries also need to match the estimator and target. A confidence interval should use a standard error or variance procedure appropriate to the design and estimation steps, including any matching, weighting, or model fitting that contributes to the estimate. Where a data-adaptive procedure changes the analyzed target, uncertainty should reflect that feature rather than treating the final sample or matched set as if it had been fixed in advance. State whether the primary report will include a confidence interval, a p-value, or both. If a test is planned, specify \(H_0\), \(H_a\), and \(\alpha\); do not select a one-sided or two-sided test after seeing the direction of the estimate.

Diagnostics and Contingency Rules

Diagnostics assess whether the planned analysis is behaving as expected; they do not prove that causal assumptions hold. The relevant checks depend on the estimator. A plan may include checks of covariate balance after a matching or weighting procedure, overlap in treatment probabilities, convergence of a fitted model, the amount and pattern of censoring, and the number of observations contributing to each estimate. The protocol should say what will be reported and what action follows a serious problem.

A contingency rule should be defined before results are known. For example, if a prespecified estimator fails to converge, the protocol might name a fallback estimator and explain whether it targets the same estimand. If a key diagnostic signals inadequate support for estimating the target, the planned response might be to report that the primary estimate is not reliably identified from the available data, rather than to redefine the population after seeing the effect estimate. The choice depends on the scientific question, but the rule must not be “try alternatives until one gives the desired answer.”

Worked Uncertainty Summary

For an estimator with an approximately normal sampling distribution and a valid estimated standard error \(SE\), a two-sided 95% confidence interval is commonly calculated as estimate \(\pm 1.96SE\). This approximation requires that the normal-based interval be appropriate for the estimator and sample size; it is not a universal interval formula. Its use should therefore be specified and justified in the analysis plan.

The calculation follows directly from the standard-normal central 95% range, approximately \([-1.96,1.96]\). If the estimation error divided by its standard error is approximately standard normal, then the interval formed by adding and subtracting \(1.96SE\) captures the target in about 95% of repeated samples under the model and assumptions.

Primary, Secondary, and Sensitivity Analyses

A primary analysis answers the protocol’s main question. Secondary analyses address additional questions named in advance, such as a different outcome scale or a prespecified subgroup. Sensitivity analyses examine how results change under alternative defensible assumptions or analytic choices. Labeling an analysis “sensitivity” does not make an arbitrary change informative: the alternative should have a rationale, and its relationship to the primary estimand should be explained.

A comparison of results can be useful even when they differ. A large change after using an alternative censoring assumption, for example, signals that the conclusion depends on that assumption. It does not establish which estimate is correct. The primary analysis remains the one specified in advance, while alternative results help describe robustness and limitations.

Common Pitfall: Treating Prespecification as a Guarantee

A common mistake is to reason, “We wrote the analysis plan before looking at the results, so the estimate must be unbiased.” Prespecification limits outcome-driven selection; it does not make a misspecified model correct, create overlap where none exists, ensure that all confounders were measured, or remove bias from informative censoring. The correction is to distinguish protection against selective analysis from identification and estimation assumptions. Both matter, and neither substitutes for the other.

Another mistake is to treat every departure from the primary plan as misconduct. Data problems sometimes require a change. The appropriate response is transparent documentation: what changed, why it changed, whether the change was made before or after outcome results were examined, and how the changed analysis relates to the original estimand. Do not present an outcome-informed modification as if it had been the original plan.