Why the Timing of Weight Choice Matters
In a survival trial with delayed or changing treatment effects, different weighted logrank tests can emphasize different parts of follow-up. A weight that gives greater emphasis to later events may be attractive when effects are expected to emerge late; another weighting may respond more strongly to an early effect. The difficulty is not merely that the weight might be poorly chosen. If investigators inspect unblinded treatment comparisons and then select the weight that looks most favorable, the reported test has benefited from a data-driven search.
The relevant error rate is the type I error rate: the probability of rejecting \(H_0\) when \(H_0\) is true. A test designed to have significance level \(\alpha\) should have rejection probability no greater than \(\alpha\) under its stated null model. That guarantee applies to the complete analysis procedure, including decisions about which weight to use. Testing the selected statistic as if its weight had been fixed in advance generally ignores the selection step.
This is different from prespecifying weights across interim looks. Under a prespecified plan, the rule determining the weights is fixed without reference to unblinded treatment-effect results. A plan can still require careful calibration, but it does not incur the same unaccounted-for search over favorable weights.
The Selection Effect Under the Null
For a candidate weight \(w\), let \(U_w\) be its weighted logrank score and \(V_w\) its null variance estimate. Its standardized statistic is \(Z_w=U_w/\sqrt{V_w}\). Under the null and suitable large-sample conditions, each prespecified \(Z_w\) is approximately standard normal. For multiple candidate weights, however, their statistics are generally correlated because they are calculated from the same event data.
The core issue can be seen using two candidate statistics. Suppose, for illustration, that under \(H_0\), \(Z_1\) and \(Z_2\) are independent standard normal variables. If the analysis chooses the larger statistic and rejects for a one-sided \(Z\) greater than \(c\), then its rejection probability is not the single-test tail probability.
Here \(\Phi\) is the standard normal cumulative distribution function. At one-sided \(\alpha=0.025\), \(c\) is approximately 1.960 and \(\Phi(c)=0.975\). Searching over the two candidate tests therefore gives a null rejection probability of \(1-(0.975)^2=0.0494\), nearly twice the intended level. The calculation assumes independence for clarity; with correlated weighted logrank statistics, the exact increase depends on their joint null distribution. The general lesson remains: the selected statistic does not have the same distribution as one candidate statistic chosen in advance.
Worked Example: Choosing the More Favorable Weight
Suppose a hypothetical trial has two candidate weighted logrank tests. Under the null, assume their standardized statistics are independent \(N(0,1)\), and the analyst chooses the larger observed value. The planned one-sided significance level is 0.025.
The critical value for one prespecified test is \(c=1.960\), since \(P(Z>1.960)\) is approximately 0.025 for \(Z\sim N(0,1)\). For the selection procedure, both candidate results must be at or below 1.960 to avoid rejection:
Thus the procedure rejects about 4.94% of the time under this illustrative null model, not 2.5%. The data are hypothetical, and independence is an simplifying assumption, not a claim about the usual dependence between weighted logrank tests. The example shows why applying the ordinary 0.025 cutoff after selecting the more favorable result is not justified.
Why Prespecified Combinations Are Different
A plan may combine candidate statistics using coefficients fixed before unblinded results are examined. Suppose \((Z_1,Z_2)\) has a joint, approximately normal null distribution, with each statistic having mean zero, variance one, and correlation \(\rho\). For fixed coefficients \(a\) and \(b\), a standardized combination is
The denominator is the null standard deviation of the numerator. Because a fixed linear combination of jointly normal variables is normal, \(Z_C\) is approximately standard normal under \(H_0\), provided the assumed covariance is correct. A one-sided test can then use its prespecified critical value. The coefficients do not change in response to which candidate happened to look more favorable.
Prespecification alone is not a substitute for correct calibration. The joint null distribution and covariance must be appropriate for the statistics being combined, and the test must be defined in advance. If the coefficients are chosen after seeing the statistics, the fixed-combination argument no longer applies: the coefficients are then random and related to the observed results.
Worked Example: A Fixed Combination of Two Tests
Consider hypothetical standardized statistics \(Z_1=1.40\) and \(Z_2=0.20\), with null correlation \(\rho=0.60\). Before seeing results, the analysis plan set \(a=0.70\) and \(b=0.30\). Assume the joint null distribution is approximately bivariate normal with the stated correlation.
The standard deviation of the numerator is
The observed combination is therefore
For a one-sided test at \(\alpha=0.025\), this statistic does not exceed 1.960, so the null hypothesis is not rejected. Under the stated joint normal model and the prespecified coefficients, the combination is calibrated as one test; it is not equivalent to selecting whichever of \(Z_1\) and \(Z_2\) is larger and then using a single-test cutoff.
Using Interim Results Without Reusing Their Evidence
Adaptation does not always invalidate every later test. A key distinction is whether the unblinded interim evidence used to choose a weight is also included in the final confirmatory statistic. If the choice is made from earlier data and the final test uses only a later, independent increment, then the later test can be valid conditional on the interim history, provided its conditional null distribution is correctly specified and the testing rule is set out in advance.
The independence of increments is an approximation or design property that must be justified for the monitoring statistic and information structure. It does not mean that all statistics from the full trial are independent. In particular, adding the interim score back into the final statistic can restore the selection dependence. Nor does a valid later test automatically validate any separate interim rejection rule: every opportunity to reject must be covered by the overall testing plan.
Worked Example: Selecting a Weight, Then Testing New Data
In a hypothetical trial, investigators use interim data only to choose between two prespecified weight functions. They do not use that interim score in the confirmatory statistic. Assume that, under \(H_0\), the later standardized score for whichever weight was selected has a standard normal distribution conditional on the interim data. This assumption requires valid independent future increments and correct conditional variance estimation.
Suppose the later score is \(Z_{\mathrm{future}}=2.05\). At one-sided \(\alpha=0.025\), its p-value is approximately \(1-\Phi(2.05)=0.0202\), which is below 0.025. Because the future score has the stated conditional null distribution for each weight and the selected weight is determined before those future data are observed, the later test has conditional type I error 0.025. Averaging over interim histories also gives type I error 0.025 for this single later test.
This conclusion would not follow if the interim score were added to the final statistic, if the choice also used later outcomes, or if unaccounted-for interim rejection opportunities were included. The example illustrates a conditionally valid strategy, not a general permission to change the analysis after viewing results.
Assumptions, Diagnostics, and Practical Comparisons
The calculations above rely on assumptions that should be made explicit in a design. Standardized candidate statistics must have the stated null distributions; their dependence must be represented when a combination or calibration requires it; and the timing and information used for adaptation must be recorded. For weighted logrank monitoring, the event process, censoring, variance estimation, and information accrued at each look also affect the quality of the normal approximation. A nominal cutoff is not protected from a misspecified null distribution.
Useful diagnostics include reviewing the analysis specification and interim decision record, checking that the weight-selection rule used only the information it was permitted to use, and examining the joint null behavior of the candidate statistics under the planned event and censoring patterns. If null calibration is assessed by simulation, the simulation must reproduce the entire adaptation procedure: generate data under \(H_0\), apply the interim selection rule, calculate the final statistic exactly as planned, and estimate the proportion of simulated trials that reject. Simulating only one fixed weight would miss the selection effect.
| Approach | What determines the analysis? | Type I error implication |
|---|---|---|
| One weight fixed in advance | A prespecified weight and test statistic | Use its null calibration, subject to model and approximation assumptions |
| Fixed combination | Coefficients fixed before unblinded results, with the relevant covariance accounted for | A properly standardized combination can have a calibrated null distribution |
| Choose the best-looking weight and reuse interim data | Unblinded observed treatment differences influence the selected test | A single-test cutoff generally ignores selection and can inflate type I error |
| Choose using interim data, test only later increments | Interim choice is separated from confirmatory future data | Can be valid if the future test is conditionally calibrated and all rejection opportunities are addressed |
Blinded adaptation is also distinct from choosing a weight based on unblinded treatment differences. For example, an adjustment based only on pooled event-rate or censoring information may avoid direct selection for a favorable treatment contrast. That does not guarantee validity automatically: the adaptation rule, estimand, variance calculation, and null calibration still need to be examined. The main practical alternatives are to retain a prespecified weight plan, use a prespecified combination with appropriate covariance, or design a properly separated future-data test. Each has different power and interpretability trade-offs.
A frequent failure is to call a rule “prespecified” and infer that its type I error must be controlled. A rule can be written down in advance yet still choose the analysis based on unblinded interim evidence and reuse that evidence. Prespecification makes the procedure reproducible; it does not make a selected statistic follow the null distribution of a single fixed statistic. For an adaptive rule that reuses interim evidence, calibration must apply to the full rule, not just to its final formula.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under the independent-null illustration, what is the rejection probability when the larger of two standard normal statistics is compared with the one-sided 0.025 cutoff of 1.960?
- Why does a fixed linear combination require the covariance between its component statistics?
- An analyst chooses a weight after examining unblinded interim results, then combines the selected interim score with later data using an ordinary single-test cutoff. What is the central validity concern?
- Under what conditions can choosing a weight at interim and testing only future increments preserve the conditional type I error rate?
- Why should a null simulation reproduce the weight-selection rule rather than simulate a single fixed weight?
Solutions. (1) \(1-(0.975)^2=0.0494\), under the stated independence assumption. (2) The covariance contributes the term \(2\rho ab\) to the variance of \(aZ_1+bZ_2\); omitting it generally gives the wrong standardization. (3) The interim score both guided selection and is reused, so the final statistic's null distribution reflects selection and is not generally the ordinary fixed-test distribution. (4) The choice must be made from interim information before the future data are observed, and the future test must have a correctly specified conditional null distribution; any other rejection opportunities must also be covered by the testing plan. (5) Selection changes the distribution of the reported statistic, so omitting selection from the simulation fails to estimate the type I error of the actual procedure.