Bootstrap Stability of Link-Sensitivity Conclusions

Why Bootstrap Link Sensitivity?

A conclusion can appear to change when the association link changes, but the observed change is only one estimate from one sample. Subjects differ, and another sample from the same target population could produce different association estimates under either link. A subject-level bootstrap asks a focused question: how often would the direction or practical interpretation differ across the candidate links under resampling of the observed subjects?

This is a stability assessment, not a contest that identifies the true link. It complements uncertainty within each link and the scientific comparison of link specifications. A bootstrap frequency describes the distribution generated by a specified resampling procedure, conditional on the observed data and the fitted-model process. It is not the probability that a link is correct or that an association is real.

Define What Counts as a Change

Association parameters from different links need not have the same units. A coefficient for a current-value link, for instance, is not automatically comparable in magnitude with a coefficient for a slope-based or cumulative link. Comparing raw coefficients can therefore confuse a change in scale with a change in scientific interpretation.

Before resampling, define a contrast that has a clear interpretation under each link. Let \(q_l\) denote the association quantity for candidate link \(l\), and let \(\Delta q_l\) be a prespecified contrast in that quantity. If the fitted hazard contribution is expressed through \(\alpha_l q_l\), define the contrast-scale log hazard ratio as \(\eta_l=\alpha_l\Delta q_l\), with hazard ratio \(\exp(\eta_l)\). Choose \(\Delta q_l\) so that it represents a meaningful comparison under that link; its scale may differ across links even when its scientific interpretation is aligned.

For example, a practical rule might label a contrast as meaningfully positive when its hazard ratio exceeds 1.25, meaningfully negative when it is below 0.80, and negligible when it lies between those cutoffs. These reciprocal cutoffs are symmetric on the log-hazard-ratio scale because \(\log(1.25)=-\log(0.80)\). Other thresholds may be more appropriate, but they must be chosen and justified in the scientific context before interpreting the bootstrap categories.

How the Paired Subject Bootstrap Works

For each replicate \(b\), sample subject identifiers with replacement and create a bootstrap data set from the selected subjects’ entire records. If a subject is selected more than once, each copy is treated as a separate bootstrap copy of that subject. Fit every candidate link to that same replicate and calculate the prespecified contrast-scale estimates \(\eta_{1}^{*(b)},\ldots,\eta_{L}^{*(b)}\). The asterisk marks a bootstrap estimate.

For two links, a direction disagreement indicator can be defined as \(I\{\eta_A^{*(b)}\eta_B^{*(b)}<0\}\). This records opposite nonzero signs. If zero estimates are possible, decide in advance how to handle them; for example, record zero as its own category rather than silently assigning it a positive or negative sign. Practical disagreement is recorded when the two estimates fall in different prespecified practical categories. Across \(B\) successful paired replicates, the estimated disagreement frequency is the number of disagreements divided by \(B\).

Here \(C_l^{*(b)}\) is the practical category for link \(l\) in replicate \(b\). The analogous direction-disagreement frequency replaces the category comparison with the sign-disagreement indicator. The notation \(\widehat{P}\) is a convenient label for the empirical fraction of bootstrap replicates; it does not make this frequency a probability that a scientific hypothesis is true.

Justification and Assumptions

The nonparametric bootstrap approximates repeated-sample variation by treating the empirical distribution of observed subjects as an estimate of the population distribution. If \(T\) is the statistic or fitted-model contrast of interest and \(F_n\) is the empirical distribution of the \(n\) subjects, the observed estimate is \(T(F_n)\). A bootstrap sample is drawn from \(F_n\), producing \(F_n^*\) and estimate \(T(F_n^*)\). The variation of \(T(F_n^*)\) around \(T(F_n)\) is used to approximate the sampling variation of \(T(F_n)\) around its population target.

This justification depends on the sampling unit being correct. Subjects should be independent draws from the target population, or the resampling must respect the actual design, such as sampling strata or clusters. Within each subject, repeated measurements and event information are dependent; resampling visits separately would break that dependence and does not approximate the sampling process for independent subjects. The fitted joint models and contrast definitions must also be prespecified and applied consistently. Bootstrap validity can be poor with very small samples, sparse events, severe censoring, weakly identified association parameters, or estimates near boundaries.

The pairing matters because the two fits in a replicate share the same resampled subjects. It controls the comparison for which subjects happened to be selected, so disagreement reflects the difference between link results on a common bootstrap data set. Using independent resamples for the two links adds avoidable resampling noise to the comparison and does not represent the intended paired analysis.

Summarizing the Bootstrap Results

A useful report separates at least three quantities: the proportion of replicates in which each link has a positive contrast, the proportion in which the links have opposite directions, and the proportion in which their practical categories differ. A cross-tabulation is especially informative because it shows whether disagreement is driven by reversals, by one estimate being negligible, or by both. Always show the number of replicates underlying the reported fractions.

Monte Carlo Error, Diagnostics, and Failure Modes

A finite number of bootstrap replicates adds Monte Carlo error to estimated frequencies. If \(K\) of \(B\) replicates meet a defined event, the estimated frequency is \(\hat{p}_{\mathrm{boot}}=K/B\). Conditional on the observed data and treating replicates as independent draws from the bootstrap procedure, its approximate Monte Carlo standard error is \(\sqrt{\hat{p}_{\mathrm{boot}}(1-\hat{p}_{\mathrm{boot}})/B}\). This quantifies numerical variability from using a finite \(B\); it is not a confidence interval for the population frequency of scientific disagreement.

Diagnostics should include convergence rates for each link, the number of usable paired replicates, and the reasons for any failures. Inspect whether failed fits cluster in particular resamples, such as those with very few events or little variation in a biomarker trajectory. Report failures rather than quietly omitting them: if only the easiest data sets converge, the successful-replicate distribution may be unrepresentative. A prespecified rule may count failed fits as unclassified and report their rate separately, or trigger a revised fitting strategy applied consistently. Do not change rules after seeing which option makes a preferred conclusion more stable.

Other warning signs include extreme estimates, repeated boundary estimates, a large share of undefined contrasts, and conclusions that shift markedly when the number of replicates or reasonable analysis choices change. A narrow Monte Carlo range does not rule out these problems. It only indicates that the estimated bootstrap frequency is numerically stable given the chosen procedure.

Alternatives and Limitations

A model-based standard error or confidence interval for each link describes uncertainty within that specification, but does not by itself summarize how often two link-specific classifications disagree under common resampling. Comparing separate confidence intervals is not a substitute for a paired comparison: overlap or non-overlap of intervals is not a direct test of the difference between estimates. A bootstrap distribution of a prespecified contrast difference can address that difference, while category cross-tabulations address the stability question emphasized here.

A parametric bootstrap instead simulates new subject records from a fitted model. It can be useful when the data-generating model is credible, but its results inherit that model’s assumptions. The subject-level nonparametric bootstrap resamples observed subjects and avoids specifying a full parametric distribution for the subject records, but it cannot create information absent from the observed sample. Neither approach resolves link misspecification. Sensitivity analysis across links remains conditional on the candidate models and other modeling choices.

The main limitations are small-sample instability, rare events, heavy censoring, dependence between subjects that is not handled by the resampling scheme, and difficulty aligning contrasts across links. Practical thresholds are scientific choices, not universal constants. If the contrast is poorly defined or changes meaning across links, a numerical category disagreement may be hard to interpret. The earlier material on patient risk rankings is related but addresses ranking stability; this tutorial instead focuses on bootstrap frequencies of association direction and practical-effect categories.

Common Pitfall

Wrong reasoning: “The association direction was positive in 53% of bootstrap replicates, so there is a 53% probability that the true association is positive.” This confuses a frequency generated by resampling the observed data with a probability statement about an unknown parameter. The frequency may also be sensitive to model assumptions, data quality, and the defined contrast.

Correction: State the result conditionally and precisely: “Under the specified subject-level bootstrap and contrast definition, 53% of successful replicates had a positive estimate for this link.” Then report the number of replicates, failures, practical thresholds, and paired disagreement frequency. Interpret these as evidence about resampling stability, not as posterior probabilities or proof of correctness.