Calibrating Delta Values With External Evidence

Why external evidence can help calibrate delta

An additive delta is not estimated from the observed outcomes in the target study alone. It represents a sensitivity assumption about how outcomes in a missing-data pattern differ from the reference-model predictions for those outcomes. External trial, registry, or follow-up data can make that assumption more concrete, but only if the external evidence addresses a sufficiently similar contrast. A reported treatment effect, a source-population mean, or an outcome measured at a different time is not automatically the target delta.

The task is therefore one of calibration, not identification. External evidence can indicate a defensible magnitude or direction under stated transport assumptions. It cannot, by itself, reveal the unobserved target outcomes or prove that the target missingness mechanism follows the external study.

Definition: Under an additive delta adjustment, let \(m_{\mathrm{ref},T}(x,t)\) be the target reference-model mean for outcome \(Y\), at covariates \(X=x\) and time \(t\). For the missing-data pattern of interest, the delta-adjusted mean is \(m_{\mathrm{ref},T}(x,t)+\delta_T(x,t)\). Thus \(\delta_T(x,t)\) is the mean difference, in the outcome’s units, between the target missing outcomes and the target reference-model prediction at \(x,t\). A negative delta means a lower mean than the reference prediction; a positive delta means a higher mean.

The reference prediction is the benchmark already specified by the target analysis. The delta concerns the discrepancy from that benchmark, not necessarily a treatment effect or a difference between two raw group means. The same sign convention applies regardless of whether higher outcomes are clinically better or worse; clinical direction must be interpreted using the outcome’s meaning.

What must match for external evidence to inform a delta?

Before using a source estimate, state the target contrast precisely: which missing-data pattern, population, outcome definition, and observation time are in view, and what reference prediction is being modified? Then ask whether the external data estimate a comparable discrepancy. A randomized withdrawal study may provide a useful contrast between outcomes after stopping treatment and outcomes under continued treatment if the target reference model predicts continued-treatment outcomes. A registry comparison of people who stopped and continued treatment may also be informative, but treatment stopping may be associated with prognosis; its raw difference can then combine the discrepancy of interest with confounding.

Three kinds of correspondence deserve particular attention:

Population: The external study should include people relevant to the target missing pattern. Differences in baseline severity, treatment history, prognosis, or other effect modifiers can change the contrast. Where covariates are measured comparably and there is adequate overlap, standardize conditional source contrasts to the target covariate distribution.
Outcome: The measure, units, direction, and interpretation should be comparable. A change in instrument or scale requires a defensible conversion, not a label-based assumption that the measures are interchangeable.
Observation period: The source contrast should refer to the target time or to a period that can be translated using credible trajectory information. A contrast at 12 weeks need not represent the contrast at 8 weeks.

These checks also apply to the reference. If the external study compares stopping with continuing treatment but the target reference model predicts a different treatment strategy, its contrast may not estimate the relevant departure from that reference. State what is being borrowed, and what additional assumptions are needed to use it.

Standardizing a source contrast

Suppose an external study estimates a conditional discrepancy \(d_S(x)\) between an outcome under a source analogue of the target missing pattern and the corresponding source reference prediction. If the conditional discrepancy can reasonably be transported at each \(x\), a target-population calibration is the target-weighted average:

\delta_T=\sum_{k=1}^{K} w_{T,k}d_S(x_k), \qquad \sum_{k=1}^{K}w_{T,k}=1.

Here \(w_{T,k}\) is the proportion of the relevant target population in covariate stratum \(k\). This calculation is justified when the strata have comparable definitions, source data support the target strata, and the source conditional discrepancies are transportable. It does not correct unmeasured differences or a wrongly defined source contrast.

The reason for weighting is direct. The target mean discrepancy is the average of the stratum-specific discrepancies over the target distribution. For discrete strata, partitioning that expectation gives \(E_T\{d_S(X)\}=\sum_k P_T(X=x_k)d_S(x_k)\), which is the displayed formula. Using source-population weights instead answers a different question: it averages the discrepancies for the source population, not the target one.

Worked Example: A Comparable External Withdrawal Contrast

In a hypothetical randomized withdrawal study, the mean symptom score at 8 weeks is 42.0 points among participants assigned to stop treatment and 46.0 points among those assigned to continue. Suppose the target reference model predicts continued-treatment outcomes, and the target outcome is the same score at 8 weeks. Assume, for this calibration exercise, that the source population and target missing pattern are sufficiently comparable and that assignment and follow-up make the randomized contrast appropriate for the intended source analogue.

The external estimate of the stopping-versus-continuing contrast is:

\widehat{d}_S=42.0-46.0=-4.0\text{ points}.

The outcome units, time, and reference strategy align, so a provisional target calibration is \(\delta_T=-4.0\) points. For example, if the target reference-model mean for a relevant covariate profile is 46.0 points, the delta-adjusted mean is \(46.0+(-4.0)=42.0\) points. This is a transported sensitivity assumption, not an observation of that target pattern’s missing outcomes.

Suppose the external study reports a standard error of 1.2 points for the contrast. Its approximate 95% confidence interval is \(-4.0\pm1.96(1.2)\), or \([-6.35,-1.65]\) points. This interval describes sampling uncertainty for the source contrast under its analysis assumptions. It does not include uncertainty about whether that contrast transports to the target population or represents the target missing-data discrepancy.

Worked examples of population and time translation

Worked Example: Reweighting Source Strata to the Target Population

A hypothetical registry reports adjusted stopping-versus-reference discrepancies for two severity strata. In the lower-severity stratum, the estimate is \(-2.0\) score points; in the higher-severity stratum, it is \(-6.0\) points. Assume the conditional estimates are comparable to the target contrast, there is source data in both target strata, and measured covariates suffice for the intended standardization. The target population is 75% lower severity and 25% higher severity.

Weighting the stratum estimates by target proportions gives:

\widehat{\delta}_T =0.75(-2.0)+0.25(-6.0) =-1.5-1.5 =-3.0\text{ points}.

The source population has different proportions: 40% lower severity and 60% higher severity. Its unstandardized average would be \(0.40(-2.0)+0.60(-6.0)=-4.4\) points. That value describes the source mixture, not the target mixture. Reweighting changes the calibration because the stratum-specific discrepancies differ.

If the stratum estimates have standard errors 1.0 and 2.0 points and are independent, treating the target weights as fixed gives an approximate standard error of \(\sqrt{0.75^2(1.0^2)+0.25^2(2.0^2)}=\sqrt{0.8125}=0.901\) points. This calculation reflects sampling uncertainty in the two estimates under the stated independence assumption. It does not account for uncertainty in transportability, stratum definitions, or the target weights.

Worked Example: Translating a Contrast Across Follow-Up Times

A hypothetical follow-up study estimates a stopping-versus-reference discrepancy of \(-4.8\) points at 12 weeks on the same outcome scale used by the target study. The target analysis concerns 8 weeks. Over weeks 8 to 12, the source stopping group’s mean changes by \(-1.0\) point, while the source reference group’s mean changes by \(-0.4\) point. Assume these changes apply to the analogous target groups and that the trajectory is sufficiently comparable for this translation.

The discrepancy changes over this interval by the difference in group changes:

(-1.0)-(-0.4)=-0.6\text{ points}.

Thus the 12-week contrast equals the 8-week contrast plus \(-0.6\) points. Solving for the target-time contrast gives \(-4.8-(-0.6)=-4.2\) points. The provisional 8-week calibration is therefore \(\delta_T=-4.2\) points, not \(-4.8\). This translation depends on the specified trajectory assumptions; if the source groups’ changes are not relevant to the target, the arithmetic does not make the transferred value valid.

Outcome mismatch, uncertainty, and practical diagnostics

When the external outcome is measured on a different scale, convert only when the relationship between measures is defensible for the population and range of interest. For example, if a hypothetical instrument is exactly a linear rescaling in which 10 points on the source scale correspond to 1 point on the target scale, a source discrepancy of \(-6.0\) points corresponds to \(-0.60\) target-scale points. This arithmetic is not justified merely because both instruments measure “symptoms.” Different item content, thresholds, or scale behavior may make a constant conversion unsuitable. If the source outcome cannot be linked credibly to the target outcome, it may inform direction or broad context, but should not be presented as a directly calibrated delta.

Keep sampling uncertainty distinct from transport uncertainty. A narrow source confidence interval can coexist with substantial uncertainty about population differences, confounding, outcome harmonization, or time translation. Report the source estimate and its sampling precision, then state the assumptions and any adjustments used to propose the target value. Do not describe a source confidence interval as covering all plausible target deltas unless the analysis explicitly includes the relevant transport uncertainties.

Useful checks include confirming that every target covariate stratum has source support, comparing source and target covariate distributions, examining whether the contrast varies across important strata, and verifying outcome units and follow-up definitions. Recalculate the standardized value under reasonable alternative target weights when those weights are uncertain. For time translation, check that the group-specific changes are observed over the needed interval and that their direction and magnitude are coherent with the outcome scale. These checks can expose fragile borrowing; they cannot test the unobserved target missing outcomes directly.

External calibration differs from choosing a delta solely as a hypothetical scenario: the former uses an empirical source contrast to motivate a value, while the latter need not rely on external data. It also differs from directly inserting an external treatment effect. The external quantity is relevant only insofar as it approximates the target discrepancy from the specified reference prediction. In either case, the target data do not identify the delta just because an externally motivated value has been selected.

Common mistakes and limitations

Using a raw source mean as the delta. A mean is not a discrepancy. Subtract the relevant source reference prediction, and verify that this reference corresponds to the target model’s benchmark.
Using source weights for a target question. If conditional discrepancies vary by covariate strata, average them using the target distribution when the desired calibration is for the target population.
Calling an observational contrast causal. In a registry, stopping and continuing groups may differ for reasons related to outcomes. Adjustment for measured covariates does not guarantee that all such differences have been removed.
Ignoring a mismatched observation period or instrument. A numerical contrast is not comparable merely because its units look similar. Translation needs explicit, defensible assumptions.
Treating external precision as transport certainty. A small standard error quantifies sampling precision under the source analysis, not the validity of applying the result to the target.

External evidence may be sparse, poorly harmonized, or unrepresentative of the target missing pattern. Even careful standardization cannot solve a lack of overlap or an unmeasured difference that changes the contrast. The calibrated delta should therefore be reported as an assumption informed by evidence, with its target, reference, units, time, and transport rationale made explicit.

Key takeaway: Calibrate a delta by first defining the target missing-versus-reference discrepancy, then selecting an external contrast that represents it, aligning population, outcome, and time, and stating what remains uncertain about transport.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

An external study estimates a missing-pattern analogue minus reference contrast of \(-3.5\) points at the target time. Under a stated transport assumption, what additive delta does it suggest, and what does the negative sign mean?
Two source strata have discrepancies of \(-1\) and \(-5\) points. The target weights are 60% and 40%. Calculate the target-standardized discrepancy.
A source contrast is \(-5.0\) points at 10 weeks. The difference in group changes from weeks 6 to 10 is \(-0.8\) points. What is the implied 6-week contrast under the same trajectory assumption used in the tutorial?
Why is a source estimate with a very small standard error not necessarily a precise estimate of the target delta?

Solutions. (1) The suggested value is \(\delta_T=-3.5\) points. It places the target missing-pattern mean 3.5 points below its reference-model prediction, if the transport assumption is adequate. (2) \(0.60(-1)+0.40(-5)=-0.6-2.0=-2.6\) points. (3) The 10-week contrast equals the 6-week contrast plus the differential change, so the implied 6-week contrast is \(-5.0-(-0.8)=-4.2\) points. (4) The standard error addresses source sampling uncertainty, not differences in population, confounding, outcome measurement, follow-up time, or the adequacy of the source contrast as a proxy for the target missing-versus-reference discrepancy.