Plot-Based Diagnostics for Temporal Imbalance

Why Examine Covariates by Period?

In a platform trial, an arm may open after a period of historical control enrollment. Even if the eligibility criteria appear comparable, the mix of people entering the trial can change over calendar time. For example, the prevalence of a prognostic risk factor may rise across historical periods and be higher still among people eligible for a current arm. A single comparison of pooled historical controls with the current eligible population can reveal a difference, but it may hide whether the difference developed gradually, appeared abruptly, or was driven by one period.

Plot-based diagnostics make those temporal patterns visible. Standardized differences summarize the size of a covariate contrast on a scale that is less dependent on sample size than a test statistic. They are descriptive diagnostics: they do not show that groups are exchangeable, establish a causal effect, or correct an imbalance. The earlier material on assessing covariate overlap across enrollment periods concerns whether comparable covariate values are represented; here the emphasis is on visualizing and quantifying changes in composition.

Definition: A temporal imbalance diagnostic compares baseline covariate distributions across specified enrollment periods, including historical-control periods and the population eligible for a current arm. A standardized difference expresses a difference in a covariate summary in units of a reference standard deviation. For descriptive comparison, its magnitude is generally more informative than its sign alone.

Set Up the Comparison Before Plotting

Specify the groups and variables before looking at the plots. Historical controls should be divided into meaningful calendar-time windows, such as the periods defined for the platform comparison. The current group should be the population eligible for the current arm, measured at baseline and before treatment assignment or initiation. Apply the same eligibility definitions, covariate definitions, measurement units, and coding rules wherever possible. If definitions or measurement procedures changed, mark those changes: an apparent shift may reflect measurement rather than a change in patient composition.

Choose covariates for clinical and design reasons, particularly characteristics that could predict the outcome or differ across time. Plot the distribution of each relevant covariate by period. Depending on its type, a plot might show period-specific means and spreads, boxplots, proportions, or a stacked bar chart for categories. Include sample sizes and use a consistent scale. A plot of period summaries can be useful, but should not conceal broad or multimodal distributions.

For a continuous covariate \(X\), compare group means using the standardized difference

d_X=\frac{\bar{x}_{C}-\bar{x}_{H}}{\sqrt{(s_C^2+s_H^2)/2}},

where \(C\) denotes the current eligible population and \(H\) the historical controls being summarized. The denominator is the square root of the average of the two sample variances. For a binary covariate with proportions \(\hat p_C\) and \(\hat p_H\), use the corresponding Bernoulli variances:

d_X=\frac{\hat p_C-\hat p_H} {\sqrt{\{\hat p_C(1-\hat p_C)+\hat p_H(1-\hat p_H)\}/2}}.

For a categorical covariate with more than two levels, calculate a binary standardized difference for each level, treating membership in that level as an indicator. Read the set of contrasts together: a change in one category necessarily affects at least one other category. Report the absolute magnitude when describing size, while retaining the signed value when direction matters.

Worked Examples

Worked Example: A Categorical Covariate Across Periods

Suppose a hypothetical trial records baseline site for historical controls in two periods, each with 100 participants, and for 100 people eligible for a current arm. Counts are: Site A, 60 in historical period 1, 50 in period 2, and 30 current; Site B, 25, 30, and 40; other sites, 15, 20, and 30. All counts sum to 100 within each period.

The historical pooled proportions must be calculated from the actual period sizes and counts. Across the 200 historical participants, the proportions are \(110/200=0.550\) at Site A, \(55/200=0.275\) at Site B, and \(35/200=0.175\) at other sites. They are consistent with the period data: each is the equal-size average of its two period proportions. The current proportions are 0.300, 0.400, and 0.300.

For Site A, the binary standardized difference is

d_A=\frac{0.300-0.550} {\sqrt{\{0.300(0.700)+0.550(0.450)\}/2}} =\frac{-0.250}{\sqrt{0.22875}} =\frac{-0.250}{0.4783} =-0.523.

Thus, the current eligible group has a lower Site A proportion, by about 0.523 reference standard deviations. The plot adds important context: Site A falls from 0.600 to 0.500 across the historical periods and then to 0.300 in the current group. The pooled difference alone would not show this sequence. Site B rises from 0.250 to 0.300 to 0.400, while other sites rise from 0.150 to 0.200 to 0.300. These patterns suggest that the shift is not confined to a single category.

Worked Example: A Continuous Covariate

In a hypothetical trial, age is measured in years. Each of two historical periods has 100 controls. The first has mean age 54 and sample standard deviation 10; the second has mean 60 and standard deviation 12. The current eligible group has 100 people, mean age 64, and sample standard deviation 11. These summaries describe the same age definition in all three groups.

The pooled historical mean is \((100(54)+100(60))/200=57\) years. To compute the pooled historical sample variance, combine within-period and between-period sums of squares. Each within-period sum of squares is \((n_j-1)s_j^2\); the between-period contribution is \(n_j(\bar{x}_j-\bar{x}_H)^2\). Therefore,

s_H^2= \frac{99(10^2)+99(12^2)+100(54-57)^2+100(60-57)^2}{199} =\frac{25{,}956}{199} =130.43\ {\rm years}^2.

The historical pooled standard deviation is \(\sqrt{130.43}=11.42\) years. The standardized difference comparing current with pooled historical age is

d_{\text{age}}= \frac{64-57}{\sqrt{(11^2+130.43)/2}} =\frac{7}{\sqrt{125.715}} =\frac{7}{11.21} =0.624.

The current eligible group is older by 7 years on average, a difference of about 0.624 reference standard deviations. A period plot would also show that the historical mean already rose from 54 to 60 years. The pooled contrast is therefore not a substitute for inspecting the temporal pattern. A mean-and-spread plot, or period-specific boxplots, can show whether the shift concerns the whole distribution or only a tail.

Worked Example: A Binary Risk Factor and Unequal Period Sizes

In a hypothetical trial, a baseline risk factor is present in 16 of 80 historical controls in period 1 and 48 of 120 in period 2. It is present in 55 of 100 people eligible for the current arm. The historical period sizes differ, so pooling the two period percentages by taking their simple average would be incorrect.

The historical total is \((16+48)/(80+120)=64/200=0.320\). The period-specific proportions are \(16/80=0.200\) and \(48/120=0.400\), and the correctly weighted pooled proportion is \(0.4(0.200)+0.6(0.400)=0.320\). The current proportion is \(55/100=0.550\). Thus,

d_X=\frac{0.550-0.320} {\sqrt{\{0.550(0.450)+0.320(0.680)\}/2}} =\frac{0.230}{\sqrt{0.23255}} =\frac{0.230}{0.4822} =0.477.

The risk factor is more common in the current eligible group, with an absolute standardized difference of 0.477. The period plot shows an increase from 0.200 to 0.400 to 0.550. That trajectory is a diagnostic signal to investigate, not proof that calendar time caused the shift or that the factor explains any outcome difference.

Why Standardize, and How to Read the Plots

A raw difference is easy to interpret in the covariate's original units: the age difference above is 7 years, and the binary risk-factor difference is 0.230, or 23 percentage points. Standardization allows rough comparison of differences measured in different units. It also avoids the sample-size dependence of a significance-test statistic: increasing the sample size can make a small difference statistically detectable without making the difference larger in practical terms.

The continuous-covariate formula uses an average of the two variances as a scale. This is a descriptive convention, not a claim that the groups have equal variances. The binary formula uses the average of the two Bernoulli variances. Either denominator can be very small when a variable is nearly constant, making a standardized difference unstable or undefined if both groups have zero variance. In such cases, show the raw counts and proportions and explain the lack of variation rather than treating the standardized number as reliable.

There is no universal threshold at which a standardized difference proves acceptable balance. Values near zero indicate similar summaries on the chosen scale; larger magnitudes warrant attention, but judgment depends on the covariate, its prognostic importance, the design, and the full distributions. A small difference in means can also conceal a difference in spread, skewness, or tails. For that reason, use plots and standardized summaries together.

A useful diagnostic sequence is:

Define groups and periods.
Specify the historical windows and the current eligible population; check that covariate timing, eligibility, and coding are comparable.

Plot covariates by period.
Show distributions or proportions, sample sizes, and period order. Mark protocol, measurement, or standard-care changes that could affect interpretation.

Calculate contrasts.
Compute period-specific and pooled standardized differences for relevant covariates, and show raw summaries alongside them.

Investigate patterns, not a pass/fail label.
Identify gradual trends, abrupt changes, sparse categories, and periods with different distributions. Decide whether the comparison remains credible in light of the design and available covariates.

A Short Justification for Sample-Size Weighting

The pooled historical proportion is a count divided by a total count. If period \(j\) has size \(n_j\) and proportion \(\hat p_j\), its count is \(n_j\hat p_j\), so the overall proportion is

\hat p_H=\frac{\sum_j n_j\hat p_j}{\sum_j n_j} =\sum_j\left(\frac{n_j}{\sum_k n_k}\right)\hat p_j.

The weights are period sample-size fractions and sum to one. Hence a pooled value must lie between the smallest and largest period proportions. If a reported pooled value falls outside that range, or conflicts with the underlying integer counts, the summaries cannot all be correct. This basic check prevents internally inconsistent diagnostics.

Common Pitfalls and Limits

Using unweighted averages of period summaries. This gives periods equal influence even when their sample sizes differ. Pool counts or use the sample-size weights, as in the binary-risk example.
Looking only at the pooled contrast. Historical periods may move in different directions or one period may drive the pooled result. Plot period-specific values as well as the pooled comparison.
Treating a small standardized difference as proof of comparability. It describes selected covariate summaries, not unmeasured factors, outcome trends, or exchangeability.
Using a p-value as a balance score. A p-value depends strongly on sample size and does not quantify the practical size of a covariate difference. Prefer plots, raw summaries, and standardized differences for this diagnostic purpose.
Ignoring sparse or changing categories. Small counts can make proportions unstable; category definitions or recording practices may also change. Display denominators and investigate data provenance.
Assuming similar means imply similar distributions. Differences in variance, skewness, or tails may matter even when means are close. Inspect distributional plots and consider whether the observed values overlap meaningfully.

These diagnostics cannot adjust for imbalance or establish that historical controls are exchangeable with current eligible participants. Nor do they distinguish a genuine change in participant mix from changes in measurement or eligibility unless those features are examined. They help identify what needs scrutiny; causal interpretation requires the design assumptions and analysis appropriate to the comparison.

Key takeaway: Plot each important baseline covariate across historical periods and the current eligible group, then use correctly weighted summaries and standardized differences to describe the size and direction of contrasts. Treat the results as diagnostic evidence, not as proof of balance or a causal conclusion.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two historical periods have sizes 50 and 150, with a binary covariate present in 10 and 60 participants, respectively. What is the pooled historical proportion?
Why can a period plot reveal a pattern that a pooled historical-versus-current standardized difference misses?
For a continuous covariate, what does a standardized difference of \(-0.40\) say about the direction of the current-versus-historical mean contrast?
Why is a large-sample p-value not a direct measure of the practical size of a covariate imbalance?

Solutions. (1) The total is \((10+60)/(50+150)=70/200=0.350\). Equivalently, \(0.25(10/50)+0.75(60/150)=0.25(0.200)+0.75(0.400)=0.350\). (2) Pooling compresses multiple historical periods into one summary, hiding whether the covariate changed steadily, abruptly, or in only one period. (3) With the stated current-minus-historical convention, the current mean is lower than the historical mean by 0.40 reference standard deviations. (4) A p-value depends on both the size of the difference and its precision, which is strongly affected by sample size; it is not an effect-size scale. A standardized difference and the raw summaries describe magnitude more directly.