Bootstrap Intervals for Differences Between Groups

Why Bootstrap the Groups Separately?

When two independent groups are measured on the same outcome, a natural question is how much their typical values differ. For example, a clinician might compare recovery times for patients receiving two different treatments. The observed difference between the groups is a useful estimate, but it varies from sample to sample. A bootstrap interval uses the observed data to estimate that uncertainty.

The key design feature is independence between groups: an observation in one group is not naturally matched to a particular observation in the other. In that situation, resample observations within each group separately. This preserves each group’s observed distribution and sample size while allowing the composition of each bootstrap sample to vary.

Definition: Let group 1 have \(n_1\) observations and group 2 have \(n_2\) observations. For a mean difference, define the parameter as \(\Delta_\mu=\mu_1-\mu_2\) and estimate it by \(\bar{x}_1-\bar{x}_2\). For a median difference, define \(\Delta_M=M_1-M_2\) and estimate it by \(\tilde{x}_1-\tilde{x}_2\), where \(M_j\) and \(\tilde{x}_j\) are the population and sample medians for group \(j\). The subtraction order must be stated because reversing it changes the sign.

How a Two-Group Bootstrap Works

A bootstrap replicate is built by drawing \(n_1\) observations with replacement from the observed values in group 1, then drawing \(n_2\) observations with replacement from the observed values in group 2. “With replacement” means that a recorded value can appear more than once or not at all in a replicate. Calculate the statistic in each resampled group and subtract group 2’s statistic from group 1’s.

Repeat this process many times, commonly several thousand or more. The resulting bootstrap differences form an estimated sampling distribution for the difference. For example, for a mean difference, replicate \(b\) gives

\Delta^{*(b)}_\mu=\bar{x}^{*(b)}_1-\bar{x}^{*(b)}_2,\qquad b=1,\ldots,B.

Here, \(B\) is the number of bootstrap replicates and the asterisk marks a statistic calculated from resampled data. For a median difference, replace each resampled mean by its resampled median. Do not combine the two groups into one pool before resampling: doing so would no longer preserve the distinct observed distributions that the two-group comparison is intended to describe.

Set the comparison.
Identify the outcome, the two independent groups, the statistic (mean or median), and the subtraction order.

Resample within each group.
Draw \(n_1\) values with replacement from group 1 and \(n_2\) values with replacement from group 2.

Calculate a replicate difference.
Find the chosen statistic in each resampled group and subtract group 2’s value from group 1’s.

Repeat and form an interval.
Use the distribution of the \(B\) replicate differences to estimate uncertainty, for example with a percentile interval.

The Percentile Interval and Its Justification

For a two-sided percentile interval with confidence level \(1-\alpha\), sort the \(B\) bootstrap differences. The interval endpoints are the estimated \(\alpha/2\) and \(1-\alpha/2\) quantiles of that distribution. A 95% interval therefore uses its 2.5th and 97.5th percentiles. In computation, quantiles are estimated from the finite set of replicates, so software may use slightly different interpolation rules.

\left[ Q^*_{\alpha/2},\ Q^*_{1-\alpha/2} \right].

The logic is that the bootstrap distribution approximates the sampling distribution of the statistic by treating the observed data as an estimate of the population distribution. Its central 95% of replicate differences then supplies approximate lower and upper limits for the unknown population difference. This approximation is not guaranteed to be accurate for every data set; the sampling design and the shape and size of the samples matter.

A short justification for resampling separately follows from the variance of a difference. If the two group samples are independent, then for sample means

\operatorname{Var}(\bar{X}_1-\bar{X}_2) = \operatorname{Var}(\bar{X}_1)+\operatorname{Var}(\bar{X}_2).

In a bootstrap replicate, the resampled mean from group 1 varies according to group 1’s empirical distribution, and the resampled mean from group 2 varies according to group 2’s empirical distribution. Independent resampling preserves that separate variation. Subtracting the two resampled means therefore reflects uncertainty from both groups. Pooling the observations would instead treat them as draws from one shared empirical distribution, which is not the intended model when the groups may differ.

The same resampling construction applies to medians, although the variance identity above is specifically a simple justification for means. For medians, the bootstrap distribution directly approximates the sampling variation of the median difference; its shape can be less regular, especially with small samples or many tied values.

Worked Example: A Difference Between Group Means

Suppose a hypothetical study records an outcome in points for two independent groups. Group 1 has values 10, 12, 13, 15, 16, and 18; group 2 has values 6, 7, 9, 10, 11, and 13. Estimate group 1 minus group 2, then describe how a percentile bootstrap interval is obtained.

Check the design. The observations are assumed to be independent within each group, and the two groups consist of different individuals rather than matched pairs. The group sample sizes are both 6. The outcome is measured in points, so the difference will also be in points. These conditions support resampling each group separately; whether the samples represent the populations of interest depends on how the hypothetical study obtained them.

The group means are

\bar{x}_1=\frac{10+12+13+15+16+18}{6}=\frac{84}{6}=14, \qquad \bar{x}_2=\frac{6+7+9+10+11+13}{6}=\frac{56}{6}=9.333.

Thus the observed difference is \(14-9.333=4.667\) points. For one illustrative bootstrap replicate, suppose the resampled group 1 values are 10, 10, 13, 16, 18, 18 and the resampled group 2 values are 6, 7, 7, 10, 11, 13. Each replicate contains six values from each group, sampled from that group’s own observed values.

The replicate means and their difference are

\bar{x}^{*}_1=\frac{85}{6}=14.167,\qquad \bar{x}^{*}_2=\frac{54}{6}=9,\qquad \Delta^*_\mu=14.167-9=5.167\text{ points}.

That single replicate is not an interval. Repeat the separate resampling many times and sort the resulting differences. For illustration, suppose a run with \(B=20{,}000\) replicates gives estimated 2.5th and 97.5th percentiles of 0.5 and 8.8 points. The approximate 95% percentile interval is therefore [0.5, 8.8] points. The observed estimate is 4.667 points, and the interval indicates substantial uncertainty about the population mean difference.

The interval’s lower endpoint is above zero, but this bootstrap interval alone does not prove that the groups differ or establish why they might differ. The interpretation also depends on the study design and on whether the bootstrap approximation is adequate.

Using the Median Instead

A difference in means targets average outcomes; a difference in medians targets the difference between the two population medians. These are different questions. A median can be useful when the outcome is skewed or when unusually large observations make the mean unrepresentative of a typical value. To bootstrap a median difference, calculate the sample median in each resampled group and subtract in the same stated order.

Worked Example: A Difference Between Group Medians

Consider hypothetical, independent measurements in hours. Group 1 records 2, 3, 3, 4, 5, 8, and 20 hours; group 2 records 1, 2, 2, 3, 4, 6, and 15 hours. Estimate the median difference, group 1 minus group 2.

Each group has seven observations, so its sample median is the fourth ordered value. The group 1 median is 4 hours and the group 2 median is 3 hours. The estimated difference is \(4-3=1\) hour. The high values 20 and 15 hours affect the means more directly than they affect these sample medians; that is consistent with targeting a median difference, not a mean difference.

For an illustrative bootstrap replicate, suppose group 1’s resampled values are 2, 3, 3, 5, 8, 20, 20 and group 2’s are 1, 2, 2, 4, 6, 15, 15. Both resamples preserve their group’s sample size of seven. Their fourth ordered values are 5 and 4 hours, so the replicate median difference is \(5-4=1\) hour.

The replicate value of 1 hour is only one result. After many separate resamples, suppose the 2.5th and 97.5th percentiles of the median differences are -2 and 5 hours. The illustrative 95% percentile interval is [-2, 5] hours. Because this interval includes zero, the data and this bootstrap procedure do not pin down the direction of the population median difference at the 95% interval level. The interval does not say that there is a 95% probability that the fixed population difference lies within these particular limits.

The reported endpoints here are illustrative output from a hypothetical bootstrap run, not values computed from the single replicate shown. In an actual analysis, report the number of replicates and enough information about the data and method for the result to be reproduced.

Unequal Sample Sizes and Careful Interpretation

The two groups need not have the same sample size. If group 1 has \(n_1=5\) observations and group 2 has \(n_2=8\), every replicate draws five values from group 1 and eight from group 2. The groups’ sizes are preserved separately, not made equal. This represents the sampling variation associated with the observed design.

Worked Example: Preserving Different Group Sizes

Suppose hypothetical independent measurements in minutes are 4, 5, 7, 8, and 11 in group 1, and 3, 4, 4, 6, 7, 8, 10, and 12 in group 2. Compare the group means, taking group 1 minus group 2.

The observed means are

\bar{x}_1=\frac{35}{5}=7,\qquad \bar{x}_2=\frac{54}{8}=6.75,\qquad \bar{x}_1-\bar{x}_2=0.25\text{ minutes}.

For one replicate, suppose group 1 yields 4, 4, 7, 8, and 11, while group 2 yields 3, 4, 6, 8, 12, 12, 12, and 12. The first resample still has five values and the second still has eight. Their means are \(34/5=6.8\) and \(69/8=8.625\), respectively, so this replicate’s difference is \(6.8-8.625=-1.825\) minutes.

A single negative replicate does not overturn the observed positive estimate. It is one draw from the bootstrap distribution. In a hypothetical run of many replicates, the central 95% of differences might, for illustration, extend from -3.4 to 3.9 minutes. That interval would express uncertainty around a small observed difference and include values in both directions. The example’s measurements are assumed to be independent within and between groups; representativeness still depends on the sampling process.

Conditions, Limitations, and Common Mistakes

A bootstrap interval is only as useful as the data and resampling scheme supporting it. Before interpreting one, check the following:

Independent groups: There should be no natural pairing between a group 1 observation and a group 2 observation. If observations are matched, preserve the pairing and bootstrap the paired differences instead.
Appropriate observations: Within each group, the observations should reasonably represent independent draws from the population relevant to the question. Strong dependence, such as repeated measurements on the same person treated as unrelated observations, calls for a different resampling plan.
Sample sizes and data support: The empirical distribution can only resample values that were observed. With very small samples, unusual shapes, or many ties, the bootstrap distribution may poorly approximate the true sampling distribution, particularly for medians.
Matching statistic and question: A mean-difference interval concerns population means; a median-difference interval concerns population medians. A confidence interval does not automatically describe every notion of a “typical” difference.
Clear direction and units: State which group is subtracted from which, and retain the outcome’s units in the estimate and interval.

A common mistake is to pool both groups and resample from that combined collection. This treats the groups as though they shared one empirical distribution and can erase the very difference being estimated. Another mistake is to resample group 1 and group 2 with an equal number of observations just because that seems convenient. If the original sample sizes differ, preserve each one. For paired data, separate resampling is also inappropriate because it breaks the observed pairing.

Finally, a confidence interval is not a probability statement about a fixed parameter after the interval has been calculated. The 95% label describes the long-run coverage of the interval procedure under its assumptions: across repeated samples and valid applications of the method, about 95% of intervals would cover the target difference.

Key takeaway: For two independent groups, resample with replacement within each group, keeping its original sample size. Calculate the chosen group 1 minus group 2 statistic in every replicate, and use the resulting distribution to estimate uncertainty in that same difference.

Check Your Understanding

Try these questions without looking back.

Two independent groups have different sample sizes. How many observations should each group contribute to every bootstrap replicate?
A study targets the difference between population medians. What statistic should be calculated in each resampled group?
Why is pooling the two groups before bootstrap resampling generally unsuitable for estimating their difference?
An interval for group 1 minus group 2 is entirely below zero. What does the sign say about the estimated direction of the difference, and what does it not establish by itself?
Why should paired measurements not be resampled separately as though they were independent groups?