A Resampling Analysis from Data to Conclusion

From Recorded Outcomes to an Evidence-Based Conclusion

A resampling analysis is most useful when it connects calculations to the question that motivated the study. Here, imagine a small randomized study comparing a sleep app with no app. Six participants are randomly assigned, three to each group, and their average nightly sleep during a follow-up week is recorded. The app group has values of 7, 8, and 9 hours; the control group has values of 6, 7, and 8 hours. These are hypothetical data, not results from a real study.

We will estimate the difference in mean sleep hours, use a bootstrap distribution to describe uncertainty in that estimate, and use the known treatment randomization to test a sharp null hypothesis. The procedures share the idea of resampling, but they do not answer the same question: the bootstrap resamples observed outcomes, while the permutation test rearranges treatment assignments.

Definition: Define the observed difference as \(D=\bar{x}_{A}-\bar{x}_{C}\), where \(\bar{x}_{A}\) and \(\bar{x}_{C}\) are the sample mean sleep hours in the app and control groups. A positive value means more observed sleep in the app group. A bootstrap distribution is the distribution of a statistic across samples made by resampling the observed data according to a specified scheme. A treatment-assignment permutation distribution is the distribution of a statistic across treatment assignments allowed by the study design, with outcomes held fixed.

Start With the Observed Difference

For these data, the app-group mean is \((7+8+9)/3=8\) hours, and the control-group mean is \((6+7+8)/3=7\) hours. Thus \(D=8-7=1\) hour. This is the estimated difference in the groups’ mean follow-up sleep. Because treatment was randomly assigned, the contrast can also be interpreted as an estimate of the average treatment effect for these participants, subject to the limitations of a very small study.

Worked Example: Calculating the Observed Effect

Use the hypothetical sleep-hour data to compute the difference in sample means, keeping the direction app minus control.

App group: \(\bar{x}_{A}=(7+8+9)/3=24/3=8\) hours.

Control group: \(\bar{x}_{C}=(6+7+8)/3=21/3=7\) hours.

Difference: \(D=\bar{x}_{A}-\bar{x}_{C}=8-7=1\) hour. The observed app-group mean is one hour higher. This calculation describes the sample; by itself, it does not show whether the difference is distinguishable from random variation.

Bootstrap Interval for the Difference

For an independent-groups bootstrap, resample three observations with replacement from the app group and, separately, three with replacement from the control group. In each bootstrap replicate, calculate the app resample mean minus the control resample mean. Resampling separately preserves the two-group structure and the original group sizes.

This procedure treats observations within each group as independent draws from that group’s outcome distribution. To use the interval for a broader population, the participants must also provide a reasonable basis for generalizing to that population. Random assignment supports a causal comparison within the experiment, but does not by itself make six participants representative of all people who might use a sleep app. With only three observations per group, the bootstrap distribution is also necessarily coarse and the resulting interval should be treated cautiously.

Worked Example: Building and Reading the Bootstrap Distribution

For the app group \(\{7,8,9\}\), the possible resample sums from three draws with replacement are 21 through 27, with respective counts \(1,3,6,7,6,3,1\) among the \(3^3=27\) ordered resamples. For the control group \(\{6,7,8\}\), sums 18 through 24 have the same respective counts. Combining every app resample with every control resample gives \(27\times27=729\) bootstrap differences.

Write \(k\) for the app resample sum minus the control resample sum. Each bootstrap difference is \(k/3\) hours. For example, an app resample with sum 23 and a control resample with sum 22 gives \(k=1\), hence a bootstrap difference of \(1/3\) hour. In the full enumeration, the counts at \(k=-3,-2,-1\) are 1, 6, and 21. Therefore the cumulative count through \(k=-2\) is \(1+6=7\), while the cumulative count through \(k=-1\) is \(1+6+21=28\). The 19th ordered bootstrap difference is consequently \(-1/3\) hour.

The distribution is symmetric around \(k=3\): reflecting app sums around 24 and control sums around 21 maps a difference \(k\) to \(6-k\). Thus the ordered values at ranks 711 and 712 are both \(7/3\) hours. For this example, use the percentile convention that selects rank \(\lceil(B+1)p\rceil\) for percentile \(p\), where \(B=729\). The lower rank is \(\lceil730(0.025)\rceil=19\); the upper rank is \(\lceil730(0.975)\rceil=712\). The resulting 95% percentile bootstrap interval is \([-1/3,\ 7/3]\), or approximately \([-0.333,\ 2.33]\) hours.

The interval includes zero and ranges from a small negative difference to a positive difference of more than two hours. It is an approximate interval for the difference in mean sleep hours under the bootstrap assumptions; it is not a statement that there is a 95% probability that the fixed population difference lies in this particular interval. A repeated-sampling interpretation is that the stated interval procedure would cover the target parameter about 95% of the time under its assumptions.

Why the Bootstrap Resamples Within Groups

The group-specific resampling scheme follows from the statistic being estimated. Each group mean is estimated from that group’s observed outcomes, so resampling within each group approximates the variation in each mean while retaining the observed group sizes. Subtracting the two resampled means then approximates the variation in their difference.

D^*=\bar{x}_{A}^*-\bar{x}_{C}^*

Here, \(D^*\) denotes a bootstrap replicate. Its distribution is centered near the observed difference, but it is not a permutation distribution under a null hypothesis. In this example, enumerating all possible resamples removes Monte Carlo simulation error from the bootstrap calculation; it does not remove the uncertainty caused by having observed only three participants per group.

Exact Treatment-Assignment Permutation Test

Now consider the sharp null hypothesis that the app has no effect on any participant’s sleep outcome. Under this null, each participant’s recorded outcome would be the same whether assigned to the app or control. Because the design randomly assigned exactly three of the six participants to the app group, there are \(\binom{6}{3}=20\) allowed assignments, all equally likely under the design. For each assignment, keep the six outcomes fixed, assign three to the app label, and calculate the difference in assigned group means.

A two-sided test counts assignments whose absolute difference is at least as large as the observed absolute difference. This inclusive comparison treats ties as at least as extreme. The randomization scheme is valid here because the treatment assignment mechanism is known and the null is sharp: it specifies every participant’s outcome under either treatment. The test is not simply a test of equal population means; a null of equal means alone does not specify each participant’s unobserved outcome under the other assignment.

Worked Example: Counting Extreme Treatment Assignments

The pooled outcomes are \(6,7,7,8,8,9\), with total 45 hours. If a particular assignment gives the app group a total \(S\), its difference in means is

\frac{S}{3}-\frac{45-S}{3}=\frac{2S-45}{3}.

The observed app total is \(7+8+9=24\), so the observed difference is \((2(24)-45)/3=1\) hour. Assignments at least as extreme in either direction satisfy

\left|\frac{2S-45}{3}\right|\geq 1, \qquad\text{so}\qquad S\leq21\ \text{or}\ S\geq24.

Among the 20 possible groups of three, five have total \(S\leq21\) and five have total \(S\geq24\). For instance, a low-total group must include the value 6; its other two values can total 14 (one combination) or 15 (four combinations). By the corresponding enumeration at the high end, there are five high-total groups. The exact two-sided p-value is therefore \((5+5)/20=0.50\).

If the sharp null is true, 50% of the allowed assignments produce a difference at least as large in absolute value as the one observed. This p-value is not the probability that the null is true. The result does not provide strong evidence against the sharp null at conventional significance levels, but it also does not establish that the app has no effect.

Connecting the Two Results

The bootstrap interval and permutation p-value tell complementary stories. The interval estimates uncertainty in a difference in means by resampling observed outcomes within groups. The exact test evaluates how unusual the observed difference is under the sharp no-effect null and the actual randomization scheme. Their assumptions and targets differ, so one should not treat them as two versions of the same calculation.

In this hypothetical study, the observed difference is 1 hour, the 95% bootstrap interval is approximately \(-0.333\) to \(2.33\) hours, and the exact two-sided randomization p-value is 0.50. Both results caution against a confident claim of a positive app effect from these data. The interval is wide, and the randomization test does not find the observed contrast unusual under the sharp null. With so few participants, neither result is precise enough to rule out effects that could matter in practice.

Define the contrast.
State the groups, outcome, units, and direction. Here the statistic is app mean minus control mean in hours.

Match resampling to the question.
For a bootstrap interval, resample outcomes within each group. For a randomization test, hold outcomes fixed and use assignments allowed by the study design.

Report the result and its scope.
Give the estimate, interval or p-value, and a conclusion in context. State the assumptions and avoid claiming more than the design and sample support.

Common Pitfall

A common mistake is to shuffle the six outcomes among the treatment labels when constructing a bootstrap interval, then describe that distribution as the uncertainty in the observed difference. That shuffle constructs a permutation distribution, not the ordinary group-wise bootstrap distribution. Conversely, resampling separately within each observed group does not test the sharp null under the treatment-assignment design. Choose the resampling scheme to match the inferential question, and describe the assumptions behind it.

Key takeaway: In a randomized study, use the bootstrap to approximate uncertainty in an estimated contrast by resampling observed outcomes within groups, and use treatment-assignment permutations to assess the observed statistic under a clearly stated sharp null. Report both in context, with appropriate caution when the sample is very small.

Check Your Understanding

Use the definitions and hypothetical sleep-app analysis above to answer these questions.

Why is the observed difference defined as app mean minus control mean, and what does a positive value mean in this analysis?
In a group-wise bootstrap replicate, which observations are resampled together, and why?
Why does the treatment-assignment test use 20 assignments rather than all possible ways to rearrange six distinct outcomes without regard to the design?
What does the exact p-value of 0.50 mean under the sharp null, and what does it not mean?
Give one reason the bootstrap interval should be interpreted cautiously in this study.