Tutorials › Introduction to Statistics › Select Procedures for Before-and-After Studies

Home  /  Tutorials  /  Choosing the Right Procedure

Choosing the Right Procedure · Tutorial 795 of 996

Select Procedures for Before-and-After Studies

Use the relationship between observations—not merely the labels “before” and “after”—to select and justify the appropriate t procedure.

Intermediate 10 min read

What You'll Learn

  • Identify when before-and-after measurements form pairs.
  • Define the paired change and its direction.
  • Select paired t inference for the mean change across pairs.
  • Select independent two-sample t inference for separate groups.
  • Check design and distribution conditions for each procedure.
  • Distinguish a paired analysis from comparing two independent means.

Why “Before and After” Is Not Enough

A study described as “before and after” might involve measuring the same people twice, or measuring one group of people before a change and a different group afterward. These designs do not provide the same kind of data. In the first, each participant’s two measurements are linked. In the second, the measurements come from separate individuals. That distinction determines whether a paired t procedure or an independent two-sample t procedure is appropriate.

The key question is not simply whether one measurement happened earlier in time. Ask instead: Does each measurement in one group have a meaningful, pre-established partner in the other group? If so, analyze the within-pair differences. If not, compare the two groups as independent samples. The choice matters because the paired procedure uses the information in the links between measurements, while an independent procedure does not.

Definition: A paired design has two related measurements for each unit, or for each deliberately matched pair of units. A before-and-after study on the same participants is paired because each participant’s before measurement is linked to that same participant’s after measurement. In an independent-groups design, the measurements in one group are not linked to measurements in the other group; the groups consist of separate units.

Paired t Inference: Analyze the Changes

For a paired study, choose and state a direction for the difference before calculating it. For example, define \(d_i=\text{after}_i-\text{before}_i\) for participant \(i\). The data to analyze are then the \(n\) differences \(d_1,\ldots,d_n\), one per complete pair. Their sample mean is \(\bar{d}\), and their sample standard deviation is \(s_d\). The target parameter is the population mean paired difference, \(\mu_d\), for the population represented by the study.

A paired t procedure is a one-sample t procedure applied to the differences. For testing \(H_0:\mu_d=\mu_{d,0}\), the test statistic is \(t=(\bar{d}-\mu_{d,0})/(s_d/\sqrt{n})\), with \(df=n-1\). A confidence interval for \(\mu_d\) has the form \(\bar{d}\pm t^*(s_d/\sqrt{n})\), where \(t^*\) is the appropriate t critical value. Here, \(n\) counts pairs, not the total number of recorded measurements.

The procedure requires independent pairs: one participant’s pair of measurements should not determine another participant’s pair. The distribution of the differences in the population should be approximately normal when the number of pairs is small, and the observed differences should not contain severe outliers. With a sufficiently large number of pairs, the t procedure is generally more robust to moderate departures from normality, but strong skewness or extreme outliers can still be problematic. The within-pair observations need not be independent; their relationship is precisely why they are paired.

Worked Example: Measuring Change in the Same Participants

Suppose, hypothetically, that eight participants have a measurement recorded before and after a program. Define the change as after minus before, so positive values represent increases. Their paired changes, in units, are \(2,4,1,3,5,0,4,1\). We want a 95% confidence interval for the population mean change.

Check the design and conditions. Each before measurement belongs to the same participant as that participant’s after measurement, so there are eight pairs. Assume the participants were sampled independently of one another. The differences range from 0 to 5 and show no severe outlier, though with only eight pairs the population distribution of differences should be reasonably close to normal for t inference to be dependable.

The sample mean change is \(\bar{d}=20/8=2.5\) units. The sum of squared deviations from 2.5 is 22, so the sample standard deviation and standard error are

$$ s_d=\sqrt{\frac{22}{8-1}}=1.773,\qquad SE=\frac{s_d}{\sqrt{8}}=0.6268. $$

The degrees of freedom are \(8-1=7\). For a 95% interval, the t critical value is approximately 2.365. Thus

$$ \bar{d}\pm t^*SE =2.5\pm 2.365(0.6268) =2.5\pm 1.482 =(1.018,\ 3.982)\text{ units}. $$

The interval estimates that the population mean increase is between about 1.018 and 3.982 units. In repeated sampling under the procedure’s conditions, 95% of intervals constructed this way would cover the population mean change. The interval is for the mean of the paired changes, not for a difference between two independent group means.

Independent Two-Sample t Inference: Compare Separate Groups

If the measurements come from separate, unpaired groups, retain the two samples as two samples. Let \(\bar{x}_1\) and \(s_1\) summarize group 1 and \(\bar{x}_2\) and \(s_2\) summarize group 2. The target is the difference between the population means, \(\mu_1-\mu_2\), with the order chosen to match the question. The Welch two-sample t procedure estimates its standard error as

$$ SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}. $$

Welch inference uses an approximate degrees-of-freedom calculation based on both sample sizes and standard deviations. It does not require equal population variances. A pooled two-sample t procedure is an alternative only when equal population variances are justified by the context and design; it combines the samples to estimate one common variance. When there is no sound reason to assume equal variances, Welch’s procedure is the appropriate choice.

For either independent two-sample procedure, the groups must be independent of one another, and observations within each group should be independent. Random sampling or random assignment can support these design conditions, but the specific study design must be considered. For small samples, each population distribution should be approximately normal, with no severe outliers. Larger samples can tolerate moderate non-normality more readily. Choosing Welch rather than pooled does not remove the need to check independence or the shape of the data.

Worked Example: Comparing Two Separate Groups

In a hypothetical study, 12 people in group 1 and 10 different people in group 2 complete a task. Their completion times, in minutes, have summaries \(\bar{x}_1=18.4\), \(s_1=4.2\), \(\bar{x}_2=15.1\), and \(s_2=3.0\). Estimate the difference in population mean time, group 1 minus group 2, with a 95% confidence interval.

Check the design and conditions. The participants are different people in the two groups, with no meaningful matching, so an independent two-sample procedure is suitable. Assume participants were independently sampled or assigned within the study and that neither group contains severe outliers. Since both sample sizes are modest, approximately normal population distributions are also needed. Use Welch inference; no equal-variance assumption is required.

The observed difference is \(18.4-15.1=3.3\) minutes. Its estimated standard error is

$$ SE=\sqrt{\frac{4.2^2}{12}+\frac{3.0^2}{10}} =\sqrt{1.47+0.90} =\sqrt{2.37} =1.539. $$

Welch’s approximate degrees of freedom are

$$ df= \frac{(1.47+0.90)^2} {\frac{1.47^2}{12-1}+\frac{0.90^2}{10-1}} \approx 19.61. $$

For a 95% interval with about 19.61 degrees of freedom, \(t^*\) is approximately 2.09. Therefore,

$$ (\bar{x}_1-\bar{x}_2)\pm t^*SE =3.3\pm 2.09(1.539) \approx 3.3\pm 3.22 =(0.08,\ 6.52)\text{ minutes}. $$

The interval estimates that group 1’s population mean completion time is between about 0.08 and 6.52 minutes greater than group 2’s. This describes a difference in population means; by itself, it does not establish that group membership caused a difference.

A Before-and-After Label Can Describe Independent Samples

Suppose a clinic records waiting times from one sample of patients before changing its appointment system, then records waiting times from a new sample afterward. If patients are not observed at both times and are not meaningfully matched, the two samples are independent groups for the purpose of procedure selection. The appropriate mean comparison is an independent two-sample t procedure, not a paired t procedure. A change over time could also reflect changes in patients, staffing, or other conditions, so the design alone may not support a causal conclusion.

Worked Example: Different Patients Before and After a Policy Change

A hypothetical clinic samples 25 patients before a scheduling change and a different 25 patients afterward. Waiting time has a mean of 42 minutes and standard deviation of 12 minutes before the change, and a mean of 35 minutes and standard deviation of 10 minutes afterward. Estimate the mean difference, before minus after, using a 95% confidence interval.

Check the design and conditions. The patients in the two samples are different, so there are no participant-level pairs. Assume the samples are independent and reasonably representative of the relevant patient populations. With 25 observations per group, inspect each group for severe outliers or strong skewness; suppose neither is present. Use Welch’s independent two-sample procedure.

The estimated difference is \(42-35=7\) minutes, and

$$ SE=\sqrt{\frac{12^2}{25}+\frac{10^2}{25}} =\sqrt{5.76+4.00} =\sqrt{9.76} =3.124\text{ minutes}. $$

Welch’s degrees of freedom are

$$ df= \frac{(5.76+4.00)^2} {\frac{5.76^2}{24}+\frac{4.00^2}{24}} \approx 46.49. $$

Using \(t^*\approx2.01\), the interval is

$$ 7\pm 2.01(3.124) =7\pm 6.28 =(0.72,\ 13.28)\text{ minutes}. $$

The interval estimates that the population mean waiting time before the change exceeded the population mean afterward by about 0.72 to 13.28 minutes. Because these are different samples observed at different times, the result is a comparison of means, not a within-patient average change.

Why Pairing Changes the Calculation

When the same person is measured twice, the two measurements may be related. A participant who tends to have a high measurement at baseline may also tend to have a high measurement afterward. A paired analysis preserves that relationship by calculating one difference per participant. The variance identity below explains why that can matter:

$$ \operatorname{Var}(Y-X) =\operatorname{Var}(Y)+\operatorname{Var}(X) -2\operatorname{Cov}(X,Y). $$

If the paired measurements tend to rise and fall together, their covariance is positive, so subtracting one from the other can reduce the variability of the differences. The paired standard error is based on the observed spread of those differences. Treating the measurements as independent would discard the links and use a different standard error, potentially giving an incorrect analysis. Conversely, labeling two unrelated samples as paired invents links that the design does not provide.

A related design has each person measured before and after in both a treatment group and a control group. Each person’s own change can be calculated, and those changes can then be compared between the two independent groups. In that analysis the individual before-after observations are paired within each person, while the group-level comparison is between independent samples of changes. The question and the structure of the data determine which quantity is being compared.

1
Identify the units.
Determine who or what generated each observation, and whether the same unit contributed both measurements.
2
Check for genuine pairs.
Use a paired procedure only when each measurement has a meaningful partner, such as the same participant at two times or a deliberate match.
3
Name the target and direction.
For pairs, define the difference order and target the mean difference. For separate groups, state which population mean is subtracted from which.
4
Check conditions and choose the t method.
For pairs, assess independent pairs and the distribution of differences. For separate groups, assess independence and group distributions; use Welch unless a common variance is justified.

Common Mistake: Pairing by Calendar Time

A common mistake is to reason, “The data are from before and after, so use a paired t procedure.” That conclusion does not follow from the timing. If the “before” observations are from one set of people and the “after” observations are from another, there is no participant-level difference to calculate. Use an independent two-sample t procedure instead. The reverse mistake is to treat repeated measurements on the same people as independent merely because they appear in separate columns. Keep the links, calculate one change per person, and use paired t inference when its conditions are reasonable.

Key takeaway: Choose the procedure from the data’s dependence structure. Same participants measured twice: analyze their differences with a paired t procedure. Separate, unpaired groups: compare their means with an independent two-sample t procedure, generally using Welch’s method unless equal variances are justified.

Check Your Understanding

For each scenario, identify the appropriate t procedure and briefly justify your choice.

  1. A researcher measures blood pressure in the same 18 volunteers before and after a hypothetical exercise program. What is the observational unit for the paired differences?
  2. A school compares exam scores from one randomly selected class in fall with scores from a different class in spring. No students appear in both samples. Is a paired or independent two-sample procedure appropriate?
  3. In a paired analysis, a student defines every difference as before minus after. What does a positive mean difference represent?
  4. Two independent groups have different sample standard deviations. Does that automatically make a t comparison impossible? Which independent-samples method is generally appropriate?
  5. A report says “before and after, therefore paired.” What additional design information is needed to justify that statement?

Previous

Next

Back to Tutorials