Equivalence Tests for Practically Important Pre-Trends

Why Test for Equivalence?

A pre-treatment trend test often asks whether the estimated difference in outcome trends between treated and comparison groups is distinguishable from zero. But a large p-value does not establish that the trends are sufficiently alike for a difference-in-differences (DiD) analysis. The estimate could be imprecise enough to include both zero and trend differences large enough to matter.

An equivalence test asks a more targeted question: is the pre-treatment trend difference small enough, in context, to be considered practically negligible? Answering requires a margin that describes how large a difference could be while still being acceptable for the intended analysis. The margin is a substantive judgment, not a number chosen because it makes the observed result pass.

Definition: Let \(\beta_{\text{pre}}\) denote a specified difference between the treated and comparison groups’ pre-treatment outcome trends, measured on a stated outcome and time scale. Choose an equivalence margin \(\Delta>0\). The trends are considered equivalent for this diagnostic purpose when \(-\Delta<\beta_{\text{pre}}<\Delta\). The margin describes the largest trend difference in either direction that is considered practically negligible.

For example, if the outcome is a test score and the trend is measured in score points per year, then \(\Delta=0.20\) means that differences smaller than 0.20 points per year in either direction are treated as practically negligible for the specified purpose. The units and the definition of the trend parameter matter: a bound for points per year is not interchangeable with a bound for a cumulative change over several years.

Hypotheses and the Confidence-Interval Criterion

Equivalence reverses the usual burden of evidence. The null hypothesis says that the trend difference is not within the acceptable bounds; the alternative says it is strictly inside them:

H_0:\ \beta_{\text{pre}}\leq-\Delta\ \text{or}\ \beta_{\text{pre}}\geq\Delta \qquad\text{versus}\qquad H_a:\ -\Delta<\beta_{\text{pre}}<\Delta.

This is tested using two one-sided tests (TOST). One test asks whether the trend difference is greater than the lower bound, and the other asks whether it is less than the upper bound. At significance level \(\alpha\), equivalence is concluded only if both one-sided null hypotheses are rejected. A result on just one side is not enough.

When an estimate \(\hat{\beta}_{\text{pre}}\) has standard error \(SE\) and a \(t\) reference distribution with \(df\) degrees of freedom, the two test statistics are:

t_L=\frac{\hat{\beta}_{\text{pre}}-(-\Delta)}{SE}, \qquad t_U=\frac{\hat{\beta}_{\text{pre}}-\Delta}{SE}.

The lower-bound test rejects when \(t_L\) is sufficiently large; the upper-bound test rejects when \(t_U\) is sufficiently negative. Equivalently, for the usual \(t\)-based procedure, equivalence at level \(\alpha\) is established when the \(100(1-2\alpha)\%\) confidence interval for \(\beta_{\text{pre}}\) lies wholly inside \((-\Delta,\Delta)\). At \(\alpha=0.05\), this is a 90% confidence interval.

Why the interval rule works: A two-sided \(100(1-2\alpha)\%\) \(t\) confidence interval has endpoints \(\hat{\beta}_{\text{pre}}\pm t_{1-\alpha,df}SE\). Its lower endpoint exceeds \(-\Delta\) exactly when the lower one-sided statistic exceeds \(t_{1-\alpha,df}\); its upper endpoint is below \(\Delta\) exactly when the upper one-sided statistic is less than \(-t_{1-\alpha,df}\). Thus the interval is wholly inside the equivalence bounds exactly when both one-sided tests reject at level \(\alpha\).

This procedure assumes that the trend parameter and margins have been defined on the same scale, that the estimated standard error and reference distribution are appropriate for the design, and that the pre-treatment observations used for the diagnostic are genuinely pre-treatment. The interval criterion does not rescue an unsuitable model or an invalid standard error.

Worked Examples

Worked Example: A Nonsignificant Test That Does Not Establish Equivalence

Suppose a hypothetical evaluation tracks a continuous outcome in treated and comparison groups over three pre-treatment years. The treated-minus-comparison differences in the annual outcome means are 4.00, 4.12, and 4.24 points. The fitted linear difference in trends is \(\hat{\beta}_{\text{pre}}=0.12\) points per year. Assume its standard error is \(0.08\) points per year and the applicable reference distribution has \(df=24\). Investigators specified \(\Delta=0.25\) points per year before examining this estimate.

The conventional test of zero trend difference gives \(t=0.12/0.08=1.50\), with a two-sided p-value of about 0.147. It does not reject a zero trend difference at the 5% level. That alone does not show equivalence.

For the equivalence assessment, the 90% confidence interval uses \(t_{0.95,24}\approx1.711\):

0.12\pm1.711(0.08) =0.12\pm0.1369 =[-0.0169,\ 0.2569]\ \text{points per year}.

The upper endpoint, 0.2569, exceeds the upper equivalence bound of 0.25. The interval is not wholly inside \((-0.25,0.25)\), so equivalence is not established. The same conclusion follows from TOST: \(t_L=(0.12+0.25)/0.08=4.625\), which rejects the lower-bound null, but \(t_U=(0.12-0.25)/0.08=-1.625\), which does not pass the 5% one-sided cutoff of approximately \(-1.711\). The data are compatible with a trend difference just large enough to exceed the prespecified margin.

Worked Example: Evidence That the Trend Difference Is Within the Bounds

In a second hypothetical evaluation, the outcome is a standardized score, and the estimated pre-treatment difference in linear trends is \(0.04\) score points per year. Suppose \(SE=0.06\) points per year, \(df=39\), and the prespecified margin is \(\Delta=0.15\) points per year.

The 90% confidence interval uses \(t_{0.95,39}\approx1.685\):

0.04\pm1.685(0.06) =0.04\pm0.1011 =[-0.0611,\ 0.1411]\ \text{score points per year}.

Both endpoints lie inside \((-0.15,0.15)\). To verify the two tests directly, \(t_L=(0.04+0.15)/0.06=3.167\) exceeds 1.685, and \(t_U=(0.04-0.15)/0.06=-1.833\) is less than \(-1.685\). Both one-sided tests reject at the 5% level. The result supports equivalence within the stated bounds: the data are sufficiently precise to rule out differences as large as either margin, under the specified model and assumptions.

This does not prove that the true trend difference is exactly zero. Nor does it prove that post-treatment trends would have remained parallel without treatment. It says that the pre-treatment trend difference is estimated precisely enough to fall within the prespecified range considered negligible for this diagnostic.

Worked Example: Why the Margin Must Reflect the Substantive Question

Suppose a hypothetical health program is evaluated using a monthly outcome rate, and the pre-treatment trend difference is estimated as \(0.03\) percentage points per month, with \(SE=0.04\) percentage points per month and \(df=29\). A 90% confidence interval uses \(t_{0.95,29}\approx1.699\):

0.03\pm1.699(0.04) =0.03\pm0.0680 =[-0.0380,\ 0.0980]\ \text{percentage points per month}.

If the substantively justified margin is \(\Delta=0.08\) percentage points per month, the interval extends above the upper bound, so equivalence is not established. If instead a margin of \(\Delta=0.12\) is justified in advance, the interval lies within \((-0.12,0.12)\), and equivalence is established at the 5% level. The arithmetic changes because the question changes: the latter conclusion is only that differences up to 0.12 percentage points per month count as negligible for this purpose.

It would be misleading to choose 0.12 only after seeing that it makes the interval fit. A defensible margin should be justified using subject-matter knowledge, the outcome’s scale, the length of the pre-treatment period, and the size of a trend departure that would materially affect the intended interpretation. Any comparison with a meaningful treatment effect must also account for the relevant time horizon and units.

Choosing Bounds and Reading the Result

The margin should be specified before the equivalence test is interpreted. It is not an estimate of the observed trend difference and it should not be selected to obtain a desired conclusion. Useful questions when setting it include:

What units describe the trend parameter: outcome points per year, percentage points per month, or another rate?
How large a pre-treatment trend difference would be substantively concerning over the period relevant to the DiD contrast?
Would a difference within the proposed margin be negligible relative to the outcome scale and the effect size of interest?
Is the margin defensible to subject-matter experts without reference to whether the observed confidence interval passes?

A very wide margin can make equivalence easy to establish while allowing a materially important departure. A very narrow margin can make equivalence difficult to establish even when the estimate is close to zero. The margin therefore expresses a substantive tolerance, while the confidence interval expresses statistical uncertainty. Both are needed to interpret the test.

In repeated-period settings, the chosen trend summary must also match the question. A single linear slope can summarize a steady difference in trends, but it can hide irregular pre-treatment movements. If the diagnostic concerns specific period-to-period contrasts, define bounds for those contrasts rather than assuming a slope captures every relevant pattern. Equivalence for one prespecified summary is not a blanket finding that all pre-treatment behavior matches.

Common Mistake: Treating a Large p-Value as Equivalence

The incorrect reasoning is: “The pre-trend test did not reject zero, so the groups had equivalent trends.” A test can fail to reject zero because its standard error is large, even when the confidence interval includes trend differences outside the acceptable bounds. As the first example showed, a nonsignificant zero test and a failed equivalence test can occur together.

The correction is to choose a substantively justified \(\Delta\), estimate the trend difference with an appropriate uncertainty measure, and check whether the confidence interval lies entirely within \((-\Delta,\Delta)\), or equivalently whether both one-sided tests reject. If it does not, report that equivalence was not established; do not describe that outcome as proof of a meaningful violation either. The interval may simply be too wide to distinguish negligible from consequential differences.

Key takeaway: Equivalence is a positive claim relative to prespecified bounds, not the absence of a statistically significant difference from zero. Even when supported, it is evidence about the defined pre-treatment trend summary and margin, not proof of the DiD identifying assumption.

Check Your Understanding

Use the bounds, confidence intervals, and test logic to answer these questions.

A 90% confidence interval for a pre-treatment trend difference is \([-0.08,0.11]\) outcome points per year. Does it establish equivalence for \(\Delta=0.10\)? For \(\Delta=0.12\)?
Why can a conventional two-sided test fail to reject a zero trend difference even when equivalence is not established?
For \(\hat{\beta}_{\text{pre}}=0.02\), \(SE=0.03\), and \(\Delta=0.08\), write the two TOST statistics before comparing them with their appropriate one-sided critical values.
Why should the equivalence margin be chosen before interpreting the estimated pre-treatment trend difference?
What does a successful equivalence test establish, and what does it not establish about the post-treatment counterfactual trend?