Worked Analysis of an Encouragement Trial With Attrition

Why Analyze Assignment Before Treatment Received?

In an encouragement trial, researchers randomly assign an offer, reminder, or other encouragement intended to change whether people receive a treatment. Some people assigned encouragement may not take up the treatment, while some not assigned encouragement may receive it. The randomized comparison is therefore a comparison by assignment, not by treatment received. That assignment contrast remains the natural starting point for causal analysis.

Incomplete outcome measurement adds a second distinction. The outcomes of people who respond may not represent the outcomes of everyone assigned to that arm. A complete-case difference is directly calculable, but it does not automatically estimate the full-sample assignment effect. The worked analysis below keeps these questions separate: how assignment changes take-up, how many outcomes are missing in each arm, what an observed-outcome contrast says under a missingness assumption, and when a complier effect can be interpreted.

Notation and Assumptions

Definition: Let \(Z_i=1\) denote assignment to encouragement and \(Z_i=0\) denote assignment to the comparison arm. Let \(D_i=1\) if individual \(i\) receives the treatment and \(D_i=0\) otherwise. Let \(Y_i(z)\) be the outcome individual \(i\) would have under assignment \(z\), and let \(R_i=1\) indicate that the outcome is observed. The assignment effect is the average contrast \(E[Y_i(1)-Y_i(0)]\). The first stage is the assignment contrast in treatment take-up, \(E[D_i\mid Z_i=1]-E[D_i\mid Z_i=0]\).

With complete outcome data and randomized assignment, the difference in mean outcomes by assignment estimates the assignment effect. Randomization means assignment is independent of the potential outcomes and potential treatment-receipt statuses; the usual no-interference and well-defined-treatment conditions are also understood. In a finite randomized trial, the arm difference is an estimate of the effect for the trial population, with uncertainty arising from assignment.

When outcomes are missing, the complete-case means compare observed individuals in each arm. To interpret their difference as the full-sample assignment effect requires a further missingness assumption. One sufficient condition is that, within each assignment arm, outcome observation is independent of the outcome that would be realized in that arm. If response instead depends on outcome, even after conditioning on assignment, the observed means need not recover the full-arm means. Similar response rates across arms do not by themselves establish this condition.

Worked Analysis of a Hypothetical Trial

Suppose 400 people are randomly assigned, 200 to encouragement and 200 to comparison. Treatment receipt is recorded for everyone. Outcomes are measured for some, but not all, participants. All figures below are hypothetical.

Worked Example: Assignment, Take-Up, and Attrition

In the encouragement arm, 140 of 200 people receive treatment and 160 have an observed outcome. In the comparison arm, 60 of 200 receive treatment and 150 have an observed outcome. Calculate the take-up and attrition contrasts.

Take-up. The take-up rate under encouragement is \(140/200=0.700\), or 70.0%. The comparison-arm take-up rate is \(60/200=0.300\), or 30.0%. Thus the estimated first stage is

\widehat{\Delta}_D = \frac{140}{200}-\frac{60}{200} = 0.700-0.300 = 0.400.

Assignment to encouragement raises treatment take-up by 40.0 percentage points in this sample. This is an assignment effect on treatment receipt, not an effect of receiving treatment on the outcome.

Outcome observation. The encouragement arm has \(160/200=0.800\), or 80.0%, of outcomes observed, so its attrition rate is \(40/200=0.200\), or 20.0%. The comparison arm has \(150/200=0.750\), or 75.0%, observed and an attrition rate of \(50/200=0.250\), or 25.0%. The observed-outcome rate is 5 percentage points higher in the encouragement arm. This difference documents differential attrition; it does not establish whether missing outcomes bias the outcome contrast.

Estimating the Assignment Effect With Missing Outcomes

Suppose the observed outcomes have mean 12 and standard deviation 4 in the encouragement arm, and mean 10 and standard deviation 4 in the comparison arm. The complete-case difference is 2 outcome units. The arithmetic is straightforward; the interpretation depends on whether the observed outcomes represent the outcomes that would have been measured for all assigned participants.

Worked Example: An Observed-Outcome Contrast

Using the hypothetical trial counts above, calculate the complete-case contrast and its approximate standard error. State the assumption needed to interpret it as an assignment effect.

The observed means are \(\bar{Y}_{1,\mathrm{obs}}=12\) for \(n_1=160\) respondents assigned encouragement and \(\bar{Y}_{0,\mathrm{obs}}=10\) for \(n_0=150\) respondents assigned comparison. The complete-case contrast is

\widehat{\Delta}_{Y,\mathrm{obs}} = \bar{Y}_{1,\mathrm{obs}}-\bar{Y}_{0,\mathrm{obs}} = 12-10 = 2 \text{ outcome units}.

Using the observed standard deviations as estimates of the within-arm standard deviations, the independent-arm standard error is

SE(\widehat{\Delta}_{Y,\mathrm{obs}}) = \sqrt{\frac{4^2}{160}+\frac{4^2}{150}} = \sqrt{0.2067} \approx 0.4546 \text{ outcome units}.

A normal-approximation 95% interval is \(2\pm1.96(0.4546)\), or approximately \([1.109,\,2.891]\) outcome units. This interval describes sampling uncertainty under the analysis assumptions; it does not account for bias from outcome-dependent attrition. If, within each assignment arm, response is independent of the potential outcome under that assignment, the complete-case means estimate the corresponding full-arm means, and the contrast estimates the assignment effect. Without such an assumption or another justified adjustment, the result is an observed-case contrast, not an established full-sample assignment effect.

The treatment-receipt contrast is calculated from all 400 people, so outcome attrition does not remove its observations in this example. If treatment receipt were also missing, its arm-specific denominators and missing-data assumptions would need to be reported separately. Reporting the number assigned, the number treated, and the number with outcomes by arm makes these distinctions auditable.

What Can Be Learned Without a Missingness Assumption?

When the outcome has a known range, simple bounds show how much the missing outcomes could change the arm contrast. These bounds do not require missing outcomes to resemble observed ones. They may be wide, but they make the role of assumptions explicit.

Worked Example: Bounds From a Known Outcome Range

Suppose the outcome is known to lie between 0 and 20 for every participant. Use the observed means and response counts above to bound each arm's full-sample mean, then bound the assignment contrast.

In the encouragement arm, the 160 observed outcomes sum to \(160(12)=1{,}920\). If the 40 missing outcomes are all 0, the full-arm mean is \(1{,}920/200=9.600\). If they are all 20, the mean is \((1{,}920+40(20))/200=13.60\). Therefore, the encouragement-arm mean is bounded by \([9.600,\,13.60]\).

In the comparison arm, the 150 observed outcomes sum to \(150(10)=1{,}500\). The 50 missing outcomes being all 0 gives a full-arm mean of \(1{,}500/200=7.500\); their all being 20 gives \((1{,}500+50(20))/200=12.50\). The comparison-arm mean is bounded by \([7.500,\,12.50]\).

The smallest possible arm contrast under these separate bounds is \(9.600-12.50=-2.900\) outcome units. The largest is \(13.60-7.500=6.100\) outcome units. Thus the contrast is bounded by \([-2.900,\,6.100]\) outcome units. These bounds are wide and include zero: the observed difference of 2 does not, by itself, establish the sign of the full-sample assignment effect. The bounds reflect the outcome range and observed data, not sampling uncertainty from random assignment.

When a Complier Effect Is Interpretable

An encouragement trial can also identify an effect for compliers, but only under additional assumptions. A complier is a person who would take up treatment if assigned encouragement and would not take it up if assigned comparison. The complier effect is not the same as the assignment effect: it concerns the subgroup whose treatment receipt is changed by assignment.

For the standard Wald interpretation, assignment must be randomized, encouragement must affect treatment take-up (a nonzero first stage), and treatment receipt must be monotone: no one would take up treatment when assigned comparison but refuse it when assigned encouragement. An exclusion restriction is also required: assignment affects the outcome only through treatment receipt. Under these conditions, and with complete outcomes or a justified procedure for recovering the full-sample assignment effect, the ratio of the assignment effect on the outcome to the first stage identifies the average treatment effect among compliers.

Theorem: Under random assignment, exclusion, monotonicity, a nonzero first stage, and valid identification of the assignment effect on the outcome despite missingness, the complier average causal effect is the assignment effect on the outcome divided by the assignment effect on treatment receipt. The result is an effect for compliers, not an average treatment effect for everyone in the trial.

Worked Example: Calculating the Complier Effect

Suppose the missingness assumption in the observed-outcome example is credible, so the estimated assignment effect on the outcome is 2. The estimated first stage is 0.400. Calculate the Wald estimate and state its interpretation.

The Wald estimate is

\widehat{\tau}_{C} = \frac{\widehat{\Delta}_Y}{\widehat{\Delta}_D} = \frac{2 \text{ outcome units}}{0.400} = 5 \text{ outcome units per treatment}.

Under the stated assumptions, this estimates the average causal effect of treatment receipt for compliers: those whose receipt is changed by assignment to encouragement. It does not mean that every treated participant gains 5 units, nor does it establish that the treatment effect for always-takers or never-takers is 5 units. If the observed-outcome contrast is biased by attrition, the ratio inherits that problem; assumptions about compliance alone do not repair missing outcome data.

Deriving the Wald Interpretation

The ratio has a simple justification under the assumptions just stated. Random assignment makes the arm contrast in outcomes equal, in expectation, to the assignment effect. Exclusion means assignment affects outcomes only by changing treatment receipt. Monotonicity divides people into always-takers, never-takers, and compliers; there are no defiers. Always-takers and never-takers have no assignment-induced change in receipt, so they contribute zero to the assignment effect on receipt. Only compliers contribute, and their share of the population is the first stage.

\begin{aligned} E[D_i\mid Z_i=1]-E[D_i\mid Z_i=0] &=P(i\text{ is a complier}),\\ E[Y_i\mid Z_i=1]-E[Y_i\mid Z_i=0] &=P(i\text{ is a complier})\, E[Y_i(1)-Y_i(0)\mid i\text{ is a complier}]. \end{aligned}

When the first stage is nonzero, dividing the outcome contrast by the receipt contrast cancels the complier proportion. The remaining quantity is the average treatment effect among compliers. If exclusion fails, for example because encouragement itself changes behavior or access to other services, the outcome contrast can include a direct encouragement effect and this derivation no longer identifies the treatment effect among compliers.

Common Pitfall and Practical Reporting

Incorrect reasoning: “The respondents assigned encouragement had outcomes 2 units higher, so treatment caused a 2-unit increase for everyone who received it.” This statement makes two unjustified moves: it treats a potentially selected observed-case contrast as the full-sample assignment effect, then treats an assignment effect as the effect of treatment receipt for all treated people.

Correction: Report the assignment contrast as an observed-case result unless missingness assumptions or a justified adjustment support a full-sample interpretation. Report take-up by assignment arm, and distinguish the first stage from the outcome contrast. A complier effect is a further, assumption-dependent interpretation of their ratio. It should be described as a complier effect only when randomization, exclusion, monotonicity, a nonzero first stage, and valid handling of missing outcomes are defensible.

A clear analysis should state the numbers assigned, treated, and observed in each arm; give arm-specific take-up and outcome-observation rates; report the assignment contrast with its uncertainty and missingness assumptions; and label any Wald ratio and its assumptions. The randomized assignment comparison is not a per-protocol comparison of people grouped by treatment received. That distinction is essential when receipt is not fully controlled by the study.

Check Your Understanding

Use the trial figures and definitions above to answer these questions.

What is the first stage, in percentage points, and what does it measure?
Why does a 5-percentage-point difference in attrition rates not, by itself, show whether the observed outcome contrast is biased?
Under what additional assumption can the complete-case outcome contrast estimate the full-sample assignment effect?
With outcomes bounded from 0 to 20, what are the calculated lower and upper bounds for the full-sample assignment contrast?
What assumptions allow the Wald ratio to be interpreted as a complier effect, and why is that effect not an average for all treated participants?