Refreshment Samples and Their Identifying Assumptions

Why a Later Sample Can Help With Attrition

In a longitudinal study, some members of the original panel may not provide an outcome at a later wave. The panel then reveals the outcome distribution among those who responded, but not necessarily among those who dropped out. A refreshment sample is a new sample collected at that later wave from a population intended to match the population represented by the original panel. Because its members are sampled anew, it can provide information about the later-wave outcome distribution without being limited to the original panel’s respondents.

The key is that a refreshment sample usually does not tell us which of its members would have responded had they belonged to the original panel. It supplies a population-level outcome distribution, not outcomes labelled by the original panel’s response status. To use it for attrition, we combine that marginal distribution with the panel’s response rate and respondents’ outcome distribution. This link requires substantive assumptions; it does not follow merely from collecting a second sample.

Definition: Let \(Y\) be the outcome at the later wave for members of the original panel, and let \(R=1\) indicate that a panel member supplied \(Y\) at that wave; \(R=0\) indicates attrition and an unobserved \(Y\). A refreshment sample is a later-wave sample intended to estimate the distribution of \(Y\) in the target population represented by the original panel. It does not ordinarily observe the original panel members’ \(R\) status.

For a binary outcome, write \(q=P(Y=1)\) for the target population prevalence, \(r=P(R=1)\) for the panel response proportion, \(q_1=P(Y=1\mid R=1)\) for the respondent prevalence, and \(q_0=P(Y=1\mid R=0)\) for the attriter prevalence. The panel estimates \(r\) and \(q_1\); a representative refreshment sample can estimate \(q\). The attriter prevalence is then identified by a mixture equation, if the required link between samples is credible.

The Mixture Relation and Its Assumptions

The population at the later wave can be divided into the original panel’s responders and attriters. By the law of total probability, its outcome distribution is a mixture of the distributions in those two groups. For a binary outcome, this gives:

$$ q = r q_1 + (1-r)q_0 $$

When \(r<1\), rearrangement identifies the attriter prevalence:

q_0 = \frac{q-rq_1}{1-r}

The same argument works for the full outcome distribution. Let \(F(y)=P(Y\le y)\) denote the target population cumulative distribution function, and let \(F_1(y)=P(Y\le y\mid R=1)\) and \(F_0(y)=P(Y\le y\mid R=0)\). If \(r=P(R=1)\), then:

F(y)=rF_1(y)+(1-r)F_0(y), \qquad F_0(y)=\frac{F(y)-rF_1(y)}{1-r}

These equations are identification statements about population distributions. In applications, the quantities are estimated from samples, so sampling variation also affects the calculation. The equations do not make the assumptions true; they show what follows if the observed panel and refreshment-sample distributions refer to the same target population and outcome.

Assumptions for the basic mixture identification: (1) The panel response proportion and respondent outcome distribution refer to the original panel at the later wave. (2) The refreshment sample represents the target population outcome distribution relevant to that original panel at that wave, with its sampling design and nonresponse handled appropriately. (3) \(Y\) has the same definition, timing, and measurement in both samples. (4) The population represented by the refreshment sample aligns with the original panel’s target population; for example, changes in cohort composition or time do not make their outcome distributions differ. If conditioning on covariates, the corresponding alignment must hold within the covariate strata as well.

The alignment condition is often the most consequential. A fresh cross-section from the same geographic area is not automatically representative of the original panel’s later-wave outcomes. The original panel may be a cohort whose members have aged, moved, or changed in ways that distinguish it from the current cross-section. Likewise, a refreshment sample with substantial unaddressed nonresponse may fail to estimate even its own source population’s outcome distribution. Design weights can account for known sampling probabilities, but they do not by themselves repair a mismatch in target population.

Deriving the Attriter Distribution

The derivation is short but clarifies what the refreshment sample contributes. Since every original panel member is either a responder or an attriter, the events \(R=1\) and \(R=0\) partition the panel. For any outcome set \(B\), add the probabilities of being in \(B\) within those two groups:

P(Y\in B) =P(Y\in B,R=1)+P(Y\in B,R=0) =rP(Y\in B\mid R=1)+(1-r)P(Y\in B\mid R=0)

Under the population-alignment assumption, the refreshment sample identifies the left side, while the panel identifies \(r\) and the respondent conditional distribution. Subtracting the responder contribution and dividing by \(1-r\) yields the attriter conditional distribution. For \(B=(-\infty,y]\), this is the CDF formula above; for \(B=\{1\}\), it is the binary prevalence formula.

The calculation requires \(r<1\) to identify an attriter distribution by subtraction. If \(r=1\), there are no attriters in the target panel, so an attriter distribution is not needed. If \(r=0\), the refreshment distribution equals the attriter distribution under alignment, because the entire panel is classified as attrited; there is no respondent distribution to subtract. In intermediate cases, \(0<r<1\), both panel components contribute to the mixture.

Worked Example: Inferring a Binary Outcome Among Attriters

Suppose a hypothetical panel starts with 500 people. At the later wave, 300 respond, and 90 of those respondents have a positive binary outcome. A representative refreshment sample of 1,000 people from the aligned target population has 400 positive outcomes. Assume the outcome definition and timing match across samples, refreshment-sample nonresponse is adequately addressed, and the original cohort’s later-wave outcome prevalence equals that of the refreshment-sample target population.

The panel response proportion is \(r=300/500=0.60\), and the respondent prevalence is \(q_1=90/300=0.30\). The refreshment estimate of the target prevalence is \(q=400/1000=0.40\). Substituting into the mixture formula:

\hat q_0 =\frac{\hat q-\hat r\hat q_1}{1-\hat r} =\frac{0.40-(0.60)(0.30)}{0.40} =\frac{0.22}{0.40} =0.55

The estimated attriter prevalence is 0.55, or 55%. As a count check, the estimated total number of positive outcomes in a panel of 500 is \(500(0.40)=200\). Respondents account for 90 of those, leaving \(200-90=110\) positives among \(500-300=200\) attriters; \(110/200=0.55\). This is a population-distribution inference under the stated alignment assumptions, not direct observation of the attriters’ outcomes.

Using Baseline Covariates

A single overall mixture can conceal important differences across baseline covariates. If \(X\) is measured comparably and the refreshment data identify the target outcome distribution conditional on \(X=x\), apply the mixture within each stratum:

P(Y=1\mid X=x) =P(R=1\mid X=x)P(Y=1\mid R=1,X=x) +P(R=0\mid X=x)P(Y=1\mid R=0,X=x)

This gives a stratum-specific attriter prevalence when the stratum’s response proportion is below one. The result still requires alignment of the conditional outcome distribution between the original cohort and refreshment sample at each relevant \(x\). A covariate measured only at the later wave may not be interchangeable with its baseline value; the matching and timing of \(X\) must be justified.

Worked Example: Stratifying by a Baseline Covariate

Consider a hypothetical panel with two baseline covariate groups, low and high. In the low group, 150 of 200 people respond, and 45 respondents have \(Y=1\). The refreshment sample, aligned to this group, estimates \(P(Y=1\mid X=\text{low})=0.40\). In the high group, 100 of 200 respond, 60 respondents have \(Y=1\), and the aligned refreshment estimate is \(P(Y=1\mid X=\text{high})=0.50\). Assume the covariate definitions, outcome measurement, and target-population alignment hold within both groups.

For the low group, \(r_{\text{low}}=150/200=0.75\) and \(q_{1,\text{low}}=45/150=0.30\). Thus:

q_{0,\text{low}} =\frac{0.40-(0.75)(0.30)}{1-0.75} =\frac{0.175}{0.25} =0.70

For the high group, \(r_{\text{high}}=100/200=0.50\) and \(q_{1,\text{high}}=60/100=0.60\). Thus:

q_{0,\text{high}} =\frac{0.50-(0.50)(0.60)}{1-0.50} =\frac{0.20}{0.50} =0.40

There are 50 low-group attriters and 100 high-group attriters. Their estimated positive count is \(50(0.70)+100(0.40)=35+40=75\), so the overall attriter prevalence is \(75/150=0.50\), or 50%. The calculation weights strata by their numbers of attriters, not by their proportions in the refreshment sample. If the assumptions hold, this reveals that the overall result masks different inferred prevalences across the two groups.

Diagnostics, Alternatives, and Limitations

A basic diagnostic is whether the inferred distribution is mathematically possible. For a binary outcome, the calculated \(q_0\) must lie between zero and one. Equivalently, the refreshment estimate \(q\) must lie between the respondent contribution \(rq_1\) and that contribution plus the maximum possible attriter contribution:

rq_1 \le q \le rq_1+(1-r)

For a general outcome, the inferred \(F_0(y)\) must be a valid CDF: its values must lie between zero and one, be nondecreasing in \(y\), and have the appropriate limiting values. Failure can reflect sampling noise, incompatible weighting or measurement, or a failure of population alignment. It is not a reason to silently truncate an impossible estimate and report it as identified. With finite samples, uncertainty intervals and joint sampling variation should also be considered; a point estimate inside the admissible range does not establish that the assumptions hold.

Worked Example: An Incompatible Plug-In Estimate

Suppose a hypothetical panel has \(r=0.80\), with respondent prevalence \(q_1=0.70\), while a refreshment sample estimates a target prevalence \(q=0.52\). Assume the measures are intended to align, but first check whether these plug-in values yield a possible attriter prevalence.

Substitution gives:

\hat q_0 =\frac{0.52-(0.80)(0.70)}{1-0.80} =\frac{-0.04}{0.20} =-0.20

A prevalence of \(-0.20\) is impossible. The equivalent feasibility check gives \(rq_1=0.56\), but \(q=0.52<0.56\), violating the required lower bound. These point estimates cannot all be exact parts of one population mixture. Sampling variation might produce this discrepancy in finite data, but so might cohort mismatch, measurement differences, or an unrepresentative refreshment sample. The correct conclusion is to investigate uncertainty and assumptions, not to interpret \(-0.20\) or automatically replace it with zero.

A refreshment sample differs from a validation sample: a validation design obtains information about outcomes or missingness for members whose original panel outcome was missing, whereas a refreshment sample estimates a later-wave population distribution without generally identifying those members’ response labels. The latter can identify an attriter distribution only through the mixture and alignment assumptions. Without a credible refreshment link, the earlier material on partial identification and sensitivity analysis provides ways to describe what is known without asserting point identification. A refreshment sample may narrow uncertainty, but it does not eliminate uncertainty about assumptions that the two samples share a target distribution.

The main limitations follow directly from this distinction. The refreshment sample does not identify the joint distribution of \(Y\) and original-panel response status by itself. Its usefulness depends on the target population, timing, covariate definitions, and selection into the refreshment sample. Stratification can make the alignment assumption more specific, but sparse or empty strata can make the relevant components unestimable or unstable. These checks are part of the identification argument, not optional refinements after the calculation.

Specify the target.
State which original panel, later wave, outcome, and population the refreshment sample is intended to represent.

Check comparability.
Assess sampling and nonresponse adjustments, outcome measurement and timing, and cohort alignment; state any covariate-specific alignment needed.

Apply and audit the mixture.
Combine the panel response proportion and respondent distribution with the refreshment distribution, then check that the result is a valid distribution and account for sampling uncertainty.

Common mistake: “The refreshment sample has the same average outcome as the population, so it directly tells us the attriters’ average.” This confuses the population marginal distribution with the distribution conditional on original-panel attrition. The refreshment sample does not label its members as would-be responders or attriters. The correction is to use the mixture relation with the panel response proportion and respondent distribution, and to make the population-alignment assumption explicit.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A panel has response proportion \(r=0.50\), respondent prevalence \(q_1=0.20\), and an aligned refreshment estimate \(q=0.35\). What attriter prevalence follows from the mixture formula?
Why does a refreshment sample not, on its own, reveal which outcomes belong to original-panel attriters?
In a stratum, suppose \(r=0.75\), \(q_1=0.60\), and the aligned population prevalence is \(q=0.50\). Is the implied attriter prevalence possible? Show the calculation.
Name two reasons a later cross-section from the same region might fail to identify outcomes among attriters in an original panel cohort.

Solutions. (1) \(q_0=[0.35-(0.50)(0.20)]/(1-0.50)=0.25/0.50=0.50\), so the implied attriter prevalence is 50%, under the alignment assumptions. (2) Refreshment members are newly sampled and ordinarily have no original-panel response status; the sample estimates a marginal outcome distribution, not \(P(Y\mid R=0)\). (3) \(q_0=[0.50-(0.75)(0.60)]/0.25=0.05/0.25=0.20\), which is a possible prevalence. (4) The original cohort may differ from the current cross-section because of aging, migration, or other compositional change; alternatively, refreshment-sample nonresponse or different outcome measurement may prevent it from representing the target distribution.