Refreshment Samples for Diagnosing Attrition

Why Add a Refreshment Sample?

A panel follows the same units over time, but some original participants stop responding. The observed outcomes at a later wave therefore describe retained participants, who may differ from the population the analysis is meant to represent. A refreshment sample supplies new observations at that later wave: a sample drawn to represent the target population at the time of measurement. Comparing it with the retained panel can show whether the retained sample’s outcome mean differs from the target-population mean.

This comparison is a diagnostic, not an automatic cure for attrition. A refreshment sample can reveal a mean discrepancy and, with the cohort’s retention rate, identify the mean outcome among attriters under a population-alignment assumption. It does not by itself reveal each attriter’s outcome, establish why people dropped out, or justify any particular adjustment.

The target population must be specified carefully. In the examples below, it is the original baseline cohort, including members who later attrit. A refreshment sample must then represent that cohort at wave \(t\), not an unaligned population that includes new entrants or excludes cohort members. If the intended estimand instead concerns the current population, including entrants, the sample and comparison must be defined for that population, and the original cohort’s retention rate cannot be used as though it were the current population’s attrition rate.

Comparing Retained and Population Outcomes

Suppose the refreshment sample is representative of the target population and measures the same outcome, at the same wave and on the same scale, as the retained panel. The difference \(\Delta_t=E(Y_{it}\mid R_{it}=1)-E(Y_{it})\) is the mean gap between retained participants and the target population. A positive gap means retained participants have a higher mean outcome; a negative gap means they have a lower mean. This gap describes selection into observed response for the outcome mean. It is not a causal effect of retention or dropout.

In practice, the two expectations are estimated from the retained panel and refreshment sample. Their comparison depends on the sampling designs: weights may be needed if either sample is not a simple random sample. It also requires comparable outcome measurement and a refreshment sample whose own nonresponse has been addressed or is negligible for the stated estimand.

Using the Retention Rate to Describe Attriters

A refreshment sample provides an estimate of the target-population mean, while the original cohort provides the fraction retained. Together these identify the mean outcome among attriters, provided both quantities refer to the same target population and wave. Let \(q_t=P(R_{it}=1)\) be the retention fraction. The population is a mixture of retained and attrited cohort members, so

Solving this identity for the attriter mean gives

This is a mixture decomposition, not a model for why attrition occurred. It recovers one group mean from the overall mean, the retained-group mean, and the group share. It does not identify the distribution of outcomes among attriters or pair any refreshment-sample member with a particular missing panel outcome.

Adjustment by Covariate Standardization

A useful adjustment is possible when attrition is related to measured covariates \(X\), and outcomes are comparable between retained and attrited people with the same \(X\). In precise terms, the key mean assumption is \(E(Y_{it}\mid X,R_{it}=1)=E(Y_{it}\mid X)\). This is conditional mean exchangeability for the outcome: within covariate groups, retained members represent the target group’s mean outcome. The covariates must be measured in both the retained panel and the refreshment sample, and each covariate pattern with positive target-population probability must also have retained observations. The latter is an overlap condition.

Under this assumption, estimate the outcome mean within covariate groups among retained panel members, then average those group means using the covariate distribution in the refreshment sample. With discrete \(X\), the target mean is estimated by \(\sum_x P(X=x)E(Y_{it}\mid X=x,R_{it}=1)\). For continuous covariates, the same idea requires standardization over their target distribution or equivalent weighting. This approach corrects measured composition differences; if outcomes still differ by retention status within \(X\), the adjustment does not remove the resulting bias.

Diagnosing Where the Gap Arises

A single overall difference can hide important patterns. If the refreshment sample and retained panel both measure \(X\), compare their \(X\) distributions and, where sample sizes allow, compare outcome means within \(X\) groups. A difference in the overall means that disappears after standardization is consistent with observed composition differences explaining the gap. A remaining within-group discrepancy suggests that measured \(X\) alone is insufficient, although sampling error and measurement differences are also possible explanations.

What a Refreshment Sample Does Not Identify

The refreshment sample supplies a cross-sectional view of the target population at wave \(t\); it does not link its members to the original panel’s individual histories. Consequently, it cannot directly reveal the joint distribution of a missing wave-\(t\) outcome with that person’s earlier outcomes, nor can it show which individuals would have remained under a different response process. This matters for dynamic or causal questions, where the target may involve histories or counterfactual outcomes rather than a single population mean.

For a mean outcome, covariate standardization or inverse-probability weighting can be appropriate when the relevant measured predictors, conditional exchangeability, and overlap assumptions are defensible. A refreshment sample can help estimate the target covariate distribution and diagnose whether retained-panel outcomes align with it. If selection remains related to unobserved outcomes after conditioning on measured covariates, an adjustment requires additional assumptions, such as a specified selection model or sensitivity analysis; the refreshment sample alone does not settle that choice.

Practical Interpretation

A credible refreshment-sample analysis makes four items explicit: the target population, the wave and outcome being compared, the sampling and response assumptions for both samples, and the assumptions behind any adjustment. Report the retained-panel mean and refreshment estimate in outcome units, alongside the estimated gap and its uncertainty. If attriter means are recovered by mixture decomposition, label them as implied group averages and state the population-alignment condition. If standardization changes the estimate, describe the covariate distribution that generated the change and the conditional assumption required for interpretation.

A refreshment sample is most informative when treated as a benchmark with a clearly defined role. It can reveal that the observed panel is not representative for a particular outcome mean, quantify the aggregate attriter mean under a mixture identity, and support covariate-based adjustments under stated conditions. It cannot make unobserved outcomes observable or replace the assumptions needed to generalize from retained panel members.