Tutorials › Causal Inference › Outcome-Dependent Sampling and Transport

Transportability and Generalizability · Tutorial 940 of 972

Outcome-Dependent Sampling and Transport

See why selecting participants by observed outcomes changes the sample and what information is needed to correct that distortion before transporting a causal effect.

Advanced 10 min read

What You'll Learn

  • Explain how outcome-dependent inclusion changes observed outcome distributions.
  • Distinguish the selected-sample effect estimate from the source-population causal effect.
  • Recover treatment-specific conditional means using known or estimable inclusion probabilities.
  • Identify the sampling information needed to recover absolute risks and risk differences.
  • Combine corrected source outcome models with the target covariate distribution.
  • Recognize positivity and transport assumptions required for identification.

Why Outcome-Dependent Inclusion Matters

A study sample is not necessarily a miniature version of the population from which it came. In particular, researchers may deliberately include people with an observed outcome more often than people without it. Case-control sampling is a familiar example: cases are recruited at a higher rate than controls. Such a design can be useful, but the sample outcome frequency is then partly set by the recruitment procedure rather than by the population.

For causal analysis, this distinction matters because inclusion occurs after the outcome is observed. Comparing outcomes among included treated and control participants without accounting for the selection process can produce a misleading estimate, even when treatment was randomized in the underlying source population. To transport an effect, the analyst must first recover the relevant source-population outcome information and then combine it with information about the target population.

Definition: Let \(R=1\) indicate that a person from the source population is included in the study sample, and let \(R=0\) indicate non-inclusion. Write \(A\) for treatment, \(Y\) for the observed outcome, and \(X\) for measured pre-treatment covariates. Outcome-dependent sampling occurs when the probability of inclusion varies with the observed outcome, so that, for some values of \(a,x\), \(P(R=1\mid A=a,Y=y,X=x)\) differs across values of \(y\).

The inclusion probability may also vary by treatment group or covariates. Define it as \(\pi(a,y,x)=P(R=1\mid A=a,Y=y,X=x,\text{source population})\). If inclusion probabilities are known from the design, or can be estimated using an appropriate sampling frame, they can be used to undo the outcome-dependent distortion. If the relevant probabilities are unknown, the observed sample distribution alone will generally not reveal the source population’s absolute outcome risks.

How Selection Changes the Outcome Distribution

For fixed \(A=a\) and \(X=x\), people with outcome value \(y\) are represented in the sample in proportion to their source-population frequency multiplied by their chance of inclusion. In density notation, this relationship is

$$ f(y\mid A=a,X=x,R=1) \propto \pi(a,y,x) f_{\text{source}}(y\mid A=a,X=x). $$

Thus, an outcome value with a higher inclusion probability appears more often in the sample than it would in the source population. The sample conditional distribution is not, in general, the source conditional distribution. The same point applies to binary outcomes: oversampling people with \(Y=1\) increases the observed sample risk.

Worked Example: Correcting Two Treatment-Group Risks

Consider a hypothetical randomized source cohort with 500 people in each treatment arm. Among the 500 assigned treatment, 50 have the outcome and 450 do not; among the 500 assigned control, 100 have the outcome and 400 do not. The true source risks are therefore \(50/500=0.10\) under treatment and \(100/500=0.20\) under control. Suppose the sampling design includes 80% of people with the outcome and 20% of people without it in each arm.

The selected treatment group contains \(0.80(50)=40\) people with the outcome and \(0.20(450)=90\) without it. Its sample risk is

$$ \frac{40}{40+90}=\frac{40}{130}=0.3077. $$

The selected control group contains \(0.80(100)=80\) people with the outcome and \(0.20(400)=80\) without it, giving a sample risk of \(80/160=0.5000\). The sample risk difference, treatment minus control, is \(0.3077-0.5000=-0.1923\), which is substantially different from the source-population risk difference of \(0.10-0.20=-0.10\).

Weight each included person by the inverse of their inclusion probability. In the treatment arm, the weighted outcome count is \(40/0.80=50\), and the weighted non-outcome count is \(90/0.20=450\). The corrected risk is \(50/(50+450)=0.10\). In the control arm, the corresponding weighted counts are \(80/0.80=100\) and \(80/0.20=400\), giving a corrected risk of \(100/500=0.20\). The corrected source risk difference is \(0.10-0.20=-0.10\), or 10 percentage points lower risk under treatment in this hypothetical cohort.

The design probabilities are the same in both arms here, but the sample risks are still distorted because the outcome distribution differs between arms. Randomization in the underlying cohort does not undo selection that happens after outcomes are observed.

Recovering Source Conditional Means

When the outcome is numerical, or when a binary outcome is summarized by its mean risk, a normalized inverse-inclusion-weighted mean recovers the source conditional mean. Within a treatment and covariate stratum, the formula is

$$ m_a^{\text{source}}(x) = \frac{ E\left[Y/\pi(a,Y,x)\mid A=a,X=x,R=1\right] }{ E\left[1/\pi(a,Y,x)\mid A=a,X=x,R=1\right] }. $$

The denominator normalizes the weights within the stratum. This is important: the goal is to restore the source distribution of outcomes among people with \(A=a,X=x\), not merely to calculate an average in the selected sample. The formula requires positive inclusion probability for every outcome value that occurs in that source stratum. If some relevant outcome values could never be sampled, reweighting cannot reconstruct their contribution.

Why the correction works: Within a fixed \(a,x\) stratum, the selected outcome density is proportional to \(\pi(a,y,x)f_{\text{source}}(y\mid a,x)\). Weighting that density by \(1/\pi(a,y,x)\) cancels the selection factor. The remaining density is proportional to \(f_{\text{source}}(y\mid a,x)\), and normalization makes it the source conditional outcome distribution.

This argument also shows what information is needed. The analyst needs the relative inclusion probabilities across outcome values within the treatment and covariate strata used for the analysis. Absolute probabilities are useful for reconstructing totals, but the common scale cancels in a normalized conditional mean. In practice, probabilities may come from the sampling design or from data that include the numbers eligible and sampled in relevant strata. If selection also depends on other variables related to the outcome, those variables and the corresponding sampling information may need to be included as well.

From Corrected Source Outcomes to a Target Effect

Correcting the sample distribution is only one part of transport. Let \(Y(1)\) and \(Y(0)\) denote the potential outcomes under treatment and control. Suppose the goal is the target-population average causal effect on the mean-difference scale:

$$ \tau_{\text{target}} = E_{\text{target}}[Y(1)-Y(0)]. $$

A standard identification strategy first estimates treatment-specific outcome means in the source population conditional on pre-treatment covariates, then averages those means over the target covariate distribution. It requires consistency, conditional treatment exchangeability in the source population, treatment positivity, conditional transportability of the potential-outcome means given \(X\), and target support for the covariate values being transported. In words, treatment assignment must be unconfounded given \(X\) in the source, both treatments must be possible in the relevant source strata, and the conditional potential-outcome means must agree between source and target populations at each relevant \(X\).

Under those conditions, and after correcting outcome-dependent inclusion, the target mean under treatment \(a\) is identified by

$$ E_{\text{target}}[Y(a)] = \int m_a^{\text{source}}(x)\,dF_{\text{target}}(x), $$

where \(F_{\text{target}}(x)\) is the target-population distribution of the measured covariates. Subtracting the two treatment-specific means gives the target average causal effect. The target covariate distribution is therefore needed even if the source outcome data have been corrected perfectly: outcome correction does not make the source covariate mix equal to the target mix.

Worked Example: Standardizing Corrected Risks to a Target Population

Suppose a hypothetical randomized source study has outcome-dependent sampling, but the inclusion probabilities are known and the source risks have been corrected within each covariate stratum. Let \(X\) identify two pre-treatment strata. Assume treatment is randomized within the source, the conditional potential-outcome means transport to the target, and both strata are represented in the source and target. The corrected source risks are as follows.

Covariate stratumControl riskTreatment riskTarget proportion
Lower-risk stratum0.100.040.70
Higher-risk stratum0.200.120.30

The target mean risk under treatment is the target-weighted average of the corrected conditional treatment risks:

$$ 0.70(0.04)+0.30(0.12) = 0.028+0.036 = 0.064. $$

The corresponding target mean risk under control is

$$ 0.70(0.10)+0.30(0.20) = 0.070+0.060 = 0.130. $$

Therefore, the target risk difference is \(0.064-0.130=-0.066\). Under the stated assumptions, treatment lowers the target-population risk by 6.6 percentage points on average. The calculation uses the target proportions, not the proportions in the selected study sample; using the latter would generally target a different population.

What Case-Control Data Do and Do Not Provide

Outcome-dependent sampling is often designed to make the sample contain a chosen number of cases and controls. The resulting proportion of cases is not an estimate of the source-population outcome prevalence. If the sampling fractions among cases and non-cases are known within the relevant strata, the source risks can be recovered by correcting for those fractions. If the fractions are unknown, the sample’s case-control composition does not by itself identify absolute source risks.

Worked Example: Why the Case Fraction Is Not a Population Risk

Suppose a hypothetical case-control sample contains 50 cases and 50 controls. Among cases, 30 are exposed and 20 are unexposed; among controls, 20 are exposed and 30 are unexposed. The sample odds ratio comparing exposure and outcome is

$$ \frac{30/20}{20/30} = \frac{30(30)}{20(20)} = 2.25. $$

The sample has a 50% case fraction, but that does not establish a 50% source-population risk. For instance, if the source prevalence were 10%, a design could sample cases at probability 0.90 and controls at probability 0.10, producing nine times as high an inclusion probability for cases. A different source prevalence of 50% could be paired with equal case and control inclusion probabilities. Both designs can yield a sample with equal numbers of cases and controls, so the sample case fraction alone cannot distinguish these source prevalences.

Under outcome-only sampling, the case-control odds ratio can retain the source odds ratio when selection within outcome groups does not additionally depend on exposure. That fact does not supply the source prevalence or identify absolute risks. For a causal risk difference or risk ratio, the outcome-specific sampling fractions, or equivalent source information, are needed in addition to assumptions that identify the causal contrast.

Common Mistakes and Practical Checks

  • Treating the selected sample risk as a population risk. If cases were recruited more often, the observed case fraction reflects both the source prevalence and the sampling design. Use the inclusion probabilities or sampling fractions to correct it.
  • Assuming randomization protects against outcome-based selection. Randomization addresses confounding of treatment in the randomized population. It does not make the selected sample’s outcome distribution representative when inclusion depends on observed outcomes.
  • Correcting outcomes but ignoring target composition. Corrected source conditional means still need to be averaged over the target covariate distribution for a target effect.
  • Using weights without checking support. If a relevant treatment-covariate-outcome stratum has zero inclusion probability, the observed data contain no information from which weighting can recover that stratum’s outcomes.
  • Calling a weighted estimate causal without stating the assumptions. Sampling correction addresses selection distortion; it does not by itself establish treatment exchangeability or transportability of conditional effects.
Key takeaway: Outcome-dependent inclusion changes the observed outcome distribution. To recover and transport a causal effect, identify the relevant inclusion probabilities or sampling fractions, correct treatment-specific outcome information, and standardize it to the target covariate distribution under appropriate causal and transport assumptions.

Check Your Understanding

Answer each question using the distinction between the selected sample, the source population, and the target population.

  1. Why can the observed case fraction in a case-control sample fail to estimate source-population prevalence?
  2. In the inverse-inclusion-weighted conditional mean, what part of the sample distribution is corrected by dividing by \(\pi(a,Y,x)\)?
  3. What sampling information is needed to recover absolute risks when inclusion depends on the observed binary outcome?
  4. Why does randomization in the source population not, by itself, correct outcome-dependent selection?
  5. After correcting the source conditional outcome means, what additional population information is needed to estimate a target average effect?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats