Rare-Event Sampling and Demand Probability Correction

Why Sampling Design Matters for Demand Probabilities

Purchase data are sometimes collected with a design that deliberately includes more purchasers than would occur in the market. For example, an analyst might sample a fixed number of customers who bought a hypothetical specialty insurance product and a fixed number who received a quote but did not buy. This can be useful when purchases are rare: it supplies enough positive outcomes to study their relationship with price and other predictors. But the purchase fraction in that sample is a design choice, not an estimate of the market purchase probability.

A logistic regression fitted to such data can still contain useful information about how purchase odds vary with predictors. Under specific conditions, outcome-dependent sampling shifts the intercept but leaves the slope coefficients unchanged. The fitted probabilities, however, need correction before they represent the source population. That distinction matters for predicted demand, expected take-up, and any calculation that uses a probability level rather than only a relative change in odds.

Definition: In outcome-dependent sampling, inclusion in the analysis sample depends on the binary outcome, such as purchase versus nonpurchase. In case-control sampling, the analyst selects cases (purchasers) and controls (nonpurchasers) at outcome-specific rates. The source population is the population whose purchase probabilities are to be estimated; the sample prevalence is the purchase fraction in the selected sample.

Prior-Probability Correction in Logistic Regression

Let $Y=1$ indicate purchase and $Y=0$ indicate nonpurchase. Write $X$ for the predictors, which may include price and customer characteristics. Suppose that in the source population the purchase probability follows a correctly specified logistic model:

\operatorname{logit} P(Y=1\mid X) =\log\!\left(\frac{P(Y=1\mid X)}{1-P(Y=1\mid X)}\right) =\alpha+\beta^\top X.

Assume each purchaser is included with probability $r_1$ and each nonpurchaser with probability $r_0$, and that these inclusion probabilities are constant across predictor values within each outcome group. Then the sampled odds at any fixed $X$ equal the source odds multiplied by $r_1/r_0$. Consequently:

\operatorname{logit} P_s(Y=1\mid X) =\alpha+\beta^\top X+\log\!\left(\frac{r_1}{r_0}\right),

where $P_s$ denotes probability under the sample distribution. The sampling design adds a constant to the log odds. It does not change $\beta$ under these assumptions. To recover source-population probabilities, subtract that constant from the sampled-model linear predictor.

Often the inclusion rates are not recorded individually, but the source purchase prevalence $\pi=P(Y=1)$ and sample purchase prevalence $\pi_s=P_s(Y=1)$ are known. Bayes’ rule gives the same correction in terms of these prevalences:

\operatorname{logit} P(Y=1\mid X) = \operatorname{logit} P_s(Y=1\mid X) + \operatorname{logit}(\pi) - \operatorname{logit}(\pi_s).

This prevalence form applies when the sampling mechanism depends on outcome alone, so the distribution of predictors within purchasers and within nonpurchasers is unchanged by sampling. The source prevalence must come from a credible external source or known population count; it cannot be replaced by the deliberately altered sample fraction.

Why the Slope Stays the Same

The result follows directly from the odds. At a fixed predictor value $X=x$, let the source conditional purchase probability be $p(x)$. The source odds are $p(x)/[1-p(x)]$. Under outcome-specific inclusion probabilities, the relative representation of purchasers versus nonpurchasers in the sample is multiplied by $r_1/r_0$. Thus:

\frac{P_s(Y=1\mid x)}{P_s(Y=0\mid x)} = \frac{r_1}{r_0} \frac{P(Y=1\mid x)}{P(Y=0\mid x)}.

Taking logarithms converts multiplication into addition:

\operatorname{logit} P_s(Y=1\mid x) = \log\!\left(\frac{r_1}{r_0}\right)+\alpha+\beta^\top x.

The added term is constant in $x$, so the coefficient on every predictor is unchanged in the population model. This is a result about the sampling mechanism and model specification, not a guarantee that every finite-sample estimate will match exactly. Sampling variation remains, and a misspecified logistic relationship may not be repaired by an intercept adjustment.

Key result: If inclusion probabilities differ by outcome but are constant within each outcome group, a correctly specified logistic model has the same source and sampled slope coefficients. The source intercept is the sampled intercept plus $\operatorname{logit}(\pi)-\operatorname{logit}(\pi_s)$. If inclusion also depends on predictors within outcome groups, a constant intercept correction generally does not suffice.

Applying the Correction

Identify the target population.
Specify the source purchase prevalence, rather than treating the sample’s purchase fraction as a market rate.

Check the sampling assumption.
Confirm that selection depends on purchase outcome only, or that the outcome-specific inclusion-rate ratio is constant over the predictors.

Calculate the log-odds adjustment.
Use $\operatorname{logit}(\pi)-\operatorname{logit}(\pi_s)$, or subtract $\log(r_1/r_0)$ when inclusion rates are given.

Correct predictions before using them.
Add the adjustment to the sampled-model linear predictor, then apply the inverse-logit function. Keep the fitted slopes, subject to the stated assumptions.

Worked Example: Correcting a Case-Control Demand Model

A hypothetical insurer’s source market has a purchase prevalence of $0.02$. An analyst constructs a case-control sample with purchase prevalence $0.20$ and fits the sampled logistic model $\operatorname{logit} P_s(Y=1\mid x)=-1+0.5x$. Find the corrected source probability at $x=2$, and state the corrected model. Use the prior-probability correction.

Method and assumption. The sample is assumed to select purchasers and nonpurchasers at different rates, but not to select differently by $x$ within either outcome group. Therefore, the correction is a constant log-odds shift.

First calculate the source and sample odds of purchase:

\text{Source odds}=\frac{0.02}{0.98}=\frac{1}{49}, \qquad \text{Sample odds}=\frac{0.20}{0.80}=\frac{1}{4}.

The odds multiplier from sample to source is $(1/49)/(1/4)=4/49$. Its logarithm is $\log(4/49)\approx-2.505526$. The corrected model is therefore:

\operatorname{logit} P(Y=1\mid x) =-1+0.5x+\log(4/49) \approx -3.505526+0.5x.

At $x=2$, the sampled linear predictor is $-1+0.5(2)=0$, so the sampled odds are $e^0=1$. Correcting those odds gives $1(4/49)=4/49$. Hence:

P(Y=1\mid x=2) = \frac{4/49}{1+4/49} = \frac{4}{53} \approx 0.0755.

The corrected model has slope $0.5$, unchanged from the sampled model, and the probability is about $0.0755$ to four decimal places. This checks because the corrected linear predictor is $-2.505526$, whose odds are $4/49$. Common mistake avoided: reporting the sampled probability $0.5000$ at $x=2$ as the source-market probability ignores the altered sample prevalence.

Worked Example: Using Known Outcome-Specific Inclusion Rates

In another hypothetical study, purchasers are included with probability $r_1=0.40$ and nonpurchasers with probability $r_0=0.10$, regardless of their predictor values within each outcome. Suppose the source model is $\operatorname{logit}P(Y=1\mid z)=-3+0.7z$. Determine the sampled model and the sampled and source probabilities at $z=2$.

Method. Since $r_1/r_0=0.40/0.10=4$, sampling adds $\log 4\approx1.386294$ to the source logit. Thus the sampled model is:

\operatorname{logit} P_s(Y=1\mid z) =-3+0.7z+\log 4 \approx -1.613706+0.7z.

At $z=2$, the source linear predictor is $-3+0.7(2)=-1.6$. Its odds are $e^{-1.6}\approx0.201897$, giving source probability $0.201897/(1+0.201897)\approx0.1680$. Sampling multiplies those odds by four, so sampled odds are $4(0.201897)\approx0.807586$, and the sampled probability is $0.807586/(1+0.807586)\approx0.4464$.

As a check, subtracting $\log 4$ from the sampled linear predictor at $z=2$ gives $-0.213706-1.386294=-1.600000$, recovering the source logit and probability. Probabilities are rounded to four decimal places. Common mistake avoided: the sampled probability is not the source probability simply because both models have the same slope.

Corrected Probabilities and Price Elasticity

A slope can remain unchanged while a probability-based measure changes. For a logistic purchase model with log price coefficient $\beta_p$, where price enters as $\log(\text{price})$, the point price elasticity of purchase probability is $\beta_p(1-p)$. This expression uses the probability $p$ at the price and customer profile of interest. If $p$ is an uncorrected sampled probability, the resulting elasticity is generally not the source-population elasticity.

Worked Example: Recomputing Elasticity After Prior Correction

A hypothetical case-control sample has purchase prevalence $\pi_s=0.10$, while the source market prevalence is $\pi=0.01$. Its fitted model at the reference price of \$100 has linear predictor $-2$, with log-price coefficient $-1.2$. Find the corrected reference-price probability and point price elasticity.

Method. The intercept adjustment is $\operatorname{logit}(0.01)-\operatorname{logit}(0.10)$. Equivalently, the source-to-sample odds ratio is:

\frac{0.01/0.99}{0.10/0.90}=\frac{1}{11}.

The sampled odds at the reference price are $e^{-2}\approx0.1353353$. The corrected source odds are $e^{-2}/11\approx0.0123032$, so the corrected probability is $0.0123032/(1+0.0123032)\approx0.0122$. More precisely, it is approximately $0.0121537$.

Using the logistic elasticity formula with the corrected probability:

\text{Source elasticity} =(-1.2)(1-0.0121537) \approx -1.1854.

For comparison, the uncorrected sampled probability at that reference price is $1/(1+e^2)\approx0.1192029$, which would produce sampled elasticity $(-1.2)(1-0.1192029)\approx-1.0570$. The source elasticity is approximately $-1.1854$, to four decimal places. The price coefficient is unchanged, but the probability level used in the elasticity is not. Common mistake avoided: carrying the sampled probability into a source-market elasticity calculation.

When an Intercept Correction Is Not Enough

A constant correction is justified only under the sampling conditions used in its derivation. If, among purchasers, low-price customers are sampled more often than high-price customers, the inclusion rate for purchasers depends on price. The sampled log odds then acquire a predictor-dependent term. That term can alter the apparent price slope as well as the intercept.

Similarly, an intercept shift does not fix an incorrect functional form, omitted predictors, or a change in the relationship between predictors and purchase in the source population. It corrects the prior odds under outcome-dependent sampling; it does not establish that the fitted demand model is otherwise adequate. Where inclusion probabilities are known and vary by observation, a source-population analysis may require a method that accounts for those probabilities rather than a single prior correction.

A useful exam check is to ask what is known about the sampling design before using any fitted probability. If the sample was formed by fixed purchaser and nonpurchaser quotas, its raw purchase fraction is not a market forecast. If the objective is only to compare conditional odds responses under the stated model, the slopes may still be informative. If the objective is a probability, expected number of purchases, or elasticity at a market profile, first restore the appropriate source-population prior.

Common exam trap: Applying the sample purchase fraction as though it were the source prevalence, or adjusting slopes when only a constant outcome-dependent sampling ratio is present. The correct procedure is to shift the logistic intercept by the source-minus-sample logit prevalence difference, while retaining slopes only when selection is outcome-dependent alone and the model assumptions hold.

Check Your Understanding

For each problem, use the stated sampling assumptions and report probabilities to four decimal places where applicable.

A source market has purchase prevalence $0.04$, while the case-control sample has prevalence $0.25$. At a particular predictor value, the sampled model gives probability $0.60$. Find the corrected source probability. (Answer: $0.1579$)
Purchasers are sampled at rate $0.30$, nonpurchasers at rate $0.06$, and selection is constant within each outcome group. The sampled model is $\operatorname{logit}P_s(Y=1\mid x)=-1.4+0.4x$. Find the source probability at $x=1$. (Answer: approximately $0.0685$)
A fitted sampled model has log-price coefficient $-0.8$. At a particular price and profile, the sampled odds are $0.20$; the source and sample prevalences are $0.02$ and $0.10$, respectively. Find the corrected source probability and its point price elasticity. (Answer: probability approximately $0.0354$; elasticity approximately $-0.7717$)
An analyst samples purchasers more heavily, and within the purchaser group also oversamples customers quoted at low prices. Is a single intercept correction guaranteed to recover both source probabilities and the source price slope? Explain briefly. (Answer: No; predictor-dependent selection within an outcome group can alter the price slope, so a constant intercept shift is not generally sufficient.)
The sample prevalence is $0.05$, the source prevalence is $0.005$, and the fitted sampled model has intercept $-2.3$. Under outcome-only sampling, find the corrected source intercept, rounded to four decimals. (Answer: approximately $-4.6489$)

What You'll Learn