Effective Sample Size Under Power Priors

Why Effective Sample Size Needs a Definition

A power-prior discount reduces the contribution of historical control data. Reporting the resulting effective sample size can make the amount of borrowing easier to communicate, but the phrase “effective sample size” does not have one universal meaning. In the fixed-discount binomial model, there is a direct, useful nominal calculation: multiply the historical sample size by the discount. Other definitions match uncertainty or information instead, and their answers need not agree.

The distinction matters in clinical trials. A nominal count summarizes how much historical likelihood has been retained; it does not, by itself, establish that the historical patients are exchangeable with current controls, or that the resulting posterior uncertainty is equivalent to observing that many new patients. The calculation below assumes one historical control study and a fixed discount. It does not combine studies or estimate the discount.

Definition: Suppose a historical control study has $x_0$ events among $n_0$ participants, with $x_0\mid p\sim\operatorname{Binomial}(n_0,p)$. Under a fixed power-prior discount $\delta$, where $0\leq\delta\leq1$, the nominal effective sample size of that historical study is $$ n_{\mathrm{eff,nom}}=\delta n_0. $$ The corresponding nominal effective event and non-event counts are $\delta x_0$ and $\delta(n_0-x_0)$.

Deriving the Nominal Count

The calculation follows directly from the binomial likelihood. Ignoring its factor that does not depend on $p$, the historical likelihood is proportional to $p^{x_0}(1-p)^{n_0-x_0}$. Raising this likelihood to the fixed power $\delta$ multiplies both exponents by $\delta$. Thus the discounted likelihood contributes $\delta x_0$ event counts and $\delta(n_0-x_0)$ non-event counts, for a total contribution of $\delta n_0$.

\begin{aligned} L(p\mid x_0,n_0) &\propto p^{x_0}(1-p)^{n_0-x_0},\\ L(p\mid x_0,n_0)^\delta &\propto p^{\delta x_0}(1-p)^{\delta(n_0-x_0)},\\ \delta x_0+\delta(n_0-x_0)&=\delta n_0. \end{aligned}

For a beta prior $\operatorname{Beta}(a,b)$, with $a>0$ and $b>0$, multiplying by this discounted likelihood gives a beta distribution with parameters $a+\delta x_0$ and $b+\delta(n_0-x_0)$. This conjugate update makes the fractional contributions explicit. The nominal effective size describes the historical contribution, not the total concentration after including the starting prior.

p\mid x_0,n_0\ \propto\ p^{a-1+\delta x_0}(1-p)^{b-1+\delta(n_0-x_0)} \quad\Longrightarrow\quad p\mid x_0,n_0\sim \operatorname{Beta}\!\left(a+\delta x_0,\ b+\delta(n_0-x_0)\right).

This result assumes a binomial sampling model with a common event probability $p$ for the historical observations, a fixed discount, and a beta starting prior independent of the historical data. The binomial model treats outcomes as conditionally independent given $p$, with the same event probability for each participant. In a clinical application, the common-$p$ assumption also represents the substantive decision to treat the historical and target control outcomes as relevant to the same parameter. Discounting can reduce the historical likelihood contribution; it does not repair a poor match between populations, outcome definitions, or follow-up.

Worked Example: A Discounted Historical Event Count

Suppose a hypothetical historical control study records 16 events among 80 participants. A fixed discount of $\delta=0.25$ is used. Calculate the nominal effective sample size and the discounted counts.

The historical sample size is $n_0=80$, so $$ n_{\mathrm{eff,nom}}=\delta n_0=0.25(80)=20. $$ The discounted event count is $0.25(16)=4$, and the discounted non-event count is $0.25(80-16)=0.25(64)=16$. Their sum is $4+16=20$, as required.

For illustration, with a starting prior $\operatorname{Beta}(1,1)$, the resulting posterior is $\operatorname{Beta}(1+4,1+16)=\operatorname{Beta}(5,17)$. The nominal effective size is 20 historical participants; the prior’s two beta parameters are not included in that number. The calculation is conditional on the assumed common-$p$ model and fixed discount.

What the Nominal Number Does—and Does Not—Say

The nominal definition is attractive because it is transparent and does not require choosing a particular event probability at which to compare uncertainty. For a fixed $\delta$, it also has straightforward boundary cases: $\delta=0$ gives zero historical contribution, $\delta=1$ retains the full historical sample size, and intermediate values retain the same fraction of the likelihood exponents. A non-integer result is valid. It is a fractional likelihood contribution, not a claim that a fractional number of people were observed.

The number is nevertheless a summary of a specified model and discount, not a universal measure of information. In the beta-binomial update, the historical data add $\delta n_0$ to the sum of the beta parameters. That algebraic fact explains the term “nominal.” But the amount by which posterior uncertainty changes also depends on the starting prior and the event rate. Moreover, a different model, target parameter, or matching criterion can produce another effective-size summary.

Worked Example: Fractional Contributions and a Posterior Update

Consider a hypothetical study with 42 events among 120 historical controls and a fixed discount $\delta=0.40$. Use a starting prior $\operatorname{Beta}(1,1)$. Find the nominal effective size, the discounted counts, and the posterior mean.

The nominal effective size is $$ n_{\mathrm{eff,nom}}=0.40(120)=48. $$ The fractional event contribution is $0.40(42)=16.8$, and the fractional non-event contribution is $0.40(120-42)=0.40(78)=31.2$. They sum to $16.8+31.2=48$. The posterior is therefore $\operatorname{Beta}(1+16.8,1+31.2)=\operatorname{Beta}(17.8,32.2)$.

For a $\operatorname{Beta}(A,B)$ distribution, the mean is $A/(A+B)$. Here it is $17.8/(17.8+32.2)=17.8/50=0.356$. This is the posterior mean event probability under the stated model and prior. The value 48 is the nominal size of the discounted historical contribution; it is not the posterior beta concentration, which is $17.8+32.2=50$.

The data are hypothetical and the arithmetic does not test whether $\delta=0.40$ is clinically justified. That choice requires a defensible assessment of historical relevance, not merely a convenient effective-size target.

Variance-Matched Effective Size: A Different Question

One alternative asks how many independent binomial observations would have a sample proportion with the same variance as a chosen prior distribution. To make that comparison precise, let the prior be $\operatorname{Beta}(A,B)$, with mean $m=A/(A+B)$ and variance $AB/[(A+B)^2(A+B+1)]$. Define a variance-matched effective size at reference probability $p_*$ by equating that prior variance to the variance $p_*(1-p_*)/n$ of a binomial sample proportion: $$ n_{\mathrm{eff,var}}(p_*)= \frac{p_*(1-p_*)}{\operatorname{Var}(p)}. $$ This is a chosen variance-matching convention, not the nominal power-prior definition. The reference probability must be stated. If it is set to the beta prior mean, $p_*=m$, the expression simplifies to $A+B+1$. That $+1$ arises from the variance of a beta distribution; it differs from the beta concentration $A+B$, which is often used as a pseudo-count or concentration summary.

Worked Example: The Reference Probability Changes a Variance Match

Suppose a hypothetical beta distribution for the control event probability is $\operatorname{Beta}(5,15)$. Its mean is $5/20=0.25$, and its variance is $$ \operatorname{Var}(p)=\frac{5(15)}{(5+15)^2(5+15+1)} =\frac{75}{400(21)} =0.008929. $$ Calculate the variance-matched size at two reference probabilities.

At $p_*=0.25$, the binomial sample-proportion variance per observation is $0.25(0.75)=0.1875$. Therefore $$ n_{\mathrm{eff,var}}(0.25)=\frac{0.1875}{0.008929}=21.00. $$ This agrees with $A+B+1=5+15+1=21$, because the reference probability equals the beta mean.

At $p_*=0.50$, the numerator becomes $0.50(0.50)=0.25$, so $$ n_{\mathrm{eff,var}}(0.50)=\frac{0.25}{0.008929}=28.00. $$ The same beta distribution therefore corresponds to 21 or 28 observations under this variance-matching rule, depending on the reference event probability. These numbers answer a different question from a nominal power-prior size. In particular, the example’s beta distribution alone does not identify how many historical participants were discounted; that requires the historical sample size and $\delta$.

Model, Parameter, and Information Dependence

A second common approach matches information, often through likelihood curvature or Fisher information. Under the same binomial model and the same probability parameter $p$, the expected Fisher information in $n_0$ observations is $n_0/[p(1-p)]$. Raising the likelihood to a fixed power $\delta$ multiplies this information by $\delta$. Matching it to the information from $m$ observations under that same model gives $m=\delta n_0$. Thus this particular information match reproduces the nominal size.

That agreement is conditional on holding the model and parameterization fixed. An information match made under another outcome model, for another estimand, or against a different reference information can give another answer. Variance matching can also depend on the selected reference probability, as the example shows. In clinical work, the phrase “effective sample size” should therefore be accompanied by its definition, model, parameter or reference value, and any prior assumptions needed for the calculation.

The nominal count has a separate limitation: two historical studies with the same $n_0$ and $\delta$ have the same nominal size even if their observed event rates differ. Their beta updates, and hence their posterior locations and uncertainty, can still differ. A nominal count is not a similarity measure between historical and current controls, a guarantee of calibrated inference, or a replacement for examining the underlying model.

Summary	Question it answers	What must be specified
Nominal size $\delta n_0$	How much of the historical binomial likelihood is retained?	Historical sample size and fixed discount
Variance-matched size	How many binomial observations give a chosen variance?	Prior distribution and reference probability
Information-matched size	How many observations give matching likelihood information?	Model, parameter, and information comparison

Diagnostics, Failure Modes, and Interpretation

Before reporting a nominal size, check that the historical observations are represented by the stated binomial data, that $0\leq\delta\leq1$, and that the discount is fixed for the calculation. Confirm that the fractional event and non-event contributions sum to $\delta n_0$. If they do not, there is an arithmetic or coding error. Also check that the analysis uses the same event definition and an appropriate follow-up window; counting outcomes that are not comparable undermines the common-probability interpretation.

An effective-size report should not conceal the event rate or the amount of discounting. Reporting $n_0$, $x_0$, $\delta$, and $\delta n_0$ makes the calculation auditable. If a variance-matched quantity is reported, state the reference probability and the variance convention. If an information-based quantity is reported, state the model and parameter used for the match. When different reasonable definitions produce very different values, that disagreement is informative: the effective-size summary is sensitive to the question being asked.

Common mistake: “The discount is 0.25, so the posterior has only 25% as much uncertainty,” or “a nominal effective size of 20 means the prior contains exactly the same information as 20 current patients.” Neither conclusion follows. The discount multiplies the historical likelihood contribution; posterior uncertainty also depends on the data, starting prior, model, and parameter. A variance- or information-equivalence claim requires its own explicit matching definition.

The main alternatives are not competing calculations of one uniquely defined quantity. The nominal count is usually the clearest summary when the goal is to communicate the fraction of a single binomial historical likelihood retained. Variance matching is useful when the goal is to compare uncertainty, provided the reference probability is clinically relevant and stated. Information matching is useful for local comparisons of likelihood precision, but its interpretation is tied to the model and parameter. None of these summaries demonstrates that borrowing is safe; each is conditional on its assumptions.

Check Your Understanding

Answer each question, then compare with the solutions below.

A hypothetical historical study has 90 controls and a fixed discount of 0.30. What is its nominal effective size?
In a study with 25 events among 100 historical controls and $\delta=0.20$, what are the nominal effective event and non-event counts?
For a $\operatorname{Beta}(3,7)$ distribution, calculate the variance-matched size at $p_*=0.30$. Use the variance-matching definition in this tutorial.
Why can a variance-matched effective size differ from the nominal power-prior size?
Does a nominal effective size of 40 establish that the historical controls are exchangeable with current controls? Explain.

Solutions. (1) $n_{\mathrm{eff,nom}}=0.30(90)=27$. (2) The event contribution is $0.20(25)=5$; the non-event contribution is $0.20(75)=15$. They sum to the nominal size $0.20(100)=20$. (3) For $\operatorname{Beta}(3,7)$, the variance is $3(7)/[10^2(11)]=21/1100=0.01909$. Therefore $n_{\mathrm{eff,var}}(0.30)=0.30(0.70)/0.01909=11.00$ (to three significant figures). (4) The nominal size counts discounted likelihood contribution, whereas the variance match equates a prior variance to a binomial sampling variance at a specified reference probability; the quantities need not be equal. (5) No. The number summarizes the amount of likelihood retained under the chosen model and discount. Exchangeability is a substantive assumption about the relevance of historical and current controls, not a consequence of the arithmetic.

Summary	Question it answers	What must be specified
Nominal size \(\delta n_0\)	How much of the historical binomial likelihood is retained?	Historical sample size and fixed discount
Variance-matched size	How many binomial observations give a chosen variance?	Prior distribution and reference probability
Information-matched size	How many observations give matching likelihood information?	Model, parameter, and information comparison