Why Effective Sample Size Matters
A prior distribution can affect a clinical trial analysis as though it contributes information before the trial begins. For a binomial event probability, a Beta prior offers a particularly clear way to quantify that contribution: its shape parameters determine how much prior weight is combined with the observed event and non-event counts. The resulting effective sample size gives a useful scale for judging whether the prior is weak relative to the trial, comparable to it, or much stronger.
This quantity is not the number of people who were literally enrolled in an earlier study. It is a conjugate-model measure of prior weight. Its interpretation depends on the likelihood and assumptions: the current observations must be modeled as binomial observations of the same underlying event probability. A numerical effective sample size is therefore informative, but it does not by itself establish that the prior is relevant or well calibrated.
Definition and Conjugate Update
Suppose a trial observes \(x\) events among \(n\) participants, with the model \(X\mid p\sim\mathrm{Binomial}(n,p)\). The binomial likelihood is proportional to \(p^x(1-p)^{n-x}\). Multiplying this likelihood by the Beta prior density gives a posterior distribution with updated shape parameters. The posterior mean can then be written as a weighted average of the prior mean and the observed event proportion.
The equality makes the interpretation explicit. The prior mean receives weight \(a+b\), and the observed event proportion receives weight \(n\), with the weights normalized by their sum. This is the reason \(a+b\) is called the update-based effective sample size for this model. The prior’s event and non-event weights are \(a\) and \(b\), respectively, and the data add \(x\) events and \(n-x\) non-events.
A Short Derivation
The Beta density is proportional to \(p^{a-1}(1-p)^{b-1}\). Multiplying it by the binomial likelihood gives a kernel proportional to \(p^{a+x-1}(1-p)^{b+n-x-1}\), which is the kernel of the stated Beta posterior. Its mean is the first shape parameter divided by the sum of the two shape parameters. To obtain the weighted-average form, write \(a=(a+b)\{a/(a+b)\}\) and \(x=n(x/n)\), then divide their sum by \(a+b+n\). Thus the weights in the posterior mean follow directly from the conjugate update.
There is a convention worth stating precisely. The Beta density has exponents \(a-1\) and \(b-1\), but the conjugate-binomial update adds \(x\) and \(n-x\) to the shape parameters. In this tutorial, effective sample size means the sum \(a+b\) under that update convention. It is not defined by simply adding the exponents in the prior density.
Worked Example: Interpreting a Prior for an Adverse Event
Suppose, hypothetically, that the adverse-event probability in a trial arm has prior distribution \(\mathrm{Beta}(2,8)\). Calculate its effective sample size and prior mean. Then suppose 12 adverse events occur among 40 participants.
The prior effective sample size is \(2+8=10\) participants’ worth of binomial weight under the conjugate update convention. Its prior mean is \(2/(2+8)=0.2000\), or 20.00%. The values \(a=2\) and \(b=8\) divide the prior weight into event and non-event components.
The hypothetical trial data have \(x=12\), \(n-x=28\), and observed event proportion \(12/40=0.3000\). The posterior is \(\mathrm{Beta}(2+12,8+28)=\mathrm{Beta}(14,36)\). Its mean is \(14/(14+36)=0.2800\), or 28.00%.
The weighted-average calculation gives the same result: \(\{10(0.2000)+40(0.3000)\}/(10+40)=14/50=0.2800\). The prior has one quarter of the total weight in this mean calculation, since \(10/(10+40)=0.2000\); the observed trial proportion has the remaining \(40/50=0.8000\). This does not say that 10 actual participants supplied the prior information.
Prior Weight and the Observed Trial
The effective sample size is most useful when considered alongside the current sample size and the prior mean. A prior with effective sample size 10 is a modest contribution beside a trial of 200 participants, but it is a substantial contribution beside a trial of 12. Neither comparison tells us whether the prior mean is appropriate. A concentrated prior centered far from the trial data can materially pull the posterior estimate even when its effective sample size is not large.
The sum \(a+b\) also provides a simple measure of how the prior’s weight changes when its shape parameters are scaled together. For any positive constant \(c\), the prior \(\mathrm{Beta}(ca,cb)\) has the same mean as \(\mathrm{Beta}(a,b)\), because \(ca/(ca+cb)=a/(a+b)\). Its effective sample size is \(c(a+b)\). Thus, scaling both shapes by a positive constant other than 1 preserves the mean but changes the effective sample size; scaling by 1 leaves the prior unchanged.
Worked Example: Same Prior Mean, Different Weight
Consider the hypothetical priors \(\mathrm{Beta}(2,8)\) and \(\mathrm{Beta}(4,16)\) for the probability of a laboratory-defined response. Compare their means and effective sample sizes. Suppose a small trial observes 3 responses among 10 participants.
For \(\mathrm{Beta}(2,8)\), the mean is \(2/10=0.2000\) and the effective sample size is 10. For \(\mathrm{Beta}(4,16)\), the mean is \(4/20=0.2000\) and the effective sample size is 20. The second prior has the same mean but twice the update-based weight.
With the first prior, the posterior is \(\mathrm{Beta}(2+3,8+7)=\mathrm{Beta}(5,15)\), with mean \(5/20=0.2500\). With the second, it is \(\mathrm{Beta}(4+3,16+7)=\mathrm{Beta}(7,23)\), with mean \(7/30=0.2333\). The observed response proportion is \(3/10=0.3000\). The more heavily weighted prior pulls the posterior mean closer to its own mean of 0.2000.
The arithmetic also checks the scale effect: multiplying both original shape parameters by 2 leaves the mean at 0.2000 and multiplies the effective sample size from 10 to 20. Had the multiplier been 1, neither the parameters nor the effective sample size would have changed.
When the Interpretation Is Appropriate
The update-based interpretation relies on the binomial model being a reasonable description of the data. In particular, conditional on \(p\), the participants’ binary outcomes are modeled as independent with the same event probability. The endpoint and follow-up definition must be consistent for the observations being combined, and the prior must concern that same probability in a sufficiently comparable population and setting. If event probabilities vary substantially among participants or centers, a single shared-\(p\) binomial model may not adequately represent the data.
For the simple update, \(n\) is the number of participants with outcomes contributing to the binomial likelihood. Missing outcomes, unequal follow-up, clustering, or a model that adjusts for covariates can require a different likelihood. In those settings, \(a+b\) remains a property of the chosen Beta prior, but it may not describe the prior’s practical influence on the resulting analysis as well as it does in the simple conjugate model.
The prior also needs to be judged substantively, not just numerically. A prior effective sample size of 20 does not indicate whether the underlying evidence is reliable, whether it is transportable to the current trial, or whether the prior mean is plausible. Those questions require examining the source and relevance of the prior information.
Worked Example: Checking the Model Before Comparing Weights
Imagine a hypothetical study of a binary safety event in two hospitals. An analyst proposes one \(\mathrm{Beta}(3,17)\) prior and plans to combine the outcomes as a single binomial sample. The prior has effective sample size \(3+17=20\) and mean \(3/20=0.1500\). Before interpreting that weight, check whether the proposed model and prior refer to the same quantity.
If every participant has the same event definition and follow-up, and the analysis reasonably treats their outcomes as independent observations with a common event probability \(p\), the conjugate update can be used. For example, 6 events among 30 participants would yield \(\mathrm{Beta}(3+6,17+24)=\mathrm{Beta}(9,41)\), with posterior mean \(9/50=0.1800\). The prior’s weight of 20 can then be compared with the 30 observations.
If the hospitals have materially different event risks, however, a single common \(p\) may not be a defensible description. The numerical sum \(20\) has not changed, but treating it as directly comparable with 30 participants may conceal model mismatch. The effective sample size is meaningful only in relation to the likelihood and the target event probability to which the prior applies.
Diagnostics, Alternatives, and Limitations
Several checks help prevent an effective sample size from being overinterpreted. Confirm that both Beta shape parameters are positive, that the data are binary counts for the specified endpoint, and that the event and non-event counts sum to \(n\). Then verify the update by checking that the posterior shapes are \(a+x\) and \(b+n-x\), and that the posterior mean equals the weighted-average expression. A mismatch often signals a reversed event coding, an incorrect denominator, or confusion about which population the prior describes.
This effective sample size is not the only way to quantify prior information. In the conjugate binomial setting, \(a+b\) is transparent because it is exactly the total prior weight in the posterior-mean update. Other approaches may define an effective sample size by matching prior or posterior variance to a reference sample, or may assess information numerically for a more complex model. These measures answer related but not identical questions, so their values need not agree.
| Approach | What it measures | When it is useful |
|---|---|---|
| Conjugate update-based \(a+b\) | Total Beta shape weight added in the binomial update | Beta prior with a binomial likelihood and a shared event probability |
| Variance- or information-matched effective size | A sample size producing a comparable variance or information under a specified reference model | Comparisons where the simple conjugate update is unavailable or not the chosen definition |
| Actual historical sample size | Number of participants enrolled in a source study | Describing study conduct; it is not automatically the effective sample size of a borrowed prior |
The distinction matters because a historical study’s enrolled sample size and the prior’s \(a+b\) need not be equal. The prior may reflect discounting, selection, or a judgment about relevance; conversely, an apparently large prior weight may be unjustified if the historical and current settings differ. Effective sample size is a summary of the prior under a specified convention, not a substitute for assessing prior-data compatibility or clinical relevance.
Common Pitfall
Correction: Under the conjugate-binomial convention, its effective sample size is 10, and its shapes contribute event and non-event weights of 2 and 8 in the update. These may be fractional or chosen to encode prior information; they do not establish that those exact patients were observed. The prior’s provenance and relevance must be considered separately.
A related mistake is to compare effective sample sizes without checking the model. The statement “the prior contributes 20 and the trial contributes 30” is a useful description only when the Beta prior and binomial observations concern the same event probability under an adequate common-probability model. It is not a universal claim that the prior always has the influence of 20 trial participants.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A hypothetical event probability has prior \(\mathrm{Beta}(3,9)\). What are the prior mean and conjugate-binomial effective sample size?
- With that prior, a trial observes 5 events among 25 participants. Give the posterior distribution and posterior mean.
- What happens to the mean and effective sample size if both shapes in \(\mathrm{Beta}(3,9)\) are multiplied by 3?
- Why is it incorrect to say that every \(\mathrm{Beta}(a,b)\) prior represents exactly \(a\) observed events and \(b\) observed non-events?
- Name one condition that should be checked before comparing \(a+b\) directly with the trial sample size.
Solutions. (1) The mean is \(3/(3+9)=0.2500\), and the effective sample size is \(3+9=12\). (2) The posterior is \(\mathrm{Beta}(3+5,9+20)=\mathrm{Beta}(8,29)\); its mean is \(8/37=0.2162\). (3) The scaled prior is \(\mathrm{Beta}(9,27)\), with mean \(9/36=0.2500\) and effective sample size 36, three times 12. (4) The shapes describe update weights under a model; they do not document actual participants or a particular historical study. (5) For example, check that outcomes are independent conditional on a common event probability and concern the same endpoint and follow-up definition.