Power and Minimum Detectable Effects for Price Tests

Why Power Matters in a Price Test

A price test can be randomized correctly and still be uninformative if it includes too few customers. A small sample may fail to distinguish a genuine change in retention or contribution from ordinary sampling variation. Power calculations help answer two planning questions: How many observations are needed to detect a specified effect? and What is the smallest effect a proposed test can reliably detect?

For a hypothetical renewal-price test, the outcome might be whether an eligible customer renews, or the contribution earned per eligible customer over a defined period. The sample-size calculation depends on the outcome’s variability, the test’s significance level, the desired power, the effect considered meaningful, the allocation between groups, and whether observations are independent. It does not establish that the proposed price change is appropriate or profitable; it quantifies the test’s ability to detect a difference under stated assumptions.

Definition: The significance level $\alpha$ is the probability, under the null hypothesis of no effect, of rejecting that hypothesis using the specified test. Power is the probability of rejecting the null when a stated alternative effect is true; $1-\text{power}$ is the Type II error probability $\beta$. The minimum detectable effect (MDE) is the smallest effect that the design is planned to detect at a chosen significance level and power.

A conventional planning choice is a two-sided significance level of 5% and power of 80%, corresponding approximately to $z_{1-\alpha/2}=1.96$ and $z_{1-\beta}=0.84$. These are choices, not universal requirements. A one-sided test, a different significance level, or a higher power changes the required sample size. The direction and test must be selected before looking at the results; choosing them afterward undermines the stated error rate.

Two-Group Planning Formulas

Suppose a randomized test compares a control group with a treatment group. Let $n_C$ and $n_T$ be the numbers of independent observations assigned to control and treatment. For a binary retention outcome, let $p_C$ and $p_T$ be the respective renewal probabilities. The estimated difference in renewal rates has approximate standard error

\operatorname{SE}(\hat p_T-\hat p_C) = \sqrt{\frac{p_T(1-p_T)}{n_T}+\frac{p_C(1-p_C)}{n_C}}.

For planning, when the alternative rate is not known, a common approximation uses the baseline rate $p_0$ for the variance in both groups. If the target absolute difference is $\delta$, the approximate sample-size condition for a two-sided normal test is

\delta \approx \left(z_{1-\alpha/2}+z_{1-\beta}\right) \sqrt{p_0(1-p_0)\left(\frac{1}{n_T}+\frac{1}{n_C}\right)}.

This is a planning approximation, not an exact guarantee of power. It is most useful when the normal approximation is reasonable and the rates are not so close to 0 or 1, or the counts so small, that a discrete method is needed. For a final design in those cases, or where the planned analysis uses a particular exact or score test, calculate power using that test’s method.

For a continuous outcome with a common standard deviation $\sigma$, such as contribution per eligible customer, the corresponding approximation is

\operatorname{SE}(\bar Y_T-\bar Y_C) = \sigma\sqrt{\frac{1}{n_T}+\frac{1}{n_C}}, \qquad \delta \approx \left(z_{1-\alpha/2}+z_{1-\beta}\right) \sigma\sqrt{\frac{1}{n_T}+\frac{1}{n_C}}.

Here $\delta$ is in the units of the outcome, such as dollars per eligible customer. If contribution has a highly skewed distribution, heavy tails, or many extreme values, a normal approximation may be unreliable for the intended analysis. Planning should then reflect the actual analysis method and plausible outcome distribution, often through simulation. Do not silently substitute the standard deviation of a different population or a different definition of contribution.

Why the Formulas Work

The formulas follow from the approximate distribution of a difference between two independent group means. For binary outcomes, each group’s sample proportion has variance approximately $p(1-p)/n$; for a continuous outcome, a sample mean has variance $\sigma^2/n$. Independence between randomized groups means the variance of their difference is the sum of the two variances. Taking the square root gives the standard errors above.

Under the null, a two-sided normal test rejects when the standardized difference exceeds $z_{1-\alpha/2}$ in absolute value. If the true difference is $\delta$, its expected standardized value is approximately $\delta/\operatorname{SE}$. To achieve the target power, this signal must be about $z_{1-\alpha/2}+z_{1-\beta}$: one amount reaches the null rejection boundary, and the additional amount places the alternative distribution far enough beyond that boundary to obtain power $1-\beta$. Thus, approximately,

\frac{\delta}{\operatorname{SE}} \approx z_{1-\alpha/2}+z_{1-\beta}.

This justification also explains the main design trade-off: a smaller target effect requires a smaller standard error, and therefore more observations. It also shows why a test with adequate total enrollment can still be underpowered if its allocation or outcome variability differs from the assumptions used to plan it.

Equal and Unequal Allocation

With equal allocation, $n_T=n_C=n$, the binary-outcome approximation simplifies to

n \approx \frac{\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2 \,2p_0(1-p_0)}{\delta^2}.

The same structure applies to a continuous outcome after replacing $p_0(1-p_0)$ with $\sigma^2$. For unequal allocation, define $r=n_T/n_C$. Under the baseline-rate approximation, the control requirement is

n_C \approx \frac{\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2 p_0(1-p_0)(1+1/r)}{\delta^2}, \qquad n_T=r n_C.

Unequal allocation generally increases the total sample size needed for a given precision compared with equal allocation, although operational constraints or treatment costs may justify it. Calculate the group requirements using the stated ratio, then round to whole observations. If the actual design must preserve at least a specified allocation ratio after rounding, set the second group’s count accordingly and check the resulting standard error.

Worked Example: Retention Sample Size With Equal Allocation

A hypothetical insurer plans to compare a renewal-price change with the current price. The control renewal rate is expected to be 80%. The target is to detect an absolute change of 5 percentage points, using a two-sided 5% significance level and 80% power. Find the required number of eligible customers per group using the baseline-rate approximation. No discounting or mortality assumptions are used; the outcome is a binary renewal indicator for each eligible customer.

Method. Use the equal-allocation formula with $p_0=0.80$, $\delta=0.05$, $z_{1-\alpha/2}=1.96$, and $z_{1-\beta}=0.84$. The variance factor is $p_0(1-p_0)=0.80(0.20)=0.16$, and the squared critical-value sum is $(1.96+0.84)^2=2.80^2=7.84$.

n \approx \frac{7.84(2)(0.16)}{0.05^2} =\frac{2.5088}{0.0025} =1003.52.

The required count is rounded up to 1,004 customers per group, or 2,008 total. As a check, the planned standard error under the baseline approximation is $\sqrt{0.32/1004}\approx0.01785$; multiplying by 2.80 gives an MDE of approximately $0.04999$, essentially the 5-percentage-point target. The common mistake avoided here is omitting one group’s variance and using $0.16$ instead of $2(0.16)$, which would understate the equal-allocation sample size by about half.

Worked Example: Unequal Allocation for a Retention Test

A hypothetical renewal test expects a baseline retention rate of 70%. The treatment group is planned to have 1.5 times as many customers as control. The goal is to detect an absolute difference of 6 percentage points with two-sided $\alpha=0.05$ and 80% power. Find a rounded design. Use the baseline-rate approximation; no discounting or mortality assumptions are relevant.

Method. Set $r=n_T/n_C=1.5$. The variance factor is $0.70(0.30)=0.21$, and $(1.96+0.84)^2=7.84$. Therefore,

n_C \approx \frac{7.84(0.21)(1+1/1.5)}{0.06^2} = \frac{7.84(0.21)(1.6667)}{0.0036} \approx \frac{2.744}{0.0036} =762.22.

Rounding the control requirement up gives $n_C=763$. To preserve at least the 1.5 treatment-to-control ratio after rounding, take $n_T=\lceil1.5(763)\rceil=1,145$. The resulting ratio is $1,145/763\approx1.5007$. Using the rounded counts, the approximate standard error is

\sqrt{0.21\left(\frac{1}{763}+\frac{1}{1145}\right)} \approx \sqrt{0.00045864} \approx 0.02142.

The implied MDE is $2.80(0.02142)\approx0.05996$, approximately 6 percentage points. The design therefore meets the target to rounding precision. The method avoids treating the two group counts as equal; it also distinguishes rounding each computed requirement from deliberately preserving a minimum ratio after rounding.

Continuous Contribution Outcomes and Clustering

For an unequal allocation ratio $r=n_T/n_C$, the continuous-outcome formula gives

n_C \approx \frac{\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2 \sigma^2(1+1/r)}{\delta^2}, \qquad n_T=r n_C.

The contribution outcome must be defined at the level of the randomized unit and over a specified measurement period. If the question is contribution per eligible customer, a customer who does not renew may have zero contribution; analyzing only customers who renew changes the population being compared and can introduce selection bias. Use a standard deviation that reflects the planned outcome definition, including such zeros where appropriate.

If assignment is by a cluster, such as a household or agency, observations within the same cluster may be correlated. For equal cluster size $m$ and intracluster correlation $\rho$, a common approximation to the design effect is

\operatorname{DE}=1+(m-1)\rho.

The design effect inflates the number of individual observations required: approximately $n_{\text{actual}}=n_{\text{independent}}\operatorname{DE}$. This approximation assumes equal cluster sizes, a common within-cluster correlation, and independent clusters. With unequal cluster sizes, few clusters, or materially different correlations, this simple adjustment may not be adequate; planning should use a method that reflects the actual cluster design. When enrollment must be in whole clusters, convert the inflated customer requirement to a number of clusters and round up.

Worked Example: Contribution MDE With Household Randomization

A hypothetical insurer randomizes households to a renewal-price test. Contribution per eligible customer over the renewal period is assumed approximately normal for planning, with common standard deviation $\sigma=\$240$. The target difference is $\$40$; the test is two-sided with $\alpha=0.05$, 90% power, and twice as many treatment as control customers. Each household has four eligible customers, and the intracluster correlation is assumed to be 0.10. Find a rounded cluster design and check its approximate MDE. No discounting or mortality assumptions are used.

Method. First calculate the independent-customer requirements, using $z_{1-\alpha/2}=1.96$ and $z_{1-\beta}=1.282$. Thus the critical-value sum is $3.242$, whose square is $10.510564$. With $r=2$,

n_C \approx \frac{10.510564(240^2)(1+1/2)}{40^2} = \frac{10.510564(57{,}600)(1.5)}{1{,}600} =567.57.

Round up to 568 independent-equivalent control customers; the corresponding treatment requirement at a 2:1 ratio is 1,136. The cluster design effect is $1+(4-1)(0.10)=1.30$. Inflating gives $568(1.30)=738.4$ control customers and $1,136(1.30)=1,476.8$ treatment customers. With four customers per household, round up to 185 control households and 370 treatment households: 740 and 1,480 customers, respectively.

For a check, the effective independent-equivalent counts are $740/1.30=569.23$ and $1,480/1.30=1,138.46$. The approximate standard error is

240\sqrt{\frac{1}{569.23}+\frac{1}{1{,}138.46}} \approx \$12.32.

The MDE is $3.242(\$12.32)\approx\$39.94$, just below the $\$40$ target. The rounded design therefore meets the target approximately. The common mistake avoided is treating all 2,220 customers as independent: household assignment means the design effect must be reflected in the calculation.

Calculating an MDE From a Fixed Sample

If the available sample is fixed, rearrange the same standard-error relationship. For a binary outcome with equal group size $n$, using baseline rate $p_0$, the approximate MDE is

\operatorname{MDE}\approx \left(z_{1-\alpha/2}+z_{1-\beta}\right) \sqrt{\frac{2p_0(1-p_0)}{n}}.

For a continuous outcome with equal group size, it is $(z_{1-\alpha/2}+z_{1-\beta})\sigma\sqrt{2/n}$. These values are absolute effects: a retention MDE of 0.04 is a four-percentage-point difference, not a 4% relative increase. For unequal allocation, retain the full $1/n_T+1/n_C$ expression. For clustered assignment, use the effective sample size or the corresponding design-effect adjustment, consistent with the design assumptions.

An MDE is not a forecast of the effect, a threshold below which effects do not exist, or a guarantee that a detected effect is commercially meaningful. A test can have high power for a statistically detectable effect that is too small to matter operationally. Conversely, a commercially meaningful effect may be difficult to detect if the outcome is noisy or the number of independent units is limited.

Common Planning Errors

Confusing percentage points with relative percent. Moving from 70% to 74% is a 4-percentage-point increase, but a relative increase of about 5.7%. Use the effect scale stated in the design.
Using the wrong allocation formula. For unequal groups, use both group counts in the variance. Do not calculate as though the groups were equal and then assign a different number to one arm without checking the resulting standard error.
Rounding down or inconsistently. Round required sample sizes up. If the ratio must be maintained after rounding, calculate the second count from the rounded first count and verify the actual ratio and MDE.
Ignoring clustering. The number of customers is not the number of independent assignments when a household, agency, or other group is randomized. Account for within-cluster correlation and plan enough clusters.
Planning around a convenient outcome definition. Changing from contribution per eligible customer to contribution among renewers changes both the target and the variance, and may change who is represented.
Assuming the approximation is exact. Small samples, rare outcomes, severe skew, and few clusters can invalidate normal and design-effect approximations. Match the power calculation to the intended analysis.

Key takeaway: State the outcome, target effect, significance level, power, allocation ratio, variance assumptions, and randomization unit before calculating a sample size or MDE. Then round to an implementable design and recompute the achieved standard error using the actual planned counts. This material is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Check Your Understanding

Use two-sided $\alpha=0.05$ and the normal approximations described above unless a question specifies otherwise.

A hypothetical retention test has baseline rate 75%, equal allocation, and 900 customers per group. Using 80% power and the baseline-rate approximation, find the MDE in percentage points. (Answer: approximately 5.72 percentage points.)
A hypothetical retention test has baseline rate 60%, equal allocation, and a 4-percentage-point target effect. Find the required number per group for 80% power, rounded up. (Answer: 2,352 customers per group.)
A hypothetical retention test has baseline rate 90% and seeks a 3-percentage-point difference with 80% power. Treatment allocation is twice control allocation. Find the approximate control and treatment requirements, rounded up while preserving the 2:1 ratio. (Answer: 1,176 control and 2,352 treatment customers.)
A hypothetical contribution test has common standard deviation $\$500$, equal allocation, and 400 customers per group. Find the MDE for 90% power. (Answer: approximately $\$114.62$ per eligible customer.)
Households are randomized, with five eligible customers per household and intracluster correlation 0.08. If 600 customers are observed, calculate the design effect and approximate independent-equivalent sample size. (Answer: design effect $1.32$; effective sample size approximately 454.5 independent observations.)

What You'll Learn