What Effect Does a Price Experiment Estimate?
A randomized price test can estimate how assignment to a particular price condition changes an outcome, on average, for the eligible population represented by the experiment. The central comparison is between groups as they were assigned, not between customers selected after assignment according to the price they ultimately received. This distinction preserves the benefit of randomization when implementation is imperfect.
Let \(Z_i=1\) indicate that unit \(i\) was assigned to the test price and \(Z_i=0\) indicate assignment to the comparison price. Let \(Y_i\) be the outcome measured for that unit over the specified follow-up period. The outcome might be renewal, purchase, or another pre-defined measure. The price experiment estimates an effect on that outcome; it does not, by itself, establish that the price change is profitable or appropriate for rollout.
For a binary outcome, such as whether a customer renews, the group mean is the observed proportion with that outcome. The ITT estimate is therefore a difference in proportions. For a continuous outcome, it is a difference in sample means. When assignment is randomized, the assigned groups are comparable in expectation, so their outcome difference estimates the causal effect of assignment. This interpretation requires that outcomes are measured consistently and that one unit's assignment does not change another unit's outcome. If randomization was by agency or another cluster, uncertainty calculations must also respect that assignment structure.
Here, \(\bar{Y}_{Z=1}\) and \(\bar{Y}_{Z=0}\) are the observed mean outcomes in the test-assigned and comparison-assigned groups. For a binary outcome, a common large-sample standard error is
where \(\hat{p}_z\) is the observed outcome proportion and \(n_z\) is the number assigned to group \(z\). This formula assumes independent randomized units and a sufficiently large sample for a normal approximation. It is not appropriate to treat individual policies as independent when assignment occurred by cluster; in that case, use an uncertainty method that accounts for clustering.
Worked Example: Estimating an ITT Effect on Renewal
In a hypothetical individual-level randomized test, 400 eligible customers are assigned a renewal quote with an 8% increase and 400 are assigned the existing quote. Within the stated follow-up period, 276 test-assigned customers and 300 comparison-assigned customers renew. Estimate the ITT effect on renewal probability and calculate a 95% normal-approximation confidence interval.
Method. Compare renewal proportions by randomized assignment. This estimates the effect of being assigned the test quote, whether or not every customer saw or accepted the quote.
The test-assigned renewal proportion is \(276/400=0.6900\). The comparison proportion is \(300/400=0.7500\). Thus
The estimated assignment effect is a decrease of 6.00 percentage points in renewal. The standard error is
The 95% interval is the estimate plus or minus \(1.96\) standard errors: \( -0.0600\pm1.96(0.0317)=-0.0600\pm0.0621\), giving \([-0.1221,\,0.0021]\). The interval includes zero, so these data do not distinguish the estimated decrease from no assignment effect at the 5% two-sided level using this approximation. This does not prove that the effect is zero.
Check. The observed renewal rate is lower in the test-assigned group by \(0.75-0.69=0.06\), matching the estimated decrease. A common mistake is to report a 6% relative decrease; the estimate here is a 6 percentage-point difference. The relative decrease would instead be \(0.06/0.75=8\%\).
Why the Assigned-Group Difference Is Causal
The justification can be stated using potential outcomes. Let \(Y_i(1)\) be unit \(i\)'s outcome if assigned the test condition and \(Y_i(0)\) the outcome if assigned the comparison condition. The unit-level assignment effect is \(Y_i(1)-Y_i(0)\), although only one of these two outcomes can be observed for any unit. Random assignment makes \(Z_i\) independent of these potential outcomes, in expectation. Consequently, the average observed outcome among units with \(Z_i=1\) estimates the average \(Y_i(1)\), while the comparison group's average estimates the average \(Y_i(0)\). Their difference estimates the average effect of assignment.
This argument also explains why an unadjusted assigned-group comparison is a sound starting point even when a test uses blocks. If randomization was stratified, calculate the treatment-comparison effect within each stratum and combine those effects using weights specified for the target population. Do not let the proportions of customers who happen to appear in each experimental arm silently determine the target weights when those proportions differ from the population the estimate is intended to represent.
Worked Example: Weighting Effects From Two Randomized Strata
A hypothetical price test is randomized within two pre-defined customer bands. The target population is 70% in Band L and 30% in Band H. In each band, 250 customers are assigned to each price condition. The average number of selected coverage options is 1.50 under comparison and 1.42 under test in Band L; in Band H the corresponding means are 1.20 and 1.00. The standard deviation in each arm is 0.50 in Band L and 0.40 in Band H. Estimate the target-population effect and its approximate standard error.
Method. Find each randomized within-band difference, then standardize to the stated target-population weights. This avoids treating the balanced experimental allocation as though it were the target population mix.
The Band L effect is \(1.42-1.50=-0.08\) options per customer. The Band H effect is \(1.00-1.20=-0.20\). Therefore the target-weighted estimate is
For an independent difference of means within a band, the estimated variance is \(s_1^2/n_1+s_0^2/n_0\). In Band L this is \(0.50^2/250+0.50^2/250=0.00200\). In Band H it is \(0.40^2/250+0.40^2/250=0.00128\). With independent randomized units across bands and fixed weights, the variance of the weighted estimate is
The standard error is \(\sqrt{0.0010952}=0.0331\) options per customer. Thus the estimate is a reduction of 0.116 selected options per target-population customer, with an approximate standard error of 0.0331. As a check on the weighting, the unweighted average of the two effects would be \((-0.08-0.20)/2=-0.14\), which answers a different question: an equally weighted average of the two bands.
Common mistake. Applying 50% weights because the test allocated equal numbers to the bands would produce \(-0.14\). The question specifies a target population that is 70% Band L, so the appropriate estimate is \(-0.116\).
Noncompliance: Keep Assignment Separate From Receipt
Noncompliance occurs when some units do not receive the condition to which they were assigned, or when some comparison-assigned units receive the test condition. The ITT comparison remains an estimate of the effect of the assignment policy. It generally does not equal the effect of actually receiving the test price, because assignment and receipt are no longer identical.
Comparing units by the price they actually received is not a safe substitute. Receipt can depend on customer characteristics, operational exceptions, or other factors related to the outcome. Conditioning on receipt can therefore destroy the comparability created by randomization. A treatment-received effect can sometimes be estimated using assignment as an instrumental variable, but that requires additional assumptions, not merely randomized assignment.
Worked Example: Noncompliance and the Assignment Effect
In a hypothetical test, 500 customers are assigned a new renewal price and 500 are assigned the existing price. The new price is actually received by 450 customers assigned to it and by 50 customers assigned to the existing price. There are 270 renewals in the test-assigned group and 300 in the comparison-assigned group. Calculate the ITT effect, the difference in treatment receipt by assignment, and the Wald estimate of an effect among compliers under the assumptions stated below.
Method. First calculate effects by assignment. Only then, if the instrumental-variable assumptions are defensible, divide the assignment effect by the assignment-induced difference in receipt.
The assigned-group renewal rates are \(270/500=0.54\) and \(300/500=0.60\). The ITT effect on renewal is \(0.54-0.60=-0.06\), or a decrease of 6 percentage points. The receipt rates are \(450/500=0.90\) and \(50/500=0.10\), so assignment increases receipt by \(0.90-0.10=0.80\), or 80 percentage points. The Wald ratio is
Under the assumptions of random assignment, a nonzero effect of assignment on receipt, no effect of assignment on the outcome except through receipt, and monotonicity (assignment does not cause anyone who would otherwise receive the test price to avoid it), this ratio estimates the average receipt effect for compliers: units whose receipt changes because of assignment. It is not automatically the effect for every customer. These assumptions may fail, for example if assignment changes communications or handling in ways that affect renewal even when the price received is unchanged.
Check. The Wald estimate is larger in magnitude than the ITT, as expected when assignment changes receipt for only 80% of units. A common mistake is to compare renewal rates for the 450 test-assigned customers who received the new price with rates for the 50 comparison-assigned customers who also received it. Those two groups are selected after randomization and need not be comparable.
Attrition and Missing Outcomes
Attrition occurs when an assigned unit's outcome is unavailable at the end of follow-up. The randomized assignment still identifies the ITT effect when outcomes are observed for everyone, but a comparison restricted to observed outcomes can be biased if observation depends on assignment or on the unobserved outcome. Reporting only completer means does not resolve this problem.
Investigators should report the number assigned, the number with observed outcomes, and the reasons for missingness by assigned group. A complete-case comparison requires assumptions about the missing outcomes; for example, it may be justified if outcome observation is independent of the outcome within the relevant groups. When that assumption is not credible, sensitivity analysis or bounds can make the uncertainty explicit.
Worked Example: Worst-Case Bounds With Missing Renewals
A hypothetical test assigns 1,000 customers to each price condition. Renewal is observed for 920 test-assigned customers, of whom 644 renew, and for 960 comparison-assigned customers, of whom 720 renew. Find the complete-case difference and worst-case bounds for the ITT effect on renewal, treating each missing outcome as either a nonrenewal or renewal as needed.
Method. The complete-case calculation describes only observed customers. For worst-case bounds on each assigned group's renewal rate, first count all observed renewals as renewals, then assign every missing outcome the value that makes the group rate as low or as high as possible.
Among observed outcomes, the test rate is \(644/920=0.7000\) and the comparison rate is \(720/960=0.7500\). The complete-case difference is \(0.7000-0.7500=-0.0500\). For the full test-assigned group, there are 80 missing outcomes. Its renewal rate is at least \(644/1{,}000=0.644\) if all 80 are nonrenewals, and at most \((644+80)/1{,}000=0.724\) if all renew.
There are 40 missing outcomes in the comparison group. Its rate is at least \(720/1{,}000=0.720\) and at most \((720+40)/1{,}000=0.760\). To find the smallest possible test-minus-comparison effect, use the lowest test rate and highest comparison rate: \(0.644-0.760=-0.116\). For the largest, use the highest test rate and lowest comparison rate: \(0.724-0.720=0.004\). Thus the worst-case effect bounds are \([-0.116,\,0.004]\), or a decrease of 11.6 percentage points through an increase of 0.4 percentage points.
Check. The complete-case estimate of \(-0.0500\) lies within the bounds, but the bounds also include zero. These broad bounds do not assert that the extremes are likely; they show what the observed data alone permit without assumptions about missing outcomes. Treating the observed rates as the full-group rates would silently assume away attrition.
Pre-Specify the Analysis
A causal estimate is easier to interpret when the analysis choices are set before outcomes are examined. Pre-specification reduces the risk of choosing definitions or subgroups because they produce a preferred result. It also makes the result reproducible. At a minimum, document:
- the primary outcome, its unit, and its follow-up window;
- the assignment groups to compare and whether the primary estimate is ITT;
- the target population and any stratum weights;
- how missing outcomes, ineligible records, and duplicate records will be handled;
- any baseline covariate adjustment and the method for calculating uncertainty;
- which outcomes or subgroup analyses are secondary or exploratory.
A baseline covariate measured before assignment can sometimes improve precision when included in a pre-specified analysis. A post-assignment variable, such as whether the customer opened a price communication, may be affected by assignment. Adjusting for it can change the estimand or introduce bias. Likewise, repeatedly examining outcomes and changing the stopping or analysis rule can make conventional confidence intervals and significance tests misleading.
The central discipline is to state the question before calculating: effect of assignment, effect of receipt under additional assumptions, or effect in a defined target population. The assigned-group comparison answers the first question directly. Noncompliance and attrition do not make the randomized test useless, but they constrain which conclusions the data support. This distinction is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each scenario, identify the requested assigned-group or sensitivity calculation.
- A randomized test has 250 customers in each assigned group. There are 150 renewals in the test group and 165 in the comparison group. What is the ITT effect on renewal probability? (Answer: \(-0.0600\), a decrease of 6.00 percentage points.)
- A target population is 80% in Segment A and 20% in Segment B. The randomized within-segment effects on a binary outcome are \(+0.03\) and \(-0.12\), respectively. What is the target-weighted effect? (Answer: \(0.80(0.03)+0.20(-0.12)=0.0000\).)
- Each assigned group contains 200 units. In the test group, 120 of 180 observed units have the outcome, and 20 outcomes are missing. In the comparison group, 130 of 190 observed units have the outcome, and 10 are missing. What are the worst-case bounds for the test-minus-comparison effect? (Answer: \([-0.1000,\,0.0500]\).)
- The outcome proportions by assignment are 0.48 in the test group and 0.52 in the comparison group. Receipt rates are 0.85 and 0.15, respectively. What are the ITT and Wald estimates? (Answer: ITT \(=-0.0400\); Wald estimate \(=-0.0400/0.70=-0.0571\), subject to the instrumental-variable assumptions.)
- Why is comparing only customers who actually received the test price not necessarily a randomized comparison? (Answer: Receipt is determined after assignment and may be related to customer characteristics or outcomes, so conditioning on receipt can break the comparability created by randomization.)