Why a Price Test Needs a Precise Design
A price change can affect more than the premium on a renewal offer. It may change whether a customer renews, the amount of premium collected, and the claims or expenses associated with the resulting coverage. A controlled test is useful only if its design makes clear which renewals are being studied, what pricing strategies are being compared, and when outcomes are measured.
The design is a specification made before results are observed. It should not be revised after seeing which customers accepted an offer or which claims occurred. Otherwise, the test population or outcome definition can become dependent on post-offer behavior, changing the question the comparison answers.
For a unit \(i\), let \(Y_i(T)\) denote its outcome under treatment strategy \(T\), and let \(Y_i(C)\) denote its outcome under control strategy \(C\). For a stated outcome, such as renewal or contribution, the target average incremental effect for a population of \(N\) eligible units is
This definition states the comparison of interest; it does not by itself provide an estimate. The treatment and control outcomes for the same unit cannot both be observed in a single test. The design therefore needs a credible comparison, and the assignment procedure must be specified separately. The central design requirement here is that the population, strategies, and measurement window are defined before assignment and outcome review.
Define the Test Population
Eligibility should identify the units to which the intended decision applies. In renewal pricing, the unit is often a renewal opportunity or policy renewal, but the definition must say exactly which one. Specify the business, jurisdiction, product, renewal dates, and any other inclusion criteria. Also state exclusions in advance, including their operational rationale.
A population definition should use information available at a fixed eligibility date. For example, excluding a policy because it cancels after receiving its offer would condition eligibility on a response that may itself be affected by price. That would no longer describe the original eligible renewal population. If a renewal changes status later, retain its eligibility record and define how its outcomes will be recorded.
Population boundaries also determine how broadly results can be applied. A test of one product and a particular renewal window does not automatically describe new business, other jurisdictions, or renewals with materially different constraints. If an optimizer’s candidate prices can only be offered to a subset because of a legal or system restriction, define that subset rather than describing the test as covering the entire portfolio.
Worked Example: Fixing Eligibility Before Offers
A hypothetical insurer plans to test renewal prices for a personal watercraft product. Its renewal system identifies 1,260 policies with effective dates from April 1 through April 30. Before prices are issued, 48 are identified as having a required coverage transition that makes the candidate strategy unavailable, and 32 are missing a required rating input. Define the eligible population and show the count.
Method: apply prespecified eligibility rules to the renewal file. The original file contains 1,260 renewal opportunities. The two exclusion categories are known before the offers are made and are stated explicitly.
The number excluded is \(48+32=80\). Therefore, the eligible population contains \(1{,}260-80=1{,}180\) renewal opportunities.
A usable specification is: policies for the named product with renewal effective dates April 1–30, with the required rating input present and no required coverage transition. Each eligible opportunity is counted once, using its status as of the prespecified eligibility date.
Check. Eligible plus excluded equals \(1{,}180+80=1{,}260\), the original file count. A common mistake would be to remove customers who later decline the offer. That is not part of this eligibility calculation; those customers were eligible when the test began.
Specify Treatment and Control Strategies
A treatment is not simply “a higher price.” It is the complete rule that produces the offer for an eligible unit. State the candidate rating or pricing rule, the reference price, applicable customer-level movement limits, floors or caps, rounding, and any other rule that changes the quoted amount. If the strategy differs by segment, specify how segment membership is determined.
The control should represent the relevant alternative decision: often the currently approved renewal pricing strategy, applied as it would have been for the same population and renewal dates. Define it concretely. “Business as usual” is insufficient if it could include discretionary overrides, changing discounts, or more than one rating version. Record which version and which rules apply.
To make the comparison interpretable, identify which elements are intended to differ and which are held consistent. Coverage, fees, payment options, communications, and eligibility rules can affect outcomes. If one of these changes along with price, the test compares a package of changes rather than isolating the effect of the price strategy.
Worked Example: Applying a Customer-Level Price Guardrail
For a hypothetical renewal, the control strategy produces a premium of $1,240. The proposed treatment strategy calls for a 9% increase, but a customer-level rule caps the increase at 7%. Both strategies use the same coverage, fees, and payment options. Determine the treatment offer and the premium difference from control.
Method: apply the treatment rule in its stated order. First calculate the unconstrained treatment premium:
The maximum permitted increase is \(0.07(\$1{,}240)=\$86.80\). The corresponding capped premium is \(\$1{,}240+\$86.80=\$1{,}326.80\). Since the unconstrained increase of $111.60 exceeds the $86.80 cap, the treatment offer is $1,326.80.
The treatment-minus-control premium difference is \(\$1{,}326.80-\$1{,}240=\$86.80\). The resulting movement is \(\$86.80/\$1{,}240=0.07\), or 7%.
Check. The treatment offer is below the unconstrained $1,351.60 and is exactly 7% above control. A common mistake is to apply the cap as a 7% reduction to the proposed $1,351.60. That would use the wrong base: the stated rule caps the change relative to the control premium.
Choose the Outcome Period and Outcomes
The outcome period should follow the timing of the decision and the effect being studied. Specify a common index date, such as the renewal effective date, and an exact follow-up interval. If the intended observation is a full policy term, say whether that means 365 days from the effective date, the scheduled expiration date, or another defined period. Do not use different follow-up durations merely because some records are available earlier.
The outcome itself must also have a precise definition. Renewal can mean accepting an offer by a particular date or having coverage in force on a specified date. Premium might mean offered premium, written premium, or earned premium; these are not interchangeable. A claims outcome needs a defined basis, such as losses incurred for events during the coverage period and valued at a specified date. Expenses and exposure likewise need consistent definitions.
An outcome period should allow enough time to observe the outcomes relevant to the decision. Retention may be observable soon after renewal, while claims and earned premium develop over the policy term and may need additional time for reporting or valuation. A design can specify distinct outcome windows for distinct measures, but each window should be declared in advance. For instance, it may define renewal status at a fixed date and policy-term loss experience through the end of coverage, valued at a stated later date.
Worked Example: Aligning Follow-Up With a Policy Term
A hypothetical test includes policies renewing on January 15, 2027. The intended insurance outcome is experience during one full 365-day term. The analyst proposes counting claims with report dates through December 31, 2027. Explain why that does not define the intended outcome period, and give a consistent specification.
Method: anchor the observation to coverage dates, not the calendar year or report date. A 365-day period beginning January 15, 2027 runs through January 14, 2028, inclusive. Thus, coverage-period outcomes should be tied to events or exposure within that defined term.
A claim from an accident on December 20, 2027 falls within the term, even if it is reported in January 2028. Conversely, a claim reported in December 2027 for an accident before January 15, 2027 is not a loss arising during this term. Report date alone does not identify the period to which the insured event belongs.
A consistent specification could measure renewal status on a stated post-offer date, measure earned premium and exposure over January 15, 2027–January 14, 2028, and value losses arising in that term as of a prespecified later valuation date. The chosen valuation date allows claim development to be observed consistently; it does not change the coverage dates defining the loss period.
Check. The term has 365 days because 2027 is not a leap year and the dates are consecutive through the day before January 15, 2028. The proposed December 31 cutoff omits the final 14 days and can omit claims reported later for events during the term.
Why the Comparison Must Match the Question
The target comparison is meaningful only when the strategies are applied to the same defined target population and outcomes are interpreted over the same measurement rules. To see why, let \(Y_i(T)\) and \(Y_i(C)\) be the outcomes that eligible unit \(i\) would have under the two strategies. The population-level contrast is the average of the unit-level differences:
Because both sums use the same \(N\) units, subtracting term by term gives
This identity explains why the population must be common to both strategies: changing the population changes the units in the contrast. Similarly, if treatment outcomes cover a full policy term but control outcomes cover only a calendar year, the terms no longer represent the same outcome definition. The formula defines the target contrast; it does not assert that a particular observed difference estimates it. Assignment, estimation, uncertainty, and power require additional methods.
Describe the candidate renewal pricing action and the alternative it may replace.
Define the population using information available at a specified date, including exclusions.
Specify the price rules, guardrails, and other offer features for treatment and control.
Choose a common index date, follow-up duration, outcome definitions, and valuation timing.
Confirm that the population, price contrast, and outcomes answer the intended business question.
Common Design Traps
- Using a moving eligibility definition. Excluding units because of post-offer cancellation or response can make the population depend on the price being tested. Define eligibility before the offer and retain all eligible units in the test records.
- Leaving the control vague. “Current pricing” can change over time or include discretionary exceptions. Name the pricing version and the rules that produce the control offer.
- Comparing more than price unintentionally. If coverage, fees, or communications also differ, describe the treatment as that combined strategy rather than attributing the entire contrast to price alone.
- Mixing calendar and coverage periods. A calendar-year claims extract may not represent a full policy term. Tie losses to event dates and define a consistent valuation date.
- Changing rules after seeing outcomes. Revising exclusions, endpoints, or follow-up after observing results can alter the question. Document the design before outcome review.
Check Your Understanding
For each problem, state the design choice or perform the requested calculation.
- A hypothetical portfolio file has 5,400 renewal opportunities. Before offers, 135 fail a prespecified product-eligibility rule and 65 lack a required rating variable. How many opportunities belong to the eligible test population? (Answer: 5,200.)
- A control offer is $960. Treatment applies a proposed 9% increase subject to a maximum customer-level increase of 7% relative to control. What is the treatment offer and its dollar difference from control? (Answer: $1,027.20; increase of $67.20.)
- A policy term begins January 15, 2027, and the outcome window is 365 consecutive days starting on that date. What is the final included date? (Answer: January 14, 2028.)
- A test measures renewal by February 10, earned premium during the full policy term, and losses from events during that term valued on June 30 after expiration. Is this one outcome window or several prespecified outcome definitions? (Answer: Several prespecified outcome definitions: renewal has its own date, while premium and event-period losses use the policy term, with losses valued later.)
- A proposed treatment changes the premium rule and removes a payment fee, while the control retains the fee. What can the comparison be described as measuring? (Answer: The effect of the combined pricing-and-fee strategy, not the isolated effect of the premium change.)