Why Assignment Fidelity Must Be Monitored
A controlled price test can be designed correctly and still be implemented incorrectly. A policy may be assigned to a test price but receive the usual price because of a manual override. A control policy may accidentally receive the test price. A system may fail to display any valid renewal offer. These are implementation problems: they affect whether the assigned treatment reaches the intended unit. They are not, by themselves, evidence about how customers respond to a price.
Monitoring assignment and treatment fidelity means checking the chain from eligibility through assignment to actual delivery. The central question is not merely whether the test groups were created, but whether each eligible randomized policy received the price specified by its assignment, at the intended time and in the intended manner. Clear records preserve the distinction between the price assigned, the price delivered, and what the customer subsequently did.
For a price test, the unit might be a policy, household, account, or another unit selected during test design. The monitoring unit should match the randomization unit established in the protocol. If assignment was made by household, for example, counting each policy as an independent assignment can misstate both the denominator and the number of treatment deliveries.
Define the Audit Before Calculating Rates
A useful audit begins with an explicit specification of the eligible population, assignment unit, assigned arm, treatment definition, and delivery window. “Received the test price” must have an operational meaning. It could mean that a valid quote was generated and shown to the customer during a specified renewal window; it should not be replaced by “the customer accepted the quote.” Acceptance is a later customer action, not a delivery record.
For a binary treatment, define treatment fidelity as the proportion of units assigned to treatment that received the specified treatment. Define control adherence as the proportion assigned to control that received the specified control condition. Control contamination is the proportion assigned to control that instead received the test treatment. The denominators are assigned units in the relevant arm, not all policies in the portfolio and not all quote events.
These measures require mutually interpretable receipt categories. A policy that received no valid quote is not automatically compliant with either arm. It should be recorded as missing or failed delivery, with its reason investigated. A policy with a documented approved exception may be a valid protocol exception, but it still should not be silently counted as having received the originally assigned price.
A Practical Monitoring Sequence
A reliable process follows the records in order. First, establish who was eligible and confirm that eligible units were assigned once to the intended arm. Next, link each assignment to the quote or renewal record using a stable unit identifier. Then compare assigned and delivered prices, validate the delivery date, and classify discrepancies by cause. Finally, summarize rates using the correct unit of analysis and inspect their pattern across operational groups and time.
Retain the eligibility decision, assignment unit, arm, assignment timestamp, and any randomization block. Do not overwrite the original arm when a later override occurs.
Match assignments to the valid quote or renewal offer using the protocol’s identifier and delivery window. Flag unmatched records and duplicate events for review.
Record whether the assigned price was delivered, another price was delivered, no valid offer was delivered, or a documented exception applied. Preserve a reason code rather than collapsing all deviations into one category.
Calculate arm-specific rates using assigned units as denominators. Compare results across blocks, channels, and time periods, then trace unusual patterns to records and processes.
Worked Examples
Worked Example: Separating Overrides From Missing Offers
A hypothetical renewal test assigns 120 policies to a test price and 120 to the usual-price control. Among the test-assigned policies, 102 receive the assigned test price, 10 receive the usual price after a manual override, and 8 have no valid offer recorded. Among control-assigned policies, 114 receive the usual price and 6 receive the test price because of a configuration error. Calculate treatment fidelity, control contamination, control adherence, and overall exact assignment-and-delivery compliance.
Method. Use the number assigned to each arm as its denominator. A manual override to the other price is not receipt of the assigned treatment, and an absent offer is also not a match.
Treatment fidelity is \(102/120=0.8500\). Thus, the treatment price reached 0.8500 of the treatment-assigned policies.
Control contamination is \(6/120=0.0500\), because six control-assigned policies received the test price. Control adherence is \(114/120=0.9500\).
There are \(102+114=216\) exact assignment-and-delivery matches among \(120+120=240\) assigned policies. Overall exact compliance is \(216/240=0.9000\). The remaining \(24\) consist of 10 overrides, 8 missing offers, and 6 control contaminations; \(10+8+6=24\), which reconciles to \(240-216=24\).
Check. Treatment-arm nonmatches are \(10+8=18\), and \(120-102=18\). Control-arm nonmatches are 6, and \(120-114=6\). The counts reconcile in both arms.
Common mistake avoided: Counting the 10 overridden policies as treated merely because they were originally assigned to the test arm would confuse assignment with receipt. Keep their treatment assignment in the assignment file, but classify their delivered price as a deviation.
Worked Example: Policy-Level Fidelity Versus Quote-Level Fidelity
A hypothetical test assigns 100 policies to a test price. For the protocol’s primary delivery check, each policy is evaluated at its first eligible renewal quote. Ninety policies receive the assigned price at that quote. The other 10 do not; each of those 10 policies has one quote event on file, while each of the 90 compliant policies has two quote events, both showing the assigned price. Calculate policy-level fidelity and the event-level matching rate, and explain which rate answers the protocol question.
Method. The assignment unit and specified evaluation point are the policy and its first eligible quote. Therefore, the policy-level denominator is 100, even though the system contains a different number of quote records.
Policy-level fidelity is \(90/100=0.9000\). This is the proportion of assigned policies that received the assigned price at the protocol-defined first eligible quote.
For the event-level calculation, the 90 compliant policies contribute \(90\times2=180\) matching quote events. The other 10 policies contribute 10 nonmatching events. The total event count is \(180+10=190\), so the event-level matching rate is \(180/190=0.9474\), rounded to four decimal places.
The two rates differ because the quote-event denominator gives more weight to policies with more recorded events. The event-level rate of 0.9474 does not answer the stated policy-level question; the relevant fidelity result is 0.9000. Event-level summaries can still be useful for diagnosing system behavior, provided they are labeled as event-level measures.
Check. The 190 events comprise 180 matching and 10 nonmatching events, so the event-level rate is below 1. The policy-level count is 90 matching policies out of 100, independently of how many additional quotes compliant policies generated.
Common mistake avoided: Treating every quote record as an independent assigned unit can make delivery look better or worse solely because some policies produce more records. Select the denominator from the test design, not from whichever data table is easiest to count.
Pooling Rates Across Blocks
Tests often use blocks, such as renewal month, territory, or an operational channel. To report an overall compliance rate, pool the counts rather than taking an unweighted average of block rates when block sizes differ. If block \(h\) contains \(n_h\) assigned units and has fidelity \(f_h\), then it has \(n_h f_h\) matching units. Summing matching counts across blocks and dividing by the total assigned count gives the pooled rate.
This follows directly from the definition of a proportion. The total number of matches is the sum of the matches in each block, \(\sum_h n_h f_h\); the total denominator is the sum of assigned units, \(\sum_h n_h\). Dividing the first total by the second yields the overall proportion. An ordinary arithmetic mean of block rates gives each block equal weight, so it agrees with the pooled rate only when the block sizes are equal (or in special cases where the rates happen to offset).
Worked Example: Checking a Pooled Fidelity Rate Across Five Blocks
A hypothetical monitoring team reviews five equally sized blocks of 20 treatment-assigned policies each. The observed block fidelity rates are 0.90, 0.80, 0.90, 0.85, and 0.80. Verify the overall fidelity rate from the counts, then consider what should be investigated if nine deviations are documented as manual overrides and six as price-mapping failures.
Method. Convert each rate to a count using its block size. The five matching counts are \(20(0.90)=18\), \(20(0.80)=16\), \(20(0.90)=18\), \(20(0.85)=17\), and \(20(0.80)=16\).
The total number of matches is \(18+16+18+17+16=85\), and the total assigned count is \(5(20)=100\). Therefore pooled fidelity is \(85/100=0.85\). Because all five blocks have equal size, the arithmetic mean of the rates also equals \((0.90+0.80+0.90+0.85+0.80)/5=0.85\).
There are \(100-85=15\) deviations. The recorded causes reconcile because \(9+6=15\). The team should trace the nine overrides to the applicable approval and processing records, and trace the six mapping failures to the system configuration and the affected quote dates. The overall rate alone does not identify which mechanism caused the failures.
Check. The computed count-based rate is 0.85, and it agrees with the mean because all block denominators are 20. If block sizes differed, the count-based calculation would remain valid, while the unweighted mean generally would not.
Common mistake avoided: A rate such as 0.85 is a proportion, not the number 85. Keeping rates on the 0-to-1 scale makes the calculation \(85/100=0.85\) consistent with the displayed block rates.
Diagnosing Deviations Without Misreading Them
A deviation rate is a monitoring signal, not a complete explanation. Useful reason codes distinguish at least manual override, system mapping or rating error, no valid offer, late or early delivery, unmatched identifier, and approved exception. Where the protocol defines a material price tolerance or a valid delivery window, apply that rule consistently and retain both the observed price and the assigned price for audit.
The pattern of discrepancies helps locate the implementation problem. Concentration in one channel may point to a channel-specific workflow; concentration after a release date may suggest a system change; concentration among certain blocks may reveal a handoff or configuration issue. These are diagnostic leads, not proof of cause. Verify them against source records, timestamps, and the documented process.
Keep assignment and receipt in separate fields. Reclassifying a policy into the arm matching its delivered price erases the fact that assignment was not followed. It also obscures whether deviations are balanced across arms and prevents a transparent account of the test as implemented. Monitoring should document deviations and their timing; determining customer response to the assigned test is a separate analysis.
Common Monitoring Traps
- Using the wrong denominator. Divide by units assigned to the relevant arm, unless the test protocol explicitly defines a different eligible delivery population.
- Counting acceptance as delivery. A customer’s decision after seeing a quote is not the same as whether the assigned quote was presented.
- Ignoring missing offers. A policy with no valid offer is not a successful match simply because no conflicting price was observed.
- Mixing units. Do not combine policy-level assignments with quote-level events without explicitly labeling and reconciling the measures.
- Averaging unequal-sized blocks equally. Pool matching counts and denominators unless equal weighting is specifically the target summary.
- Overwriting assignment after an override. Preserve the assigned arm and record actual delivery separately, with a reason and timestamp where available.
Check Your Understanding
For each problem, use the assignment unit and denominator specified in the scenario. Treat prices and delivery records as hypothetical.
- A test assigns 80 policies to a test price. Seventy-two receive it, five receive the usual price after overrides, and three have no valid quote. What is treatment fidelity, and how many policies are classified as nonmatches? (Answer: 0.9000 fidelity; 8 nonmatches.)
- A control arm has 150 assigned households. Nine receive the test price by mistake; 141 receive the usual price. Calculate contamination and control adherence. (Answer: contamination \(=9/150=0.0600\); adherence \(=141/150=0.9400\).)
- A test has 40 assigned policies. Thirty-six receive the assigned price at the first eligible quote. Those 36 policies each have three quote records, all matching; the other four each have one nonmatching quote. Find policy-level fidelity and the event-level matching rate. (Answer: policy-level fidelity \(=36/40=0.9000\); event-level matching \(=108/112=0.9643\).)
- Two blocks have 20 and 80 treatment-assigned policies. Their fidelity rates are 0.80 and 0.95, respectively. Find pooled fidelity and explain why the simple mean is not the pooled rate. (Answer: \((20(0.80)+80(0.95))/100=0.9200\); the simple mean is 0.8750 and weights the unequal-sized blocks equally.)
- Among 200 assigned policies, 170 receive the assigned price, 18 receive another price after overrides, and 12 have no valid offer. Check the compliance count and rate, and state why the 12 missing offers should not be counted as matches. (Answer: \(170+18+12=200\); exact compliance is \(170/200=0.8500\); missing offers do not establish that the assigned price was delivered.)