Why Use Cohorts for Plan Evaluation?
A before-and-after comparison of an experience rating plan can mistake a general change in claim costs for an effect of the plan. For example, losses may fall after a plan change because of a broad safety improvement that also affects risks not subject to the change. A comparison cohort helps separate that contemporaneous movement from the change observed among affected risks.
The basic design follows two groups over time: a treated cohort whose risks are exposed to the plan change, and a comparison cohort that remains under the prior arrangement during the evaluation. Outcomes are measured in a pre-change period and a post-change period. The comparison is informative only if the groups are sufficiently comparable, or their outcomes are standardized to a comparable basis.
Let \(Y_{T,0}\) and \(Y_{T,1}\) be the treated cohort’s mean outcomes before and after the change, and let \(Y_{C,0}\) and \(Y_{C,1}\) be the comparison cohort’s corresponding outcomes. With one pre-change period and one post-change period, the additive estimate is
The estimate is not simply the treated cohort’s before-and-after change. It removes the comparison cohort’s observed change, which serves as an estimate of the change that would have occurred for the treated cohort without the plan change.
What Makes the Comparison Credible?
A cohort design organizes the comparison; it does not by itself prove that the plan change caused the measured difference. The central identifying condition is parallel trends: in the absence of the plan change, the average outcome for the treated cohort would have changed by the same amount as the average outcome for the comparison cohort. The cohorts do not need equal outcome levels before the change, but their untreated trends must be comparable.
Other conditions matter in practical ratemaking work. The outcome must be defined and measured consistently across groups and periods. Cohort membership should not shift in a way that changes the groups’ risk mix, unless the analysis standardizes that mix. The comparison cohort must not itself be affected by the plan change, and there should be no important anticipation of the change before the defined pre-period. Differences in claim development, exposure measurement, or loss-cost trend can also undermine the comparison if they affect the cohorts differently.
For example, a loss cost measured per exposure unit is often more interpretable than total losses when cohort sizes differ. But the exposure base must be appropriate and consistently applied. If risks have materially different class or territory mixes, calculate outcomes at a common mix or compare within suitable strata before aggregating.
Why the Difference-in-Differences Formula Works
The justification is a counterfactual comparison. Let \(Y_{T,1}(0)\) denote the treated cohort’s post-period outcome had it not received the plan change. The treatment effect is \(Y_{T,1}(1)-Y_{T,1}(0)\), where \(Y_{T,1}(1)\) is the observed treated outcome under the changed plan. The untreated post-period outcome is not observed for the treated cohort, so it must be estimated.
Parallel trends says that the treated cohort’s change without treatment would equal the comparison cohort’s observed change. In symbols,
Rearranging gives \(Y_{T,1}(0)=Y_{T,0}+(Y_{C,1}-Y_{C,0})\). Subtract this counterfactual from the observed treated outcome:
This is the DiD formula. The result is an estimate of the plan’s incremental effect only to the extent that the counterfactual trend assumption and the other design conditions are plausible.
Worked Examples
Worked Example: Choosing the Correct Pre-Change Baseline
A hypothetical insurer evaluates a plan change using loss cost in dollars per exposure unit. Its affected cohort has loss costs of $102, $110, and $90 in pre-change period 1, pre-change period 2, and the post-change period, respectively. A comparison cohort has corresponding loss costs of $90, $100, and $86. Calculate the DiD using pre-change period 2 as the baseline. Then calculate the estimate using period 1 instead, and distinguish both from the pre-change differential trend.
Method. Calculate the treated and comparison changes from the same selected baseline, then subtract the comparison change from the treated change.
Using pre-change period 2 as the baseline, the treated change is \(90-110=-20\) dollars per exposure unit. The comparison change is \(86-100=-14\) dollars per exposure unit. Therefore,
Using pre-change period 1 instead, the treated change is \(90-102=-12\), while the comparison change is \(86-90=-4\). Thus,
The earlier pre-change differential trend is a separate calculation. The treated-minus-comparison gap was \(102-90=12\) in period 1 and \(110-100=10\) in period 2, so the gap changed by \(10-12=-2\) dollars per exposure unit before the plan change. It is not either post-change DiD estimate.
Check. The period-1-baseline estimate is $2 lower than the period-2-baseline estimate: \(-8-(-6)=-2\), exactly the change in the pre-change gap. The specified period-2-baseline estimate is \(-6\), not \(-2\). The estimate suggests a $6 decrease relative to the comparison cohort, subject to the design assumptions.
Common mistake avoided: Confusing the \(-2\) earlier movement in the cohort gap with the post-change estimate, or using different baseline periods for the two cohorts.
Worked Example: Standardizing to a Common Risk Mix
A hypothetical plan evaluation divides risks into two strata, A and B. Loss costs, in dollars per exposure unit, are as follows: the treated cohort has pre-change rates of $80 in A and $160 in B, and post-change rates of $70 in A and $140 in B. The comparison cohort has pre-change rates of $70 in A and $150 in B, and post-change rates of $66 in A and $142 in B. Evaluate the change using a fixed mix of 60% A and 40% B in every group and period.
Method. Apply the same fixed weights to each cohort-period combination before calculating DiD. This avoids letting a changing proportion of A and B drive the comparison.
The standardized treated pre-change outcome is \(0.60(80)+0.40(160)=48+64=112\). Its standardized post-change outcome is \(0.60(70)+0.40(140)=42+56=98\). The treated change is \(98-112=-14\).
The standardized comparison pre-change outcome is \(0.60(70)+0.40(150)=42+60=102\). Its standardized post-change outcome is \(0.60(66)+0.40(142)=39.6+56.8=96.4\). The comparison change is \(96.4-102=-5.6\).
Check. Within stratum A, the DiD is \((70-80)-(66-70)=-10-(-4)=-6\). Within B, it is \((140-160)-(142-150)=-20-(-8)=-12\). Weighting these gives \(0.60(-6)+0.40(-12)=-3.6-4.8=-8.4\), the same result. This agreement checks the standardization arithmetic.
Common mistake avoided: Comparing crude averages that use different mixes of A and B. The fixed weights define the target mix for this estimate; they should be selected for a clear actuarial reason and applied consistently.
Worked Example: Estimating a Multiplicative Change in Frequency
A hypothetical insurer observes 100 claims over 1,000 exposure units in the treated cohort before a plan change and 80 claims over 1,000 exposure units afterward. In the comparison cohort, there are 80 claims over 1,000 exposure units before and 72 claims over 1,000 afterward. Estimate the multiplicative DiD in claim frequency and interpret it.
Method. For a multiplicative comparison, divide the treated post-to-pre rate ratio by the comparison post-to-pre rate ratio. This estimates the relative change in the treated rate beyond the comparison trend.
The treated frequencies are \(100/1{,}000=0.100\) and \(80/1{,}000=0.080\) claims per exposure unit. Its rate ratio is \(0.080/0.100=0.80\). The comparison frequencies are \(80/1{,}000=0.080\) and \(72/1{,}000=0.072\), giving a rate ratio of \(0.072/0.080=0.90\).
The treated frequency fell to about 88.89% of the level implied by applying the comparison cohort’s relative trend. The estimated incremental relative decrease is \(1-0.8889=0.1111\), or 11.11%. For contrast, the additive DiD is \((0.080-0.100)-(0.072-0.080)=-0.020-(-0.008)=-0.012\) claims per exposure unit. These are different measures: the first is a relative effect, and the second is an absolute rate difference.
Check. If the treated rate had changed only by the comparison ratio of 0.90, its counterfactual post-change rate would be \(0.100(0.90)=0.090\). The observed rate is \(0.080\), and \(0.080/0.090=0.8889\), confirming the multiplicative result.
Common mistake avoided: Subtracting percentage changes as though they were an absolute rate difference. Use the measure appropriate to the question, and state whether the answer is a rate ratio or a difference in rates.
Practical Diagnostics and Exam Traps
When multiple pre-change periods are available, compare the treated-minus-comparison outcome gaps across those periods. A marked change in the gap before treatment is evidence against a simple parallel-trends assumption, though stable historical gaps do not prove that the post-period counterfactual trend would remain parallel. Where feasible, report estimates under more than one reasonable baseline and explain their sensitivity rather than presenting one as assumption-free.
A reliable calculation sequence is:
Specify the outcome, its unit, the treatment date, the affected risks, and the comparison risks.
Use consistent exposure denominators and adjust or standardize material differences in risk mix.
Use the same pre- and post-period definitions for the treated and comparison cohorts.
Subtract the comparison change from the treated change, then state the units, direction, and assumptions behind the estimate.
A common exam error is to compare total losses for cohorts with different exposure volumes, or to subtract the comparison group’s level from the treated group’s level only after the change. The DiD uses changes in the outcome, and the outcome must be defined on a comparable basis. Another error is to label a negative estimate as a causal plan effect without mentioning the parallel-trends condition. The calculation is straightforward; the credibility of the counterfactual is the actuarial judgment.
Check Your Understanding
For each item, identify the appropriate comparison and keep the outcome’s units or scale explicit.
- Affected risks have loss costs of $120 before and $105 after a plan change. Comparison risks have loss costs of $100 before and $92 after. Calculate the additive DiD in dollars per exposure unit. (Answer: \(-$7\) per exposure unit.)
- The treated-minus-comparison gap is $8 in pre-period 1 and $13 in pre-period 2. Post-period rates are $95 for the treated cohort and $86 for the comparison cohort; pre-period 2 rates were $110 and $95, respectively. Calculate the period-2-baseline DiD and state the pre-change gap movement. (Answer: \((95-110)-(86-95)=-15-(-9)=-$6\) per exposure unit; the pre-change gap increased by $5, which raises a parallel-trends concern.)
- Use fixed stratum weights of 25% A and 75% B. Treated rates are A: $40 before and $36 after; B: $100 before and $90 after. Comparison rates are A: $35 before and $34 after; B: $95 before and $91 after. Calculate the standardized additive DiD. (Answer: treated standardized rates are $85 and $76.5, a change of \(-$8.5\); comparison rates are $80 and $76.75, a change of \(-$3.25\); DiD \(=-$5.25\) per exposure unit.)
- Treated frequency changes from 0.12 to 0.09 claims per exposure unit, while comparison frequency changes from 0.10 to 0.09. Calculate the multiplicative DiD ratio and its relative decrease. (Answer: \((0.09/0.12)/(0.09/0.10)=0.75/0.90=0.8333\), a 16.67% relative decrease.)
- Explain why a validly calculated DiD may still not support a causal conclusion if the comparison cohort is affected by the plan change or if, without the change, the treated cohort would have had a different trend. (Answer: The comparison no longer identifies the treated cohort’s untreated counterfactual change, so the DiD cannot isolate the plan’s effect.)