Why Pair Reserve-Method Comparisons?
Suppose an analyst wants to compare reserve estimates from two methods, such as a credibility-weighted projection and an alternative development projection. Both methods respond to the same underlying features of a portfolio: claim emergence, severity, exposure, and the amount already paid. If each method is evaluated on different simulated portfolios, random differences between those portfolios can obscure the difference attributable to the methods.
A paired simulation controls this problem by applying both methods to the same simulated input in each replicate. The comparison is then made within each replicate, and the average of those within-replicate differences estimates the average difference between methods under the chosen simulation model. This tutorial is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
The direction of subtraction must be stated and kept consistent. If \(D_j=R_{A,j}-R_{B,j}\), a negative result indicates that method \(A\) produced a lower reserve estimate than method \(B\). The comparison is about the selected target—usually the mean method difference under a specified simulation model—not a claim that one method is correct in every situation.
Estimating the Difference and Its Simulation Error
The observed paired differences \(D_1,\ldots,D_M\) are treated as a sample from the distribution of paired differences generated by the simulation procedure. Their sample variance is \(s_D^2\), and the Monte Carlo standard error of the estimated mean difference is \(s_D/\sqrt{M}\). This measures simulation error in estimating the mean difference; it is not the predictive standard deviation of future unpaid losses.
The uncertainty estimate is meaningful only when the replicates are generated according to the intended simulation design. For example, if the goal is to compare methods across a range of possible future claim outcomes, the same simulated future outcomes must be supplied to both methods in each replicate. If the goal is conditional on one observed triangle, the simulated inputs should instead follow the stated conditional design. The pairing itself does not decide which design is appropriate.
This identity follows by expanding the squared deviation of a difference. If \(X=R_A-E[R_A]\) and \(Y=R_B-E[R_B]\), then \(\operatorname{Var}(R_A-R_B)=E[(X-Y)^2]=E[X^2]+E[Y^2]-2E[XY]\). The first two terms are the marginal variances, while \(E[XY]\) is the covariance. Thus, matching scenarios matters because it preserves the dependence between the two methods' estimates.
Worked Paired Comparisons
Worked Example: A Difference Driven by a Shared Simulation Input
A hypothetical reserve analyst compares two methods on simulated portfolio scenarios. In every replicate, both methods use the same \(Z\), where \(Z\sim N(0,1)\). For this illustration, the reserve outputs in thousands of dollars are \(R_A=100+2Z\) and \(R_B=96+Z\). Four simulated standard-normal inputs are \(-1,0,1,2\). Assume a 0% effective annual interest rate, so the displayed amounts are undiscounted. Mortality is not applicable.
Method. Calculate the two outputs on each shared input, subtract within each replicate, and then calculate the mean and Monte Carlo standard error of the paired differences.
| Replicate | \(Z\) | \(R_A=100+2Z\) | \(R_B=96+Z\) | \(D=R_A-R_B\) |
|---|---|---|---|---|
| 1 | -1 | 98 | 95 | 3 |
| 2 | 0 | 100 | 96 | 4 |
| 3 | 1 | 102 | 97 | 5 |
| 4 | 2 | 104 | 98 | 6 |
The differences sum to \(3+4+5+6=18\), so \(\bar{D}=18/4=4.5\) thousand dollars. Their deviations from 4.5 are \(-1.5,-0.5,0.5,1.5\). The squared deviations sum to \(2.25+0.25+0.25+2.25=5\), giving \(s_D^2=5/(4-1)=5/3\) and \(s_D=\sqrt{5/3}=1.2910\) thousand dollars. Therefore, \(\widehat{\operatorname{SE}}(\bar{D})=1.2910/\sqrt{4}=0.6455\) thousand dollars. To four decimal places, the estimated mean difference is \(\$4,500.00\), with Monte Carlo standard error \(\$645.50\).
Check. Algebraically, \(D=(100+2Z)-(96+Z)=4+Z\). The four inputs therefore imply differences \(3,4,5,6\), matching the table. The method avoids treating the two sets of outputs as unrelated: both are functions of the same \(Z\).
Worked Example: Comparing Two Reserve Projections Across Shared Claim Emergence
A hypothetical insurer compares two reserve projections using a common simulated claim-emergence index \(N\sim\operatorname{Poisson}(4)\) in every replicate. For this example, the specified projection outputs, in thousands of dollars, are \(R_A=120+5N\) and \(R_B=110+3N\). Five simulated values of \(N\) are \(1,2,4,5,8\). Assume a 0% effective annual interest rate; amounts are undiscounted. Mortality is not applicable.
Method. Apply both projection rules to each shared \(N\), then calculate the paired differences and their sample standard error.
| Replicate | \(N\) | \(R_A\) | \(R_B\) | \(D=R_A-R_B\) |
|---|---|---|---|---|
| 1 | 1 | 125 | 113 | 12 |
| 2 | 2 | 130 | 116 | 14 |
| 3 | 4 | 140 | 122 | 18 |
| 4 | 5 | 145 | 125 | 20 |
| 5 | 8 | 160 | 134 | 26 |
The differences sum to \(12+14+18+20+26=90\), so \(\bar{D}=90/5=18\) thousand dollars. The deviations from 18 are \(-6,-4,0,2,8\); their squared sum is \(36+16+0+4+64=120\). Hence \(s_D^2=120/(5-1)=30\), \(s_D=\sqrt{30}=5.4772\), and \(\widehat{\operatorname{SE}}(\bar{D})=\sqrt{30}/\sqrt{5}=\sqrt{6}=2.4495\) thousand dollars. To four decimal places, the estimated mean difference is \(\$18,000.00\), with Monte Carlo standard error \(\$2,449.49\).
Check. Subtracting the projection formulas gives \(D=(120+5N)-(110+3N)=10+2N\). Substituting the five \(N\) values gives \(12,14,18,20,26\), as calculated. The result describes these specified projections under the assumed Poisson simulation model; it does not establish that either projection is unbiased.
Worked Example: Seeing the Cost of Ignoring Pairing
For a third hypothetical comparison, both reserve methods are applied to each of the same four simulated claim triangles. The simulation model uses claim counts \(K\sim\operatorname{Poisson}(3)\) and individual claim severities \(X\sim\operatorname{Gamma}(2,10)\), with shape 2 and scale 10; counts and severities are independent. The table gives the resulting method outputs in thousands of dollars. Assume a 0% effective annual interest rate; amounts are undiscounted. Mortality is not applicable.
| Replicate | \(R_A\) | \(R_B\) | \(D=R_A-R_B\) |
|---|---|---|---|
| 1 | 111 | 108 | 3 |
| 2 | 125 | 124 | 1 |
| 3 | 139 | 140 | -1 |
| 4 | 153 | 152 | 1 |
Paired calculation. The differences sum to \(3+1-1+1=4\), so \(\bar{D}=1\) thousand dollars. Deviations from 1 are \(2,0,-2,0\), with squared sum \(8\). Thus \(s_D^2=8/3\) and the standard error is \(\sqrt{(8/3)/4}=\sqrt{2/3}=0.8165\) thousand dollars.
Unpaired calculation for contrast. The method \(A\) mean is \((111+125+139+153)/4=132\); its squared deviations sum to \(980\), so \(s_A^2=980/3=326.6667\). The method \(B\) mean is \((108+124+140+152)/4=131\); its squared deviations sum to \(1100\), so \(s_B^2=1100/3=366.6667\). If these were incorrectly treated as independent samples, the estimated standard error of the difference would be \(\sqrt{(s_A^2+s_B^2)/4}=\sqrt{173.3333}=13.1650\) thousand dollars.
The contrast is large because the outputs move together across shared triangles. The sample covariance is \((483+49+63+441)/3=345.3333\), using products of the paired deviations from their respective means. Consequently, \(s_D^2=s_A^2+s_B^2-2(345.3333)=2.6667\), agreeing with the direct paired calculation. To four decimal places, the paired mean difference is \(\$1,000.00\), with Monte Carlo standard error \(\$816.50\). This example's sample is small; the arithmetic illustrates the benefit of pairing, not a claim that this error estimate is precise.
Check. The paired differences calculated directly from the table are \(3,1,-1,1\), whose mean and variance agree with the paired calculations. The unpaired standard error answers a different calculation by discarding the observed matching between methods.
Design and Interpretation
A paired comparison is most useful when both methods can be applied to the same complete simulated scenario and their outputs are expected to respond similarly to the scenario's random features. A replicate should preserve the matching of all relevant inputs, not just one convenient variable. If one method uses simulated claim counts, severities, and payment timing, while another uses the same counts but independently generated severities, the comparison is only partly paired.
The simulation target also matters. A conditional comparison holds observed information fixed and simulates only specified future uncertainty; an unconditional comparison also simulates variation in the inputs that are treated as random. Use the design that matches the question, and apply both methods consistently within that design. Pairing does not repair a misspecified model, an inconsistent input definition, or a reserve method implemented incorrectly.
Interpret \(\bar{D}\) and its Monte Carlo standard error together. A small standard error means the simulation has estimated its mean difference relatively precisely under the chosen model. It does not establish that the methods' reserves are close for every scenario, nor does it quantify all uncertainty in the reserve estimate. The spread of individual \(D_j\) values describes scenario-to-scenario differences in this simulation, while \(s_D/\sqrt{M}\) describes error in estimating their average.
Common Exam Trap
A second practical check is the scenario identifier. Before calculating differences, confirm that row \(j\) in one method's output corresponds to the same simulated portfolio and random realization as row \(j\) in the other method's output. Pairing misaligned rows can create an arbitrary covariance and make the reported uncertainty misleading.
Check Your Understanding
For each problem, use the paired differences when the same simulated input is supplied to both methods. Unless stated otherwise, reserves are undiscounted; mortality is not applicable.
- Two methods produce paired reserve outputs, in thousands, of \(A=(42,50,61)\) and \(B=(40,47,60)\). Calculate the mean of \(D=A-B\). (Answer: \(2.0000\) thousand dollars)
- Four paired differences, in thousands, are \(2,4,6,8\). Calculate the sample variance of the differences and the Monte Carlo standard error of their mean. (Answer: \(s_D^2=6.6667\) thousand-dollars squared; standard error \(=1.2910\) thousand dollars)
- In each replicate, \(R_A=75+3N\) and \(R_B=70+N\), in thousands, using the same simulated \(N\). For \(N=1,3,5\), calculate the three differences and their mean. (Answer: differences \(8,12,16\); mean \(=12.0000\) thousand dollars)
- A paired analysis gives \(s_A^2=100\), \(s_B^2=64\), and sample covariance \(60\), all in thousands squared. Calculate the sample variance of the paired difference. (Answer: \(100+64-2(60)=44\) thousand-dollars squared)
- For paired differences \(-2,0,2,4\), calculate the mean difference and its Monte Carlo standard error. State which method has the larger average output if \(D=R_A-R_B\). (Answer: mean \(=1.0000\) thousand dollars; standard error \(=\sqrt{20/3}/2=1.2910\) thousand dollars; method \(A\) has the larger average output)