Why Separate Loss Regimes?
A single correlation summarizes linear association across the observations included in its calculation. It can obscure important differences when ordinary-loss scenarios and high-loss scenarios behave differently. For example, paired losses might move together in ordinary conditions but show little association among high-loss outcomes. The reverse can also occur. An insurer using an unconditional correlation to describe both situations should therefore check whether the association changes after observations are classified by loss regime.
A regime can be a category given in the data, such as ordinary versus stressed conditions, or an event defined by a threshold, such as one loss exceeding a stated amount. The choice of regime matters: a different threshold or classification can select different observations and produce a different conditional correlation. These are descriptive comparisons, not evidence that the classification itself caused the change.
In a population model, conditional means, variances, and covariance are calculated under the distribution given the regime. In a finite data set, select only the paired observations in that regime and calculate their sample Pearson correlation. The sample calculation uses the subgroup’s own sample means and standard deviations; the common sample-size factors in covariance and variance cancel in the correlation ratio.
A high-loss subset is not automatically a formal measure of asymptotic tail dependence. It is a finite-threshold group defined for a particular analysis. The resulting correlation describes linear association within the selected group, and may depend materially on how that group was selected.
Conditional Calculation and Interpretation
For a specified regime, calculate the conditional means first, then the conditional covariance and variances. Centering each variable on its regime-specific mean is essential. Using the full-data means inside a subgroup calculation does not produce the conditional covariance.
State the event or category precisely, including any loss threshold and whether equality belongs in the high-loss group.
For data, retain the paired observations in the regime. For a probability model, condition on the regime and ensure the conditional probabilities sum to one.
Find each conditional mean, variance, and covariance using the observations or distribution within that regime.
Divide conditional covariance by the product of conditional standard deviations, then compare with other regime correlations and the unconditional result.
Worked Example: A Strong Overall Association but No Within-Regime Association
A hypothetical risk team models eight equally likely paired-loss scenarios, with losses in thousands of dollars. It defines the high-loss regime as \(X\geq 7\). The scenarios are \((1,1),(1,3),(3,1),(3,3),(7,7),(7,9),(9,7),(9,9)\). Calculate the unconditional correlation and the correlation in each regime. Identify the interpretation trap.
Method. Treat the scenarios as a discrete probability distribution, so each has probability \(1/8\). The ordinary regime has the first four pairs; the high-loss regime has the last four.
In the ordinary regime, the conditional means are \(E[X\mid R]=2\) and \(E[Y\mid R]=2\). Each variable has conditional variance \(1\). The four products of centered deviations are \(1,-1,-1,1\), so their average, the conditional covariance, is \(0\). Thus \(\rho_{XY\mid R=\text{ordinary}}=0/\sqrt{1\cdot1}=0\).
In the high-loss regime, each conditional mean is \(8\), and each conditional variance is \(1\). The centered products are again \(1,-1,-1,1\), so the conditional covariance is \(0\) and \(\rho_{XY\mid R=\text{high}}=0\).
Across all eight scenarios, \(E[X]=E[Y]=5\). The mean of \(X^2\) is \(35\), so \(\operatorname{Var}(X)=35-25=10\); by symmetry, \(\operatorname{Var}(Y)=10\). The mean of \(XY\) is \(34\), giving \(\operatorname{Cov}(X,Y)=34-(5)(5)=9\). Therefore, the unconditional correlation is \(9/\sqrt{10\cdot10}=0.9\).
Check. The unconditional correlation is positive and below \(1\), while both subgroup correlations are zero. The regime-specific shifts in the two losses move together, even though there is no linear association within either regime.
Common mistake. Averaging the two subgroup correlations gives \((0+0)/2=0\), but that is not the unconditional correlation: the correct result is \(0.9\). The overall correlation includes association between the regime means as well as association within regimes.
Why Conditional and Unconditional Correlations Differ
The relationship between conditional and unconditional association can be made precise with the law of total covariance. Let \(R\) identify the regime. The overall covariance contains two components: the average covariance within regimes and the covariance between the regime-specific means.
To justify the identity, condition the product \(XY\) on \(R\). By the definition of conditional covariance, \(E[XY\mid R]=\operatorname{Cov}(X,Y\mid R)+E[X\mid R]E[Y\mid R]\). Taking expectations gives \(E[XY]=E[\operatorname{Cov}(X,Y\mid R)]+E[E[X\mid R]E[Y\mid R]]\). Also, the law of total expectation gives \(E[X]=E[E[X\mid R]]\) and \(E[Y]=E[E[Y\mid R]]\). Subtracting \(E[X]E[Y]\) from \(E[XY]\) yields the stated identity, because the remaining difference of the regime-mean product terms is \(\operatorname{Cov}(E[X\mid R],E[Y\mid R])\).
This covariance identity does not say that the unconditional correlation is an average of conditional correlations. A correlation also divides covariance by standard deviations, and the overall variances include both within-regime variation and variation between regime means. Even when conditional correlations are known, the regime probabilities and conditional variances are needed to recover the unconditional correlation.
Worked Example: Opposite Overall and Within-Regime Directions
Consider eight equally likely hypothetical scenarios, again with losses in thousands of dollars. In the ordinary regime, the pairs are \((0,8),(1,9),(3,11),(4,12)\). In the high-loss regime, defined by \(X\geq 5\), they are \((6,0),(7,1),(9,3),(10,4)\). Calculate the correlation in each regime and overall.
Method. Calculate conditional moments within each group, then use total covariance and total variance to combine them. In the ordinary regime, the means are \(2\) and \(10\). Deviations for \(X\) and \(Y\) are both \((-2,-1,1,2)\). The average squared deviation is \((4+1+1+4)/4=2.5\), and the average product of paired deviations is also \(2.5\). Thus each variance is \(2.5\), the covariance is \(2.5\), and the conditional correlation is \(1\).
In the high-loss regime, the means are \(8\) and \(2\), and the deviations of both variables are again \((-2,-1,1,2)\). Consequently, both conditional variances and the conditional covariance are \(2.5\), giving a conditional correlation of \(1\).
The overall means are \(E[X]=(2+8)/2=5\) and \(E[Y]=(10+2)/2=6\). The average within-regime covariance is \(2.5\). The covariance of the regime means is \(\{(2-5)(10-6)+(8-5)(2-6)\}/2=(-12-12)/2=-12\). Therefore, \(\operatorname{Cov}(X,Y)=2.5-12=-9.5\).
For \(X\), the average within-regime variance is \(2.5\), and the variance between regime means is \(\{(2-5)^2+(8-5)^2\}/2=9\), so \(\operatorname{Var}(X)=11.5\). For \(Y\), the within-regime variance is \(2.5\), and the variance between means is \(\{(10-6)^2+(2-6)^2\}/2=16\), so \(\operatorname{Var}(Y)=18.5\). The overall correlation is \(-9.5/\sqrt{11.5\cdot18.5}=-9.5/\sqrt{212.75}\approx-0.6513\).
Check. Both subgroup correlations are positive, while the overall correlation is negative. This is possible because the regime means move in opposite directions: the high-loss group has a larger mean \(X\) but a smaller mean \(Y\). The result is rounded to four decimal places.
Common mistake. Reporting the two within-regime correlations as evidence of a positive overall association ignores the between-regime component. Here that component is \(-12\), large enough to outweigh the average within-regime covariance of \(2.5\).
Conditioning on a Threshold With Unequal Probabilities
When scenarios have unequal probabilities, selecting a regime is not enough: the retained probabilities must be divided by the probability of that regime. This normalization is what makes the subsequent moments conditional moments. The following example shows why using the original probabilities directly can give the wrong answer.
Worked Example: Normalize Scenario Probabilities Before Calculating
A hypothetical joint loss model has the following scenarios, with \(X\) and \(Y\) measured in thousands of dollars: \((1,1)\) with probability \(0.10\), \((1,3)\) with probability \(0.10\), \((3,1)\) with probability \(0.10\), \((3,3)\) with probability \(0.10\), \((8,5)\) with probability \(0.06\), \((8,9)\) with probability \(0.12\), \((12,5)\) with probability \(0.18\), and \((12,9)\) with probability \(0.24\). Define the high-loss regime by \(X\geq8\). Find the ordinary- and high-regime correlations.
Method. The ordinary scenarios have total probability \(0.40\), with equal conditional probabilities \(0.25\) each. Their means are \(2\) and \(2\); both variances are \(1\), and the conditional covariance is \(0\). Thus the ordinary-regime correlation is \(0\).
The high-loss scenarios have total probability \(0.06+0.12+0.18+0.24=0.60\). Dividing each probability by \(0.60\), the conditional weights are \(0.10,0.20,0.30,0.40\), in the order listed. Hence \(E[X\mid H]=8(0.10+0.20)+12(0.30+0.40)=10.8\) and \(E[Y\mid H]=5(0.10+0.30)+9(0.20+0.40)=7.4\).
The second moments are \(E[X^2\mid H]=64(0.30)+144(0.70)=120\) and \(E[Y^2\mid H]=25(0.40)+81(0.60)=58.6\). Therefore the conditional variances are \(120-10.8^2=3.36\) and \(58.6-7.4^2=3.84\).
The conditional product moment is \(E[XY\mid H]=40(0.10)+72(0.20)+60(0.30)+108(0.40)=79.6\). Thus the conditional covariance is \(79.6-(10.8)(7.4)=-0.32\), and the high-regime correlation is \(-0.32/\sqrt{3.36\cdot3.84}\approx-0.0891\).
Check. The high-regime covariance is slightly negative, and both conditional variances are positive. The correlation is rounded to four decimal places.
Common mistake. The probabilities \(0.06,0.12,0.18,0.24\) are unconditional, not high-regime conditional weights. Using them as if they summed to one would not calculate moments given \(H\); first divide by \(P(H)=0.60\).
Practical Interpretation and Exam Traps
Conditional correlations can help describe whether an association persists in a specified loss regime, weakens, strengthens, or changes sign. For risk management, that comparison can flag that a single correlation may not describe every operating condition. It does not, on its own, establish a mechanism for the change or determine a capital requirement.
- State the selection rule. “High losses” is not precise enough unless the event or category is defined. Record the threshold and which variable or combined criterion it uses.
- Use subgroup means. Centering observations on overall means when calculating a subgroup covariance mixes within-regime and between-regime effects.
- Do not average correlations to obtain the overall result. The total covariance and total variances depend on regime probabilities and regime means as well as within-regime moments.
- Check that the correlation exists. If either loss is constant in a selected subset, its conditional variance is zero and the Pearson correlation in that subset is undefined.
- Interpret the selected population. Conditioning on a threshold narrows or otherwise changes the data being described. A different threshold can produce a different result, so report the selection rule alongside the correlation.
A sound analysis reports the unconditional association and each relevant conditional association, identifies the regimes and their probabilities or sample sizes, and explains what each measure describes. Correctly calculated regime correlations are one diagnostic of dependence in loss data; their use here is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Check Your Understanding
For each problem, use the specified regime and conditional moments. Treat probabilities as population probabilities, not sample frequencies.
- A regime contains four equally likely pairs: \((2,4),(2,8),(6,4),(6,8)\). Find the conditional Pearson correlation. (Answer: \(0.0000\))
- In a high-loss regime, the four equally likely pairs are \((1,2),(1,6),(5,2),(5,6)\). Calculate the conditional covariance and correlation. (Answer: Covariance \(=0\); correlation \(=0.0000\))
- Two equally likely regimes each have conditional correlation \(1\). Their conditional means are \((E[X\mid R],E[Y\mid R])=(2,10)\) and \((10,6)\). In each regime both variances are \(8/3\). Find the unconditional covariance and correlation. (Answer: Covariance \(=-16/3\); correlation \(\approx-0.4781\))
- A regime has probability \(0.25\), conditional means \((12,2)\), and conditional covariance \(-1\). The other regime has probability \(0.75\), means \((4,10)\), and covariance \(3\). Calculate the unconditional covariance. (Answer: \(-10\))
- In a selected high-loss subset, every observation has the same \(X\) value, while \(Y\) varies. Is the conditional Pearson correlation defined? State why. (Answer: No; \(\operatorname{Var}(X\mid H)=0\), so the correlation denominator is zero.)