Hidden Categories And Conditional Expectations

Why Hidden Categories Change an Expected Value

A model may distinguish groups that are not directly observed, such as equipment types, risk classes, or operating conditions. These groups are latent categories: the category affects the distribution of a quantity of interest, but the category itself may be unknown when evidence is observed. If evidence makes some categories more plausible than others, the relevant expected value must use the updated category weights.

The calculation has two parts. First, determine the expected value within each category, using the information available. Second, weight those means by the probabilities of the categories under the conditioning information. The weights are not automatically the original category probabilities: evidence can change them.

Definition: Let $C_1,\ldots,C_k$ be mutually exclusive and exhaustive events, with $P(C_j)>0$. They form a category partition: exactly one category occurs. For a random variable $X$ with finite expectations, $E[X\mid C_j]$ is its mean within category $C_j$. If $B$ is evidence with $P(B)>0$, then $E[X\mid B,C_j]$ is the mean when both $B$ and category $C_j$ occur.

Conditioning on Evidence and Category

When evidence $B$ is observed, the category weights become $P(C_j\mid B)$. For categories with $P(B\mid C_j)>0$, Bayes' theorem gives $P(C_j\mid B)=P(C_j)P(B\mid C_j)/P(B)$. The denominator is found by adding the category-specific joint probabilities: $P(B)=\sum_{j=1}^k P(C_j)P(B\mid C_j)$.

Once these updated weights and the category-specific conditional means are known, the conditional mean is their weighted average. Importantly, this formula does not require $X$ and $B$ to be independent within a category. If the evidence is related to $X$, that relationship must be reflected in $E[X\mid B,C_j]$, rather than ignored.

Theorem: For a category partition $C_1,\ldots,C_k$ and evidence $B$ with $P(B)>0$, $$ E[X\mid B]=\sum_{j=1}^k P(C_j\mid B)E[X\mid B,C_j]. $$ In particular, without additional evidence, $$ E[X]=\sum_{j=1}^k P(C_j)E[X\mid C_j]. $$ These identities apply when the relevant expectations exist. If a category has zero probability given $B$, its term contributes zero and its conditional mean need not be defined.

The reason is that, once $B$ is known, the category events $C_j$ still divide the possible outcomes into disjoint cases. Within each case, $X$ has mean $E[X\mid B,C_j]$; the probability of that case given $B$ is $P(C_j\mid B)$. Adding the case contributions gives the mean over all outcomes consistent with $B$.

E[X\mid B] =\sum_{j=1}^k E[X\,\mathbf{1}_{C_j}\mid B] =\sum_{j=1}^k P(C_j\mid B)E[X\mid B,C_j].

For the second equality, within the event $B$, the indicator $\mathbf{1}_{C_j}$ is one in category $C_j$ and zero otherwise. Thus $E[X\mathbf{1}_{C_j}\mid B]=P(C_j\mid B)E[X\mid B,C_j]$. Taking $B$ to be the whole sample space gives the unconditional formula.

List the categories.
Check that the categories are mutually exclusive and cover all possibilities.

Find the relevant weights.
For a conditional mean, calculate $P(C_j\mid B)$, using Bayes' theorem if needed. For an unconditional mean, use $P(C_j)$.

Find each category-specific mean.
Use the distribution within that category and respect any conditioning on the evidence.

Weight and add.
Multiply each mean by its matching category probability and sum the products.

Worked Examples

Worked Example: Updating Category Weights After a Sensor Alert

A hypothetical fleet has three equipment types, $C_1,C_2,C_3$, with probabilities $0.50,0.30,0.20$. A sensor alert $B$ occurs with probabilities $0.40,0.20,0.50$ in the respective categories. Given both the alert and category, expected repair time $X$, in hours, is $10,25,50$, respectively. Find $E[X\mid B]$. State the method and identify a common mistake.

Method. Find the joint probabilities of category and alert, convert them to category probabilities given the alert, and weight the conditional repair-time means by those updated probabilities.

The joint probabilities are $P(C_1\cap B)=0.50(0.40)=0.20$, $P(C_2\cap B)=0.30(0.20)=0.06$, and $P(C_3\cap B)=0.20(0.50)=0.10$. Therefore, $P(B)=0.20+0.06+0.10=0.36$.

The category probabilities after the alert are $P(C_1\mid B)=0.20/0.36=0.5556$, $P(C_2\mid B)=0.06/0.36=0.1667$, and $P(C_3\mid B)=0.10/0.36=0.2778$, to four decimal places.

Now weight the category-specific means by these probabilities. Equivalently, use the joint probabilities directly in the numerator:

E[X\mid B] =\frac{0.20(10)+0.06(25)+0.10(50)}{0.36} =\frac{2+1.5+5}{0.36} =\frac{8.5}{0.36} =23.6111\text{ hours}.

The updated-weight calculation checks the result: $0.5556(10)+0.1667(25)+0.2778(50)\approx5.556+4.168+13.890=23.614$, with the small difference caused by rounding the weights. Using the exact fractions gives $23.6111$ hours.

Common mistake. Weighting the three means by the original probabilities gives $0.50(10)+0.30(25)+0.20(50)=22.5$ hours. That is not $E[X\mid B]$: it ignores that the alert changes the category proportions. The corrected weights are $P(C_j\mid B)$, not $P(C_j)$.

Check. The category probabilities given the alert sum to one before rounding, and the mean lies between the smallest and largest category-specific means, as expected.

Worked Example: Evidence Defined by the Random Variable

A hypothetical sorting system handles parcels under one of two latent operating conditions, $C_1$ and $C_2$, with probabilities $0.60$ and $0.40$. The number $X$ of parcels delayed in a batch has the following conditional distributions:

Category	$P(X=0\mid C_j)$	$P(X=2\mid C_j)$	$P(X=4\mid C_j)$
$C_1$	0.20	0.50	0.30
$C_2$	0.50	0.30	0.20

An alert is generated when $B=\{X\geq2\}$. Find $E[X\mid B]$. In particular, do not treat the alert as independent of $X$.

Method. Within each category, calculate both the probability of the alert and the alert-weighted mean of $X$. Then combine the categories using their joint probabilities with the alert.

In category $C_1$, $P(B\mid C_1)=0.50+0.30=0.80$. The contribution to $E[X\mathbf{1}_B\mid C_1]$ is $2(0.50)+4(0.30)=1.00+1.20=2.20$. Thus $E[X\mid B,C_1]=2.20/0.80=2.75$.

In category $C_2$, $P(B\mid C_2)=0.30+0.20=0.50$. Its alert-weighted mean is $2(0.30)+4(0.20)=0.60+0.80=1.40$, so $E[X\mid B,C_2]=1.40/0.50=2.80$.

The joint probabilities of category and alert are $P(C_1\cap B)=0.60(0.80)=0.48$ and $P(C_2\cap B)=0.40(0.50)=0.20$. Hence $P(B)=0.48+0.20=0.68$, and $P(C_1\mid B)=0.48/0.68$, $P(C_2\mid B)=0.20/0.68$.

E[X\mid B] =\frac{0.48(2.75)+0.20(2.80)}{0.68} =\frac{1.32+0.56}{0.68} =\frac{1.88}{0.68} =2.7647.

The same numerator can be checked directly: $0.60(2.20)+0.40(1.40)=1.32+0.56=1.88$. The result is between 2 and 4, the only possible values of $X$ when the alert occurs. The key point is that the alert depends on $X$; using the unconditional mean of $X$ in each category would not give the conditional category means.

Worked Example: Averaging a Hypothetical Limited Payment Across Categories

A hypothetical warranty pays $Y=\min((X-100)^+,400)$ dollars for a covered loss $X$, where $(z)^+=\max(z,0)$. A claim comes from category $A$ with probability $0.80$ and category $B$ with probability $0.20$. The loss distributions are:

Category	Possible losses $X$ and their conditional probabilities
$A$	$0$ with probability $0.25$; $100$ with probability $0.50$; $500$ with probability $0.25$
$B$	$0$ with probability $0.20$; $300$ with probability $0.50$; $800$ with probability $0.30$

Find the unconditional expected payment. The stated deductible is 100 dollars and the maximum payment is 400 dollars; apply both to the loss before averaging.

Method. Calculate the payment at each possible loss, find its mean within each category, and then weight those two means by the category probabilities.

For category $A$, the payments at losses $0,100,500$ are $0,0,400$, respectively. Therefore, $E[Y\mid A]=0.25(0)+0.50(0)+0.25(400)=100$ dollars.

For category $B$, the payments at losses $0,300,800$ are $0,200,400$, respectively: the payment at 800 is capped at 400. Thus, $E[Y\mid B]=0.20(0)+0.50(200)+0.30(400)=0+100+120=220$ dollars.

E[Y]=P(A)E[Y\mid A]+P(B)E[Y\mid B] =0.80(100)+0.20(220) =80+44 =124\text{ dollars}.

The category probabilities sum to one, and the overall mean, 124 dollars, lies between the category-specific means of 100 and 220 dollars. Weighting the loss means first and applying one deductible to that average would be incorrect: the payment is a function of each loss, so it must be evaluated within each category before the category means are combined.

Common Traps and Practical Checks

The main challenge is usually not the final multiplication; it is identifying which probabilities and means belong together. Use the following checks before accepting a result:

Match the conditioning. Pair $P(C_j\mid B)$ with $E[X\mid B,C_j]$. Do not pair updated weights with unconditional category means unless the evidence does not change those means within categories.
Use the evidence to update weights. When the question asks for $E[X\mid B]$, the original category probabilities are generally not the appropriate weights.
Do not assume conditional independence. If $B$ is defined using $X$, or otherwise depends on it, calculate $E[X\mid B,C_j]$ from the conditional distribution rather than substituting $E[X\mid C_j]$.
Check that the weights sum to one. The probabilities $P(C_j\mid B)$ should add to one, apart from rounding.
Check the range. A weighted mean with nonnegative weights summing to one must lie between the smallest and largest category-specific means being averaged.

This method is useful whenever a model has an unobserved group that affects the distribution of a quantity: selecting an expected claim amount after a signal, averaging process output over operating conditions, or finding an expected payment across risk classes. It is aligned with the SOA Exam P (Probability) syllabus because it combines conditional probability, Bayes' theorem, and expectation in one calculation.

Key takeaway: A mean across hidden categories is a weighted average of category-specific means. For an unconditional mean, use the category probabilities; after observing evidence, use the posterior category probabilities and means conditional on that evidence.

Check Your Understanding

For each problem, identify the relevant category weights and conditional means before combining them.

A device belongs to category $C_1,C_2,C_3$ with probabilities $0.20,0.50,0.30$. The mean number of faults is $4,10,16$, respectively. Find the unconditional mean number of faults. (Answer: 10.6)
A sensor test $B$ occurs with probabilities $0.10$ and $0.60$ in categories $A$ and $D$, whose prior probabilities are $0.70$ and $0.30$. Given both $B$ and the category, mean service time is 8 hours in $A$ and 20 hours in $D$. Find $E[X\mid B]$. (Answer: 16.6400 hours)
A batch is from category $C_1$ with probability $0.40$ and $C_2$ with probability $0.60$. In $C_1$, $X$ is 0, 5, or 10 with probabilities $0.20,0.50,0.30$; in $C_2$, those probabilities are $0.50,0.30,0.20$. An alert occurs when $X\geq5$. Find $E[X\mid X\geq5]$. (Answer: 6.9355)
A hypothetical payment is $Y=\min((X-100)^+,400)$. Category $A$ has probability $0.80$, with losses $0,100,500$ of probabilities $0.25,0.50,0.25$; category $B$ has probability $0.20$, with losses $0,300,800$ of probabilities $0.20,0.50,0.30$. Find $E[Y]$. (Answer: $124.00)
Three latent categories have prior probabilities $0.20,0.50,0.30$. Evidence $B$ occurs with probabilities $0.30,0.40,0.80$ in the categories, and the respective means $E[X\mid B,C_j]$ are $5,12,30$. Find $P(B)$ and $E[X\mid B]$. (Answer: $P(B)=0.5000$, $E[X\mid B]=19.8000$)

Category	Possible losses \(X\) and their conditional probabilities
\(A\)	\(0\) with probability \(0.25\); \(100\) with probability \(0.50\); \(500\) with probability \(0.25\)
\(B\)	\(0\) with probability \(0.20\); \(300\) with probability \(0.50\); \(800\) with probability \(0.30\)

Category	\(P(X=0\mid C_j)\)	\(P(X=2\mid C_j)\)	\(P(X=4\mid C_j)\)
\(C_1\)	0.20	0.50	0.30
\(C_2\)	0.50	0.30	0.20

What You'll Learn