Conditional Means Under Truncation

Why Truncation Changes a Mean

A conditional mean such as $E[X\mid X>a]$ describes the average value among outcomes that exceed a threshold. It is generally not the original mean: the threshold removes part of the distribution and changes the relative weights of the remaining values. This calculation appears whenever attention is restricted to observations above a detection limit, individuals meeting an eligibility criterion, or outcomes within a specified range.

The basic operation is to keep the values in the conditioning event, weight them by their original probabilities, and divide by the probability of that event. The resulting formulas work for discrete, continuous, and mixed distributions, provided the conditioning event has positive probability and the relevant expectation exists.

Definition: Let $X$ be an integrable random variable and let $A$ be an event with $P(A)>0$. The conditional mean of $X$ given $A$ is $$ E[X\mid A]=\frac{E[X\,1_A]}{P(A)}, $$ where $1_A$ is 1 on $A$ and 0 otherwise. This is a scalar mean within the event $A$, not the random variable $E[X\mid \mathcal{G}]$ associated with a sigma-field.

For a threshold event $A=\{X>a\}$, the numerator is the first moment contributed by values above $a$, and the denominator is the upper-tail probability. If $X$ is discrete with mass function $f$, then

E[X\mid X>a] = \frac{\displaystyle\sum_{x>a} x f(x)} {\displaystyle\sum_{x>a} f(x)}, \qquad P(X>a)>0.

For an absolutely continuous $X$ with density $f$, the corresponding formula is

E[X\mid X>a] = \frac{\displaystyle\int_a^\infty x f(x)\,dx} {\displaystyle\int_a^\infty f(x)\,dx}, \qquad P(X>a)>0.

In both formulas, the conditional mean is finite when $E[|X|1_{\{X>a\}}]<\infty$. When $X$ has atoms, the event notation matters: $X>a$ excludes an atom at $a$, whereas $X\geq a$ includes it.

The Truncated Distribution

Conditioning on an event defined by $X$ produces a distribution restricted to that event and renormalized to have total probability one. This gives a direct route to conditional means and to other conditional moments.

Theorem: Let $A=\{X\in C\}$ for a measurable set $C\subseteq\mathbb{R}$, with $P(X\in C)>0$. For any measurable set $B$, $$ P(X\in B\mid X\in C) = \frac{P(X\in B\cap C)}{P(X\in C)}. $$ If $X$ has density $f$, its conditional density on $C$ is $$ f_C(x)=\frac{f(x)1_C(x)}{P(X\in C)}. $$ If $X$ is discrete, its conditional mass function is $f_C(x)=f(x)/P(X\in C)$ for $x\in C$, and $0$ otherwise.

To verify the density statement, integrate it over any measurable set $B$:

\int_B f_C(x)\,dx = \frac{\int_{B\cap C} f(x)\,dx}{P(X\in C)} = \frac{P(X\in B\cap C)}{P(X\in C)} = P(X\in B\mid X\in C).

The function $f_C$ is a valid density because it is nonnegative and its integral over the real line is $P(X\in C)/P(X\in C)=1$. The mass-function assertion follows by the same argument with sums in place of integrals. Therefore, when the conditional mean is finite, integrating $x$ against this truncated distribution gives $E[X\mid X\in C]$.

Worked Example: An Exponential Upper-Tail Mean

Let $X\sim\operatorname{Exponential}(2)$, with density $f(x)=2e^{-2x}$ for $x\geq 0$, and find $E[X\mid X>1]$.

The probability of exceeding 1 is $$ P(X>1)=\int_1^\infty 2e^{-2x}\,dx=e^{-2}. $$ The truncated first moment is $$ \int_1^\infty x\,2e^{-2x}\,dx. $$ Integration by parts, with $u=x$ and $dv=2e^{-2x}dx$, gives $$ \int_1^\infty x\,2e^{-2x}\,dx = \left[-xe^{-2x}\right]_1^\infty+\int_1^\infty e^{-2x}\,dx = e^{-2}+\frac{e^{-2}}{2} = \frac{3}{2}e^{-2}. $$ Thus $$ E[X\mid X>1] = \frac{(3/2)e^{-2}}{e^{-2}} = \frac{3}{2}. $$ The conditional mean is above the threshold, as it must be, and exceeds it by $1/2$.

A Tail-Probability Formula

An upper-tail mean can also be expressed using the survival probabilities $P(X>t)$, avoiding a direct calculation of the truncated first moment. The relevant quantity is the expected excess above the threshold, $X-a$, among values that exceed $a$.

Theorem: Let $a\in\mathbb{R}$, suppose $P(X>a)>0$, and assume $E[(X-a)1_{\{X>a\}}]<\infty$. Then $$ E[X\mid X>a] = a+\frac{\displaystyle\int_a^\infty P(X>t)\,dt}{P(X>a)}. $$

Proof. On the event $X>a$, the excess $X-a$ is nonnegative. For any nonnegative random variable $Y$, the layer-cake identity $E[Y]=\int_0^\infty P(Y>u)\,du$ follows by writing $Y=\int_0^\infty 1_{\{Y>u\}}du$ and applying Tonelli's theorem to the nonnegative integrand. Apply this identity to the conditional distribution of $X-a$ given $X>a$:

E[X-a\mid X>a] = \int_0^\infty P(X-a>u\mid X>a)\,du.

For $u\geq0$, the event $\{X-a>u\}$ is $\{X>a+u\}$, which is contained in $\{X>a\}$. Hence

P(X-a>u\mid X>a) = \frac{P(X>a+u)}{P(X>a)}.

Substitute this expression into the integral and set $t=a+u$. The integral is finite by the stated assumption, so

E[X-a\mid X>a] = \frac{1}{P(X>a)}\int_a^\infty P(X>t)\,dt.

Adding $a$ proves the formula. $\square$

Worked Example: A Normal Mean Above One Standard Deviation

Let $Z\sim N(0,1)$, with density $\phi(z)=(2\pi)^{-1/2}e^{-z^2/2}$ and distribution function $\Phi$. Find $E[Z\mid Z>1]$.

The upper-tail probability is $1-\Phi(1)$. Since $\phi'(z)=-z\phi(z)$, the numerator is $$ \int_1^\infty z\phi(z)\,dz = \left[-\phi(z)\right]_1^\infty = \phi(1). $$ Therefore $$ E[Z\mid Z>1] = \frac{\phi(1)}{1-\Phi(1)} \approx \frac{0.24197}{0.15866} \approx 1.525. $$ The result is larger than 1, as expected: values just above 1 and the less frequent, larger values farther into the tail contribute to the conditional average.

The tail formula expresses this same calculation in terms of the survival function. In particular, the expected excess over $a$, conditional on exceeding $a$, is the area under the survival curve beyond $a$, divided by the probability of crossing $a$. This also shows why the formula requires a finite tail integral when a finite conditional mean is desired.

Conditioning on an Interval

For an interval event, the same renormalization applies. With a continuous distribution, the choice of including or excluding either endpoint does not affect the result, because individual points have probability zero. With a discrete or mixed distribution, endpoint choices can change both the event probability and the conditional mean.

E[X\mid \ell<X\leq u] = \frac{\displaystyle\int_\ell^u x f(x)\,dx} {\displaystyle\int_\ell^u f(x)\,dx}, \qquad P(\ell<X\leq u)>0, $$ $$ \text{for a density } f;\qquad E[X\mid \ell<X\leq u] = \frac{\displaystyle\sum_{\ell<x\leq u}x f(x)} {\displaystyle\sum_{\ell<x\leq u} f(x)}, \qquad \text{for a mass function } f.

Worked Example: A Standard Normal Mean Within an Interval

Let $Z\sim N(0,1)$. Find $E[Z\mid -1<Z\leq2]$. Because $Z$ has a density, including or excluding either endpoint makes no difference.

The interval probability is $$ P(-1<Z\leq2)=\Phi(2)-\Phi(-1) \approx 0.97725-0.15866 =0.81859. $$ Using $\phi'(z)=-z\phi(z)$, the truncated first moment is $$ \int_{-1}^{2} z\phi(z)\,dz = \left[-\phi(z)\right]_{-1}^{2} = \phi(-1)-\phi(2) \approx 0.24197-0.05399 =0.18798. $$ Consequently, $$ E[Z\mid -1<Z\leq2] = \frac{0.18798}{0.81859} \approx 0.230. $$ Although the interval contains values on both sides of zero, its positive portion extends farther, so the conditional mean is positive.

Worked Example: A Poisson Mean Above a Threshold

Let $X\sim\operatorname{Poisson}(2)$, so $P(X=k)=e^{-2}2^k/k!$ for integers $k\geq0$. Find $E[X\mid X>1]$.

The event $X>1$ is $X\geq2$, and its probability is $$ P(X>1)=1-P(X=0)-P(X=1)=1-3e^{-2}\approx0.59399. $$ For the numerator, the identity $kP(X=k)=2P(X=k-1)$ holds for $k\geq1$, since $$ k e^{-2}\frac{2^k}{k!} = 2e^{-2}\frac{2^{k-1}}{(k-1)!}. $$ It follows that $$ E[X1_{\{X>1\}}] = \sum_{k=2}^{\infty} kP(X=k) = 2\sum_{j=1}^{\infty}P(X=j) = 2(1-e^{-2}). $$ Therefore $$ E[X\mid X>1] = \frac{2(1-e^{-2})}{1-3e^{-2}} \approx \frac{1.72933}{0.59399} \approx2.911. $$ The conditional mean is not obtained by simply rounding the threshold up: it averages all Poisson values from 2 onward, with their conditional probabilities.

Endpoint Choices and a Common Pitfall

The denominator must be the probability of precisely the event used for conditioning. If there is an atom at a cutoff $a$, then $P(X\geq a)=P(X>a)+P(X=a)$, and the numerator must likewise include $aP(X=a)$ when the event is $X\geq a$. For a continuous random variable, $P(X=a)=0$, so the distinction disappears; for a discrete variable, it may be decisive.

Wrong reasoning: “The values above $a$ have been selected, so their mean is the original mean plus $a$.”
Correction: The original mean includes values below the threshold, while the selected mean uses only the truncated distribution. Compute the truncated first moment and divide by the probability of the selected event, or use the tail-probability formula when its conditions hold.

A useful check is that $E[X\mid X>a]\geq a$ whenever the conditional mean exists, because every retained value exceeds $a$. For an interval $\ell<X\leq u$, the conditional mean must lie between $\ell$ and $u$. These checks can reveal an incorrect event, numerator, or denominator, but they do not replace the calculation.

Key takeaway: A conditional mean under truncation is the mean of the renormalized distribution that remains after restricting to the event. For upper tails, it can also be written as the threshold plus a survival-probability integral divided by the tail probability.

Check Your Understanding

For each question, identify the conditioning event precisely before calculating.

State the density of $X$ conditional on $X>a$ when $X$ has density $f$ and $P(X>a)>0$.
Why does the tail-probability formula use $\int_a^\infty P(X>t)\,dt$ to describe the conditional excess above $a$?
For a discrete random variable with positive mass at $a$, explain why conditioning on $X\geq a$ can give a different mean from conditioning on $X>a$.
Write the numerator and denominator needed for $E[X\mid \ell<X\leq u]$ when $X$ has mass function $f$.
What inequality should $E[X\mid X>a]$ satisfy when it is finite, and why?