Computing Jensen Bounds for Distributions

From Jensen’s Inequality to a Calculation

Jensen’s inequality becomes useful in a concrete problem only after two quantities have been calculated: the average value of the transformed variable, $E[\phi(X)]$, and the transformed average, $\phi(E[X])$. These are generally different. For convex $\phi$, Jensen’s inequality says that the first cannot be smaller than the second, provided the expectations exist and the mean lies in the domain of $\phi$.

The order of operations matters. To find $E[\phi(X)]$, first evaluate $\phi$ at each possible value of $X$, then average those transformed values using the distribution of $X$. To find $\phi(E[X])$, first compute the mean of $X$, then apply $\phi$ once.

Definition: Let $X$ be an integrable random variable taking values in an interval $I$, and let $\phi:I\to\mathbb{R}$ be convex. If $E[|\phi(X)|]<\infty$, Jensen’s inequality gives $$ \phi(E[X])\leq E[\phi(X)]. $$ The difference $E[\phi(X)]-\phi(E[X])$ is called the Jensen gap. Under these conditions it is nonnegative.

A practical calculation therefore has three checks: identify the distribution of $X$, confirm that $\phi$ is convex on the values $X$ can take, and verify that both quantities are finite. For a twice-differentiable function on an interval, $\phi''(x)\geq 0$ throughout that interval is a convenient way to verify convexity.

Finite-Support Distributions

If $X$ takes finitely many values $x_1,\ldots,x_k$, with probabilities $p_1,\ldots,p_k$, then the probabilities are nonnegative and sum to one. The mean is a probability-weighted average of the support points. The transformed expectation is a weighted average of the transformed support points.

Result: For a finite-support random variable, $$ E[X]=\sum_{i=1}^{k}p_i x_i, \qquad E[\phi(X)]=\sum_{i=1}^{k}p_i\phi(x_i). $$ Thus the Jensen comparison is $$ \phi\!\left(\sum_{i=1}^{k}p_i x_i\right) \leq \sum_{i=1}^{k}p_i\phi(x_i), $$ when $\phi$ is convex on an interval containing the support.

These expectation formulas follow directly from the definition of expectation for a discrete random variable. The event $X=x_i$ has probability $p_i$; the random variable $\phi(X)$ takes value $\phi(x_i)$ on that event. Summing value times probability gives the two formulas. The inequality is Jensen’s inequality applied to this distribution. This gives a useful computational rule: do not average the support points after transforming them, or transform the support points after averaging them, by mistake. Calculate both sides separately.

Worked Example: A Bernoulli Variable and a Quadratic Function

Let $X\sim\operatorname{Bernoulli}(1/3)$, so $P(X=1)=1/3$ and $P(X=0)=2/3$. Take $\phi(x)=x^2$, which is convex because $\phi''(x)=2$ for every real $x$. Compute the Jensen comparison.

First calculate the mean: $$ E[X]=0\left(\frac{2}{3}\right)+1\left(\frac{1}{3}\right)=\frac{1}{3}. $$

Applying $\phi$ to this mean gives $$ \phi(E[X])=\left(\frac{1}{3}\right)^2=\frac{1}{9}. $$

For the other side, transform each possible value and then average: $$ E[\phi(X)] =0^2\left(\frac{2}{3}\right)+1^2\left(\frac{1}{3}\right) =\frac{1}{3}. $$

The comparison is $1/9\leq 1/3$, and the gap is $$ E[\phi(X)]-\phi(E[X])=\frac{1}{3}-\frac{1}{9}=\frac{2}{9}. $$

Worked Example: Three Support Points

Suppose $X$ takes the values $-1,1,3$ with probabilities $1/4,1/2,1/4$, respectively. These probabilities sum to $1$. Again let $\phi(x)=x^2$, convex on the real line.

The mean is $$ E[X]=(-1)\left(\frac{1}{4}\right)+1\left(\frac{1}{2}\right)+3\left(\frac{1}{4}\right) =-\frac{1}{4}+\frac{1}{2}+\frac{3}{4}=1. $$ Hence $\phi(E[X])=1^2=1$. Separately, the expectation of the transformed variable is $$ E[\phi(X)] =(-1)^2\left(\frac{1}{4}\right)+1^2\left(\frac{1}{2}\right)+3^2\left(\frac{1}{4}\right) =\frac{1}{4}+\frac{1}{2}+\frac{9}{4}=3. $$

Thus Jensen’s inequality reads $1\leq 3$, with gap $3-1=2$. Notice that the negative support point contributes positively to $E[X^2]$: squaring changes its value from $-1$ to $1$ before the probability-weighted average is taken.

Continuous Distributions

When $X$ has a density $f$, expectations are computed by integration. If the relevant integrals are finite, the mean is the integral of $x f(x)$, and the transformed expectation is the integral of $\phi(x)f(x)$. The density weights each value according to how probability is distributed over the support.

Result: If $X$ has density $f$ on a support $S$, and the indicated integrals exist, then $$ E[X]=\int_S x f(x)\,dx, \qquad E[\phi(X)]=\int_S \phi(x)f(x)\,dx. $$ For convex $\phi$, the Jensen comparison is $\phi\!\left(\int_S x f(x)\,dx\right)\leq \int_S\phi(x)f(x)\,dx$, under the integrability conditions stated above.

The density formula is the continuous expectation formula applied twice: once to the function $g(x)=x$, and once to $g(x)=\phi(x)$. In an actual calculation, write the support and density explicitly, compute $E[X]$, and then evaluate the separate integral for $E[\phi(X)]$. Keeping those steps separate makes the comparison easy to check.

Worked Example: A Uniform Distribution on an Interval

Let $X\sim\operatorname{Uniform}(0,2)$, whose density is $f(x)=1/2$ for $0<x<2$. Let $\phi(x)=x^2$, which is convex on this support. The mean is $$ E[X]=\int_0^2 x\left(\frac{1}{2}\right)\,dx =\frac{1}{2}\left[\frac{x^2}{2}\right]_0^2 =1. $$ Therefore, $\phi(E[X])=1^2=1$.

Now calculate the expectation after transforming: $$ E[\phi(X)] =\int_0^2 x^2\left(\frac{1}{2}\right)\,dx =\frac{1}{2}\left[\frac{x^3}{3}\right]_0^2 =\frac{1}{2}\cdot\frac{8}{3} =\frac{4}{3}. $$

The bound is $1\leq 4/3$, and the Jensen gap is $4/3-1=1/3$. Both the density and the integration limits matter: the factor $1/2$ is the density, not an optional normalizing adjustment.

Worked Example: A Fourth-Power Function

Let $X\sim\operatorname{Uniform}(-1,1)$, with density $f(x)=1/2$ on $(-1,1)$, and choose $\phi(x)=x^4$. The function is convex on the real line because $\phi''(x)=12x^2\geq 0$.

By symmetry, or by direct integration, $$ E[X]=\int_{-1}^{1}x\left(\frac{1}{2}\right)\,dx=0, $$ so $\phi(E[X])=0^4=0$. The other quantity is $$ E[\phi(X)] =\int_{-1}^{1}x^4\left(\frac{1}{2}\right)\,dx =\frac{1}{2}\left[\frac{x^5}{5}\right]_{-1}^{1} =\frac{1}{2}\left(\frac{1}{5}-\left(-\frac{1}{5}\right)\right) =\frac{1}{5}. $$

Consequently $0\leq 1/5$, with Jensen gap $1/5$. This example also illustrates why a symmetric distribution need not make $E[\phi(X)]$ zero: symmetry makes the mean zero here, while the even function $x^4$ has nonnegative values on both sides of zero.

A Useful Check: The Quadratic Jensen Gap

For $\phi(x)=x^2$, the gap has a familiar form whenever $E[X^2]<\infty$. Since $\operatorname{Var}(X)=E[(X-E[X])^2]$, expanding the square gives

\operatorname{Var}(X) =E[X^2]-2E[X]E[X]+(E[X])^2 =E[X^2]-(E[X])^2.

Here we used linearity of expectation and the fact that $E[X]$ is a constant. Thus

E[X^2]-(E[X])^2=\operatorname{Var}(X)\geq 0.

This is a complete derivation of the quadratic Jensen comparison: $E[\phi(X)]-(\phi(E[X]))=E[X^2]-(E[X])^2=\operatorname{Var}(X)$, so the inequality follows from nonnegativity of variance. It is also a useful arithmetic check. If a calculation with $\phi(x)=x^2$ produces a negative gap, at least one of the mean, second moment, or subtraction has been computed incorrectly. $\square$

Common Mistakes and a Reliable Workflow

A Jensen bound compares a transformed mean with a mean of transformed values. It does not say that every individual value of $\phi(X)$ is above $\phi(E[X])$, nor does it say that $\phi(E[X])$ is an upper bound for $E[\phi(X)]$. Check the direction using convexity: the function evaluated at the mean is the lower quantity.

Reversing the order of operations. $E[\phi(X)]$ means transform first and average second. It is not $\phi(E[X])$.
Forgetting probabilities or the density. Discrete values must be probability-weighted; continuous values must be integrated against the density.
Using a function that is not convex on the relevant interval. Convexity must hold over the values $X$ can take, not merely at the mean.
Ignoring existence conditions. Check that $E[X]$ is finite and that $E[|\phi(X)|]$ is finite before treating the comparison as one between finite numbers.

A reliable calculation can be organized into four steps:

Record the law.
Write the support and probabilities, or the support and density, so the weights in the calculation are explicit.

Check the function.
Verify that $\phi$ is convex on an interval containing the support and check that the required expectations are finite.

Compute both sides separately.
Find $E[X]$ and then $\phi(E[X])$; also find $E[\phi(X)]$ by a weighted sum or density integral.

Compare and check.
Confirm that $\phi(E[X])\leq E[\phi(X)]$, and subtract to obtain the Jensen gap.

Jensen calculations are valuable because they replace a potentially difficult expectation with a bound expressed in terms of the mean. Even when the expectation can be evaluated exactly, calculating both sides reveals how the distribution and the curvature of the function affect the gap. The quadratic case is especially transparent: its gap is exactly the variance.

Check Your Understanding

For each question, compute the requested quantities rather than relying only on the inequality.

Let $X$ equal $0$ with probability $3/4$ and $2$ with probability $1/4$. For $\phi(x)=x^2$, find $E[X]$, $\phi(E[X])$, $E[\phi(X)]$, and the gap.
For $X\sim\operatorname{Uniform}(0,1)$ and $\phi(x)=x^2$, set up and evaluate both sides of Jensen’s inequality.
For a discrete random variable with support $x_i$ and probabilities $p_i$, explain which values receive weights $p_i$ in the calculation of $E[\phi(X)]$.
Why does the identity $E[X^2]-(E[X])^2=\operatorname{Var}(X)$ verify the Jensen comparison for $\phi(x)=x^2$?
Name two checks, other than evaluating the two sides, that should be made before applying Jensen’s inequality.