Overlaying Grouped Density Curves

Why Overlay Density Curves?

A density curve can summarize the distribution of a quantitative variable more smoothly than a histogram. Overlaying curves for several groups makes it possible to compare their locations, spreads, peaks, and tails in one view. The comparison is useful only when the curves use a common horizontal scale and comparable smoothing settings. Otherwise, a visual difference may come from the display choices rather than from the observed groups.

A density curve describes the distribution within a group. It does not show how many observations belong to that group. This distinction matters when group sample sizes differ: a group with many observations can have a curve similar in height to a smaller group if their distributions are similar. The larger sample generally gives a more stable estimate, but its curve is not automatically taller.

The plotting choices and interpretation in this tutorial are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

What a Grouped Density Curve Represents

Definition: For observations $x_{g1},\ldots,x_{gn_g}$ in group $g$, a kernel density estimate at value $x$ is $$ \hat f_g(x)=\frac{1}{n_g h}\sum_{j=1}^{n_g}K\left(\frac{x-x_{gj}}{h}\right). $$ Here $n_g$ is the group sample size, $h>0$ is the bandwidth, and $K$ is a kernel function with total area 1. The estimate is a smoothed description of the group’s distribution; its units are the reciprocal of the units on the horizontal axis.

Each observation contributes a small, centered kernel shape. The bandwidth determines the width of those shapes, and averaging their contributions gives the estimated curve. A larger bandwidth spreads each observation’s contribution over a wider range and usually produces a smoother curve. A smaller bandwidth retains more local detail, including bumps that may reflect random variation.

The vertical coordinate is density, not a count or a probability at one exact value. The area under the full density estimate is 1. The area over an interval gives an estimated proportion of that group’s observations in the interval. For example, an area of 0.30 from 10 to 20 would represent an estimated 30% of that group in that range.

Why the Area Is One

The normalization by $n_g h$ makes the estimate an average of kernels that each have area 1. To see this, consider one observation’s contribution and substitute $u=(x-x_{gj})/h$, so $dx=h\,du$. Its integrated contribution is $$ \int_{-\infty}^{\infty}\frac{1}{h}K\left(\frac{x-x_{gj}}{h}\right)dx =\int_{-\infty}^{\infty}K(u)\,du=1. $$

There are $n_g$ such contributions, each multiplied by $1/n_g$. Therefore $$ \int_{-\infty}^{\infty}\hat f_g(x)\,dx =\frac{1}{n_g}\sum_{j=1}^{n_g}1=1. $$ This is why the estimator describes a distribution within the group rather than displaying the group’s raw size.

Overlaying Groups in R

For a fair visual comparison, evaluate every curve on the same horizontal grid, use the same kernel and bandwidth, and use common axis limits. The following base R pattern fixes a bandwidth and grid for two groups. Suppose the quantitative variable is named amount and the grouping variable is segment.

\text{same }h,\quad \text{same kernel},\quad \text{same x grid and axis limits}

In R, the display can be constructed as follows. Choose axis limits broad enough to show the relevant data and the smoothed tails.

Estimate each group.
Use a fixed bandwidth, the same kernel, and identical values for from, to, and n in each call to density().

Set a shared vertical range.
Find the largest estimated density across all groups and use it as the upper limit for the common y-axis.

Draw and label the curves.
Plot one estimate, add the others with lines(), and include a legend that identifies each group.

For example, if d1 and d2 are the estimates, use plot(d1, ylim=c(0, max(d1$y, d2$y))) and then lines(d2). In an actual analysis, set meaningful common xlim values as well, and use the same grid settings when creating both estimates. A common grid makes the curves directly comparable point by point; a common bandwidth avoids giving one group more smoothing than another.

Worked Example: Comparing Two Groups on a Shared Scale

A hypothetical analyst records processing time, in thousands of minutes, for three cases in each of two service teams. Team A has values 10, 12, and 14; Team B has values 11, 13, and 15. To make a hand calculation transparent, use the triangular kernel $K(u)=1-|u|$ for $|u|<1$, and $K(u)=0$ otherwise, with bandwidth $h=4$. Estimate both densities at $x=12$.

Method. Apply the kernel density formula separately to each group, dividing the sum of kernel contributions by the group size times the bandwidth. The density units are per thousand minutes.

Team A. Its standardized distances are $(12-10)/4=0.5$, $(12-12)/4=0$, and $(12-14)/4=-0.5$. The kernel values are $0.5$, $1$, and $0.5$, with sum $2$. Thus $$ \hat f_A(12)=\frac{2}{3(4)}=\frac{2}{12}=0.1667. $$

Team B. Its standardized distances are $(12-11)/4=0.25$, $(12-13)/4=-0.25$, and $(12-15)/4=-0.75$. The kernel values are $0.75$, $0.75$, and $0.25$, with sum $1.75$. Thus $$ \hat f_B(12)=\frac{1.75}{3(4)}=\frac{1.75}{12}=0.1458. $$

Rounded to four decimal places, the estimated densities at 12 are 0.1667 for Team A and 0.1458 for Team B, per thousand minutes. This single point suggests similar density there; the full curves are needed to compare distributions across the full range. Check: both calculations use the same $n=3$ and $h=4$, and each divides its kernel sum by 12. The common mistake avoided is treating the kernel sum itself as the density, without dividing by $nh$.

Sample Size and Bandwidth Are Different Issues

Because the estimate divides by the group size, duplicating every observation does not double the density curve. A larger sample does affect how much the curve is likely to vary from sample to sample: with more observations, the estimate will generally be more stable. But sample size and bandwidth should not be conflated. Bandwidth controls smoothing; sample size affects the amount of information behind the estimate.

A software default may select a bandwidth separately for each group. That may be appropriate for some individual distribution summaries, but it makes a direct overlay less controlled: differences in smoothness may partly result from different bandwidths. For comparison, specify one bandwidth in the units of the variable. If the groups have very different sample sizes, note that the smaller group’s curve is typically less stable even when the same bandwidth is used.

Worked Example: Unequal Sample Sizes Do Not Mean Unequal Density

A hypothetical audit compares processing times for two groups. Group A has four observations, 2, 4, 6, and 8 minutes. Group B has 40 observations: each of the four values 2, 4, 6, and 8 occurs ten times. Use the triangular kernel and bandwidth $h=2$. Calculate each estimated density at $x=4$, then interpret the role of sample size.

Group A. The standardized distances are $1,0,-1,-2$. Their triangular kernel values are $0,1,0,0$, which sum to 1. Therefore $$ \hat f_A(4)=\frac{1}{4(2)}=\frac{1}{8}=0.125. $$

Group B. Each of the four distinct observations occurs ten times. The kernel-value sum is therefore $10(0+1+0+0)=10$. Hence $$ \hat f_B(4)=\frac{10}{40(2)}=\frac{10}{80}=0.125. $$

The estimates at 4 agree. In fact, the full density estimates agree because each group has the same relative distribution of values: multiplying every observation’s contribution by ten also multiplies $n_g$ by ten, so the ratio is unchanged. The curves are not equally supported by data, however; Group B has more observations and its estimate would generally be more stable. Check: both estimates equal 0.125 per minute. The common mistake avoided is interpreting the larger group’s tenfold count as a tenfold density.

Bandwidth Changes the Shape of the Display

A bandwidth is a modeling choice for the smoothness of the estimate, not a change to the observations. If it is too small, the curve can show many narrow local peaks. If it is too large, separate clusters or meaningful differences can be smoothed together. A fair overlay should use a common bandwidth, but that alone does not guarantee that the chosen bandwidth communicates the data well. Consider whether the main visible pattern persists under a modest change in bandwidth.

Worked Example: Reading the Effect of Bandwidth

A hypothetical group has four processing times: 0, 0, 4, and 4 minutes. Use the triangular kernel and compare $h=1$ with $h=4$. Calculate the density estimates at $x=0$, $x=2$, and $x=4$. Interpret the difference.

With $h=1$. At $x=0$, the two observations at 0 have kernel value 1 each; the observations at 4 have standardized distance $-4$, outside the kernel’s support. The sum is 2, so $\hat f(0)=2/[4(1)]=0.5$. At $x=2$, all four standardized distances have absolute value 2, so all kernel values are zero and $\hat f(2)=0$. By symmetry, $\hat f(4)=0.5$.

With $h=4$. At $x=0$, the observations at 0 contribute 1 each, while those at 4 have standardized distance $-1$ and contribute zero. The density is $2/[4(4)]=0.125$. At $x=2$, all four observations have standardized distances with absolute value $0.5$, giving kernel value $0.5$ each. The sum is 2, so $\hat f(2)=2/16=0.125$. By symmetry, $\hat f(4)=0.125$.

To four decimal places, the estimates at $0,2,4$ are $0.5000, 0.0000, 0.5000$ for $h=1$, and $0.1250, 0.1250, 0.1250$ for $h=4$. The narrow bandwidth preserves the two separated clusters; the wide bandwidth spreads their contributions across the middle. The complete curves are not flat at $h=4$; these are just three evaluated points. Check: both bandwidth choices produce an estimate with total area 1. The mistake avoided is concluding from a lower peak alone that the group has less probability mass.

Interpreting Overlap and Avoiding Misreadings

When curves overlap substantially, the groups have similar estimated distributions over the overlapping range. A shift in the peaks may indicate a difference in typical values; a wider curve may indicate greater spread; and a longer tail may indicate more observations extending toward that end. These are descriptions of the displayed estimates, not conclusions about causation or formal evidence that the underlying populations differ.

The common horizontal and vertical scales are essential. Changing the horizontal limits separately can make similar distributions appear different; changing vertical limits can distort perceived peak heights. Also, a density ordinate is not the probability of observing exactly that value. For a continuous variable, probability is represented by area over an interval, not by the height at a single point.

Do not compare raw group sizes from density heights. Each group’s density is normalized to area 1.
Do not let automatic bandwidths silently differ. Specify a common bandwidth when the aim is direct comparison.
Do not overstate a small wiggle. A curve is an estimate, and the smaller group’s estimate is generally less stable.
Do not treat overlap as a test. An overlay is exploratory; it does not by itself establish statistical significance or explain why groups differ.

Key takeaway: Overlay density curves to compare distributions within groups. Use the same kernel, bandwidth, grid, and axis scales; interpret vertical values as density and area as proportion; and keep unequal sample sizes in mind when judging how stable each curve may be.

Check Your Understanding

Work through these original problems, then compare each result with the answer shown.

Two groups each have three observations. With triangular kernel and $h=2$, Group A is $(0,2,4)$ and Group B is $(1,3,5)$. Calculate both estimates at $x=2$. (Answer: $\hat f_A(2)=0.1667$ and $\hat f_B(2)=0.1667$.)
A density estimate is 0.08 per dollar from 20 to 25 dollars. What approximate proportion of the group lies in that interval if the density is constant there? (Answer: $0.08(25-20)=0.4000$, or 40%.)
Group C has observations $(1,3,5)$; Group D contains each of those observations five times. With the same kernel and bandwidth, how do their estimated density curves compare, and what differs in interpretation? (Answer: The curves are identical; Group D has more observations, so its estimate is generally more stable, but it is not five times as high.)
For a triangular kernel, observations are $0,0,6,6$. With $h=3$, calculate the estimate at $x=3$. (Answer: Each standardized distance has absolute value 1 and kernel value 0, so $\hat f(3)=0.0000$.)
An analyst overlays two groups but uses a bandwidth of 2 for one and 5 for the other. Name the comparison problem and the correction. (Answer: The displayed smoothness is confounded with group differences; use a common bandwidth in the variable’s units, along with shared grid and axis scales.)

What You'll Learn