Why Scatterplot Points Disappear
A scatterplot represents each observation as a point at its pair of values. When many observations have the same or very similar values, their marks can cover one another. The visible plot may then suggest fewer observations than the data contain, or conceal where values are concentrated. This is overplotting.
Overplotting is especially common when one or both variables are discrete, rounded, or recorded in repeated categories. For example, many policyholders may share the same age band and the same number of reported events. A conventional scatterplot puts all those records at identical coordinates. The plot is still based on every record, but only the topmost mark is visible.
Two useful display adjustments are transparency and jitter. Transparency makes overlapping marks accumulate visually. Jitter adds small random displacements to plotted coordinates so marks that were coincident can be seen separately. Neither adjustment changes the recorded values or should be used to modify data in a model.
Transparency: Let Overlap Accumulate
In base R, a point color can be given an alpha component between 0 and 1: alpha 0 is fully transparent and alpha 1 is opaque. For example, plot(x, y, pch = 16, col = rgb(0, 0, 0, alpha = 0.25)) draws solid-shaped black points with 25% opacity. In ggplot2, a corresponding layer can use geom_point(alpha = 0.25).
If many marks at the same location use the same color and opacity, their combined darkness increases. Under the idealized standard compositing rule, each additional mark covers a fraction \(\alpha\) of the remaining background. After \(k\) identical marks, the remaining fraction of the background is \((1-\alpha)^k\); the fraction covered by the combined marks is therefore \(1-(1-\alpha)^k\).
This relationship explains why transparency can reveal concentration. It does not make darkness a direct, linear count scale: doubling the number of overlapping points does not double the combined opacity. The formula also describes an idealized overlay of identical marks; screen rendering, point shape, color, and the background affect the appearance. Use the plot to compare relative concentrations cautiously, not to read an exact count from a shade.
Worked Example: Reading an Overlapping Cluster
A hypothetical portfolio extract contains 12 records at the same plotted coordinates. A second location has one record. The analyst draws black points with alpha \(0.20\). Find the idealized combined opacity at each location and explain what the plot can support.
Method. Apply \(1-(1-\alpha)^k\), where \(k\) is the number of coincident marks. At the single-record location, \(k=1\), so the combined opacity is \(1-(1-0.20)^1=0.20\).
At the 12-record location, \(1-\alpha=0.80\). Compute \(0.80^{12}=0.068719476736\), so the combined opacity is \(1-0.068719476736=0.931280523264\), or approximately \(0.9313\).
The 12-record location should appear much darker than the single-record location. The opacity is not \(12(0.20)=2.40\); a fraction of the remaining background is covered at each overlay, so the combined opacity cannot exceed 1. The display signals a concentration, but a viewer should not infer “12” from the shade alone. Common mistake avoided: treating opacity as a count that adds linearly.
A useful starting point is often a moderately low alpha, followed by visual inspection at the final output size. If alpha is too high, dense areas become nearly opaque and overlap is hidden again. If it is too low, isolated points may be difficult to see. Also consider point size: smaller marks can reduce overlap, but very small marks may disappear when the figure is resized or printed.
Jitter: Separate Coincident Marks Carefully
Jitter is most interpretable when values are discrete or rounded and a small shift does not change the substantive meaning of the coordinate. In base R, a plotted coordinate can be jittered with the jitter() function; in ggplot2, a point layer can use position_jitter(width = ..., height = ...). The width and height specify the scale of horizontal and vertical displacement. Set the scale in the units of the plotted variables, not by choosing a visually convenient amount without checking the axis.
For a categorical variable represented by positions 1, 2, and 3, a small horizontal shift can expose repeated observations within each category. If each value is displaced by no more than 0.12 units, adjacent categories remain separated: their original distance is 1, and even the largest possible relative shift of 0.24 leaves at least \(1-0.24=0.76\) units between neighboring category positions. This kind of bound helps verify that the display adjustment will not make neighboring groups overlap.
Jitter changes where a mark appears, not what was observed. Do not replace the original variable with jittered values before calculating summaries, fitting a model, or validating a model. To make a jittered figure reproducible, set and record an R random seed when generating random displacements, and preserve the unjittered data for analysis.
Worked Example: Jittering a Discrete Predictor
A hypothetical service study records a three-level inspection category as \(x=1,2,3\), and a numeric response \(y\). Three records have coordinates \((1,4)\), \((1,4)\), and \((2,5)\). The first two points lie exactly on top of each other. The analyst wants to show their multiplicity while keeping the category assignments clear.
Method. Jitter only the horizontal display coordinate, using width at most 0.12 and height 0. For illustration, take two possible horizontal displacements for the duplicated observations of \(-0.08\) and \(+0.08\). Their plotted positions become \((0.92,4)\) and \((1.08,4)\). The third record remains at \((2,5)\).
The two records are now visible, while both plotted x-coordinates remain close to category 1. To check that a point assigned to category 1 cannot cross into the neighboring category under this bound, its largest possible plotted coordinate is \(1+0.12=1.12\), and category 2's smallest is \(2-0.12=1.88\). The remaining gap is \(1.88-1.12=0.76\). A ggplot2 layer could use position_jitter(width = 0.12, height = 0).
The actual offsets are random unless they are explicitly controlled; the values above illustrate a valid set of offsets rather than a promise about a particular R seed. The category for both observations is still 1, and their true response remains 4. Common mistake avoided: interpreting the displaced coordinates as revised observed values or using jittered values in subsequent calculations.
Choosing Between Transparency and Jitter
Transparency is usually a sound first choice when both axes are quantitative and the exact locations matter. It retains the original coordinates and allows dense regions to become more visible. Jitter is often more useful for discrete or rounded coordinates where exact overlap is mechanical and the small displacement can be understood as a display device. The methods can also be combined, but too many visual adjustments can make a figure harder to interpret rather than easier.
| Situation | Useful approach | Check before interpreting |
|---|---|---|
| Many identical or near-identical quantitative pairs | Try transparency; consider smaller points | Do dark areas show concentration without hiding isolated points? |
| A numeric response plotted against a few discrete categories | Use small horizontal jitter, often with zero vertical jitter | Does each displaced point still clearly belong to its original category? |
| Values are rounded to fixed increments on both axes | Consider modest jitter on the rounded axes, or transparency | Could displacement suggest values that are not plausible or change the apparent pattern? |
| Coordinates are continuous and differences are meaningful | Prefer transparency or smaller marks over jitter | Would random shifts obscure real separations or create apparent ones? |
When both axes are jittered, the displayed point may move in two directions, potentially creating a pattern or separation that is not in the data. If only one axis is discrete, jitter that axis alone where possible. If both axes are discrete, state clearly that jitter is for visibility and choose a displacement small relative to the spacing between values. Where exact frequencies matter, a plot that explicitly encodes counts may be preferable to relying on point darkness or random displacement.
Worked Example: Choosing a Display for Rounded Measurements
A hypothetical claims review has 10 records. The predictor is recorded in five-dollar increments, and the response is recorded in ten-dollar increments. At the plotted pair \((20,50)\), four records coincide; at \((25,50)\), two coincide; and at four other distinct coordinate pairs, there is one record each. The analyst is considering transparency with alpha \(0.30\) or jitter of up to 2 dollars on each axis.
Method. First compare the adjustment scale with the recorded spacing. Predictor values are 5 dollars apart and response values are 10 dollars apart. With jitter bounded by 2 dollars on each axis, the largest relative displacement between two points on the same axis is 4 dollars. Along the predictor axis, adjacent recorded values retain at least \(5-4=1\) dollar of separation; along the response axis, adjacent recorded values retain at least \(10-4=6\) dollars. The bounds do not make adjacent recorded levels cross, though they can make the displayed values look less precisely rounded.
Next compare the transparency option at a coincident location. For four overlapping records, combined opacity is \(1-(1-0.30)^4=1-0.70^4=1-0.2401=0.7599\). For two records, it is \(1-0.70^2=1-0.49=0.51\). Thus transparency should make the four-record location darker than the two-record location without moving the recorded coordinates.
Because the analyst wants to retain the meaning of dollar increments, begin with transparency and inspect the result. If the four-record and two-record clusters remain indistinguishable, carefully bounded jitter is an alternative for revealing individual marks, but the displacement should be disclosed. Common mistake avoided: choosing jitter solely because the points overlap, without checking displacement against the scale and meaning of the recorded values.
A Practical R Workflow
The following sequence keeps the purpose of each adjustment explicit. It applies after the scatterplot variables and units have been chosen.
Look for discrete or rounded axes, repeated coordinate pairs, and dense regions that conceal individual marks. Do not assume that a visible dot represents just one record.
Reduce point size or use transparency, for example with a base R color created by rgb(..., alpha = 0.25) or a ggplot2 geom_point(alpha = 0.25) layer. Inspect both dense and sparse areas.
Choose the axis or axes to jitter, set a displacement small relative to the data units, and verify that distinct categories or meaningful values remain distinguishable.
Keep the original coordinates for summaries and models. If random jitter is used to create a figure, set a seed for reproducibility and explain that the shifted locations are display-only.
Common Interpretation Traps
- Counting visible dots as records. Several records can occupy one mark. Check for repeated values or use a display that makes multiplicity visible.
- Reading shade as an exact count. Transparency accumulates nonlinearly, and rendering affects perceived darkness. It gives a visual cue to concentration, not a precise frequency label.
- Jittering too far. A large displacement can move points across group positions or make the data appear more variable than recorded. Bound the shift relative to the units and spacing.
- Analyzing the display coordinates. Jittered values are not observations. Compute summaries and fit models using the original data.
- Jittering continuous values without a reason. When small differences in coordinates matter, random shifts can create or hide apparent separations. Prefer transparency or smaller marks in that situation.
For predictive analytics work, a plot is an aid to understanding the observed data, not a license to alter them. A clear caption or report note should identify jitter as a display adjustment and avoid implying that visual density proves a particular cause. These practices are aligned with the SOA Exam PA (Predictive Analytics) syllabus, where visual choices should support accurate interpretation rather than overstate what the data show.
Check Your Understanding
For each situation, choose or evaluate a display adjustment and check the numerical reasoning.
- A scatterplot has seven records at one exact coordinate. With alpha \(0.10\), what is the idealized combined opacity at that location? (Answer: \(1-0.9^7=0.5217\), approximately.)
- A discrete predictor uses positions 1, 2, and 3. Jitter is bounded by 0.20 on each point. What is the smallest possible gap between points assigned to adjacent positions? (Answer: \(1-2(0.20)=0.60\) units.)
- At a plotted location, five records overlap and alpha is \(0.25\). Find the idealized combined opacity to four decimal places. (Answer: \(1-0.75^5=0.7627\).)
- A predictor is recorded in increments of 4 units. A proposed horizontal jitter is bounded by 2.5 units. Can points in neighboring recorded levels cross in their plotted order? (Answer: Yes; the largest relative displacement is 5 units, greater than the 4-unit spacing, so crossing is possible.)
- A continuous predictor and response have meaningful small differences, but some pairs are nearly coincident. The analyst wants to preserve exact coordinates and make concentrations easier to see. Which adjustment is the better first choice, and why? (Answer: Transparency, because it retains the original coordinates while making overlapping marks accumulate visually.)