Choose the Display That Answers the Question
A case report may contain sound model results yet leave its audience unsure what those results imply. A table can make exact values easy to compare; a plot can make patterns, trade-offs, or departures from a reference relationship easier to see. The display should follow the decision question, not the other way around. In particular, comparisons are hard to interpret when the models were evaluated on different data, a chart hides its denominators, or an axis makes a small difference appear large.
This tutorial focuses on four displays used to communicate predictive results: comparison tables, receiver operating characteristic (ROC) plots, calibration displays, and subgroup charts. Each answers a different question. None removes the need to describe the validation data, the metric, or the limitations of the evidence.
Before choosing a display, state what the reader must compare. For exact values across a few models, a table is often clearest. To show how sensitivity and false-positive rate move as the threshold changes, use a ROC plot. To assess whether predicted probabilities correspond to observed frequencies, use a calibration display. To examine whether a metric differs across groups, use a subgroup chart that makes both the rate and its denominator visible.
Comparison Tables and ROC Plots
A comparison table is useful when readers need to locate exact values, compare a small number of models, or see several metrics together. A fair comparison identifies the evaluation population and gives each model the same evaluation basis. State the sample size and, when relevant, the event count or prevalence. For classification results, identify the threshold used for each row: a metric such as sensitivity depends on that threshold.
A ROC plot addresses a different question: how do sensitivity and false-positive rate vary across thresholds? Define a positive case as an observed event and a positive prediction as a case classified above the selected threshold. If TP, FN, FP, and TN denote true positives, false negatives, false positives, and true negatives, respectively, then sensitivity is the proportion of observed events detected, while false-positive rate is the proportion of observed non-events classified as events.
The ROC point for a threshold has false-positive rate on the horizontal axis and sensitivity on the vertical axis. The denominators differ: sensitivity is calculated among observed events, and false-positive rate among observed non-events. This is why neither coordinate is simply the fraction of all cases classified as positive.
Worked Example: Compare Operating Points and Locate Them on a ROC Plot
A hypothetical insurer evaluates two classification models on the same holdout set of 1,000 policies. There are 100 observed events and 900 non-events. Model A is evaluated at threshold 0.35 and has TP = 72, FN = 28, FP = 180, and TN = 720. Model B is evaluated at threshold 0.25 and has TP = 80, FN = 20, FP = 250, and TN = 650. Construct a compact comparison and identify the ROC points.
Method. Calculate each metric from its definition, preserve the threshold and the denominators in the table, and use the resulting false-positive rate and sensitivity as the ROC coordinates.
For Model A, the event denominator is \(72+28=100\), and the non-event denominator is \(180+720=900\). Therefore, \(72/100=0.72\) sensitivity and \(180/900=0.20\) false-positive rate. Its precision, the fraction of positive classifications that are events, is \(72/(72+180)=72/252=0.2857\) to four decimals.
For Model B, the event denominator is \(80+20=100\), and the non-event denominator is \(250+650=900\). Therefore, \(80/100=0.80\) sensitivity and \(250/900=0.2778\) false-positive rate to four decimals. Its precision is \(80/(80+250)=80/330=0.2424\) to four decimals.
| Model | Threshold | Holdout cases | Events | Sensitivity | False-positive rate | Precision |
|---|---|---|---|---|---|---|
| A | 0.35 | 1,000 | 100 | 0.72 | 0.20 | 0.2857 |
| B | 0.25 | 1,000 | 100 | 0.80 | 0.2778 | 0.2424 |
The ROC point for A is \((0.20,0.72)\); for B it is approximately \((0.2778,0.80)\). B detects a larger fraction of observed events at this operating point, while also incorrectly flagging a larger fraction of non-events. The table makes exact values and thresholds easy to retrieve; the plot makes this trade-off easier to see. These two points alone do not define either model’s full ROC curve: that curve requires evaluating the model over its range of thresholds.
Common mistake and correction. A report might rank A as better because its accuracy is \( (72+720)/1{,}000=0.792\), compared with B’s \( (80+650)/1{,}000=0.730\). That calculation is correct, but accuracy is not a substitute for the stated comparison: the event prevalence is only \(100/1{,}000=0.10\), and accuracy combines the two classes in one number. Report the relevant class-specific rates and thresholds; do not infer that A is preferable from accuracy alone.
For an informative ROC graphic, label both axes with the rates and their meanings, use a common 0-to-1 scale, identify the evaluation sample, and explain how thresholds are represented. The diagonal from \((0,0)\) to \((1,1)\) is a reference line, not a decision threshold. Avoid describing a model as preferable based only on a visually higher curve if the intended operating conditions or decision costs have not been specified.
Calibration Displays
A calibration display asks whether predicted probabilities agree with observed event frequencies. For example, among cases assigned probabilities near 0.20, one would expect roughly 20% to experience the event if predictions are well calibrated for that population and period. A common display groups predictions into bins, plots each bin’s mean predicted probability against its observed event proportion, and includes the reference line \(y=x\).
A grouped display summarizes cases within bins; it does not show every individual prediction. Report the binning rule and the number of cases in each bin. Unequal bin sizes matter because a rate from a small bin is generally less stable than the same rate from a large bin. Calibration in the evaluation data also does not guarantee calibration in a different population or time period.
Worked Example: Construct and Interpret a Calibration Display
A hypothetical service model assigns probabilities to 400 accounts in a holdout sample. For four bins of 100 accounts each, the mean predicted probabilities are 0.10, 0.30, 0.50, and 0.70. The observed event counts in those bins are 8, 27, 58, and 74. Calculate the plotted coordinates and the overall predicted and observed event proportions.
Method. For each bin, divide the observed event count by its case count; plot that observed proportion against the bin’s mean predicted probability.
The observed proportions are \(8/100=0.08\), \(27/100=0.27\), \(58/100=0.58\), and \(74/100=0.74\). Thus the four plotted points, with predicted probability first, are \((0.10,0.08)\), \((0.30,0.27)\), \((0.50,0.58)\), and \((0.70,0.74)\). Relative to the reference line, the first two bins have observed proportions below their mean predictions, while the last two are above.
The total predicted event count, using the bin means, is \(100(0.10)+100(0.30)+100(0.50)+100(0.70)=10+30+50+70=160\). The observed count is \(8+27+58+74=167\). Therefore the overall mean predicted probability is \(160/400=0.4000\), and the observed event proportion is \(167/400=0.4175\). The observed proportion exceeds the mean prediction by \(0.4175-0.4000=0.0175\), or 1.75 percentage points.
Check. The four bin proportions sum to \(0.08+0.27+0.58+0.74=1.67\). Because each bin has 100 cases, their equally weighted average is \(1.67/4=0.4175\), matching \(167/400\). The example uses equal-sized bins; with unequal sizes, an unweighted average of bin rates would not generally equal the overall event proportion.
Common mistake and correction. Do not treat a point near the reference line as proof that every individual probability is accurate. Each point aggregates 100 cases, so different individual predictions and outcomes can produce the same plotted point. Describe this as grouped calibration evidence, and show bin sizes so readers can judge how much data support each point.
Keep calibration axes on the same probability scale, usually from 0 to 1, and label which axis shows predicted probability and which shows observed proportion. Include the reference line and the bin counts. Do not connect points in a way that suggests unobserved intermediate bins are known; if connecting them for readability, explain that the line joins grouped summaries.
Subgroup Charts: Rates, Denominators, and Uncertainty
A subgroup chart can reveal differences that an overall metric conceals, but the plotted metric must be defined precisely. For example, a true-positive rate for a group is calculated only among that group’s observed events. A chart of rates without their denominators can make a result based on a small group look as stable as a result based on many observations. Show counts alongside rates, and include uncertainty intervals when the data support them.
The intervals in the next example use the approximate 95% Wald interval for a proportion. For observed rate \(\hat p\) from \(n\) observations, its estimated standard error is \(\sqrt{\hat p(1-\hat p)/n}\); the interval is \(\hat p\) plus or minus 1.96 standard errors. This is a normal approximation, not an exact guarantee. It can perform poorly for very small samples or rates near 0 or 1; use an appropriate alternative, such as a Wilson interval, when those conditions apply.
Worked Example: Display Subgroup True-Positive Rates With Uncertainty
A hypothetical review compares true-positive rates across three groups. Among observed events, group A has 72 detected out of 100, group B has 15 out of 25, and group C has 48 out of 80. Calculate each rate and an approximate 95% Wald interval, then describe how to present the results.
Method. For each group, calculate \(\hat p=x/n\), then calculate the standard error \(\sqrt{\hat p(1-\hat p)/n}\). Use \(\hat p\pm1.96\) times that standard error and round endpoints to three decimals.
For A, \(\hat p=72/100=0.72\). The standard error is \(\sqrt{0.72(0.28)/100}=\sqrt{0.2016/100}=\sqrt{0.002016}=0.0449\). The interval is \(0.72\pm1.96(0.0449)=0.72\pm0.0880\), or approximately \((0.632,0.808)\).
For B, \(\hat p=15/25=0.60\). The standard error is \(\sqrt{0.60(0.40)/25}=\sqrt{0.24/25}=\sqrt{0.0096}=0.0980\). The interval is \(0.60\pm1.96(0.0980)=0.60\pm0.1921\), or approximately \((0.408,0.792)\).
For C, \(\hat p=48/80=0.60\). The standard error is \(\sqrt{0.60(0.40)/80}=\sqrt{0.24/80}=\sqrt{0.003}=0.0548\). The interval is \(0.60\pm1.96(0.0548)=0.60\pm0.1074\), or approximately \((0.493,0.707)\).
A chart should show the rates 0.72, 0.60, and 0.60, with their intervals, and label the denominators as 72/100, 15/25, and 48/80. Use a common vertical axis from 0 to 1 so the size of the rates is not exaggerated. B’s interval is widest, consistent with its smaller denominator. This display does not establish why the group rates differ, and overlapping intervals alone do not prove that the rates are equal.
Common mistake and correction. Showing only the three percentages hides that B’s estimate is based on 25 observed events while A’s is based on 100. Plot the denominator and uncertainty with the rate; state that the displayed intervals are approximate and avoid treating visual overlap as a formal test of equality.
A Practical Selection and Annotation Sequence
Use a short sequence to keep the visual form tied to the evidence and the question. A polished display is not merely attractive: its labels, scales, and notes help a reader recover how each result was obtained.
State whether the audience is comparing model values, threshold trade-offs, predicted and observed probabilities, or group-specific rates.
Identify the evaluation population, period, and sample size. Make clear when model results use different evidence rather than presenting them as a direct comparison.
Use a table for exact values across a few alternatives; a ROC plot for threshold trade-offs; a calibration display for probability agreement; and a subgroup chart for rates across defined groups.
Label axes and units, identify thresholds or binning, show denominators, and include uncertainty where appropriate. Use scales that preserve the meaning of the visual distance.
Describe what the display supports and what it does not. A grouped calibration plot does not establish individual accuracy, and a subgroup rate difference does not by itself explain its cause.
A truncated axis can be useful for examining a narrow range, but it can exaggerate differences if the visual scale is not obvious. For rates and probabilities, a 0-to-1 scale is a clear default when the purpose is comparing magnitudes. If a different range is necessary for legibility, label it prominently and avoid implying that a small visual difference is large in absolute terms. The same principle applies to tables: align decimal places, retain meaningful precision, and do not add columns that distract from the stated comparison.
These choices support a report that is both legible and appropriately cautious. They are aligned with the SOA Exam PA (Predictive Analytics) syllabus because they require translating predictive evidence into a clear, bounded comparison rather than relying on a metric or graphic without context.
Check Your Understanding
For each problem, identify the display or calculation that best answers the stated question.
- A hypothetical screening model is evaluated on 500 cases: 50 are observed events and 450 are non-events. At one threshold it has TP = 30, FN = 20, FP = 70, and TN = 380. Calculate sensitivity and false-positive rate, and give the ROC point. (Answer: Sensitivity \(30/50=0.6000\); false-positive rate \(70/450=0.1556\); ROC point \((0.1556,0.6000)\).)
- A probability model is evaluated in two bins of 200 cases each. Their mean predicted probabilities are 0.20 and 0.60; observed event counts are 50 and 110. Give both calibration points and compare overall observed and predicted proportions. (Answer: Points \((0.20,0.25)\) and \((0.60,0.55)\); predicted total \(40+120=160\), or 0.4000; observed total \(50+110=160\), or 0.4000.)
- Two groups have rates 24/40 and 54/120. Calculate the rates and state one essential annotation for a subgroup chart. (Answer: \(24/40=0.60\) and \(54/120=0.45\); show the denominators 40 and 120 alongside the rates.)
- On the same holdout set of 800 cases, Model X has 64 correct event detections and 576 correct non-event classifications. Model Y has 72 correct event detections and 528 correct non-event classifications. Calculate accuracy for each and explain why accuracy alone may not answer a question about event detection. (Answer: X accuracy \(=(64+576)/800=0.800\); Y accuracy \(=(72+528)/800=0.750\). Accuracy combines event and non-event classifications, so event sensitivity and the relevant threshold should also be reported.)
- A report shows subgroup rates on a vertical axis from 0.55 to 0.75, with no group counts or intervals. Give two changes that would make the chart easier to interpret without overstating the evidence. (Answer: Show each subgroup’s denominator and uncertainty interval; use a clearly labeled common scale, preferably 0 to 1 when comparing rate magnitudes, or prominently disclose and justify the truncated range.)