Caveating Small or Sparse Subgroups

Why Subgroup Results Need a Caveat

A subgroup result can look striking while resting on very few observations. For example, an event rate of 20% might represent 4 events among 20 cases or 400 among 2,000. The point estimates are identical, but the first is much less precise. A report that gives only the percentages can therefore make weak evidence look conclusive.

When reporting model performance across subgroups, state the sample size and, when relevant, the number of events and non-events. Then describe uncertainty in a way that matches the metric and the data. For a proportion, an interval estimate is often more informative than the point estimate alone. It does not remove uncertainty or prove that one subgroup differs from another; it makes the uncertainty visible.

Definition: A subgroup estimate is a summary calculated for a defined subset of observations, such as the proportion of cases in that subset with a specified outcome. A sparse subgroup result is one supported by limited data, often because the subgroup has few observations, few events, or both. The number of observations and the number of events should be reported separately: a reasonably sized subgroup can still have very few events.

For a binary outcome, let \(n\) be the number of observations in a subgroup and \(x\) the number with the event of interest. The observed event proportion is \(\hat p=x/n\). Its interpretation depends on both \(x\) and \(n\). In particular, a zero observed event count does not establish that the underlying event probability is zero.

Intervals for a Subgroup Proportion

A confidence interval communicates how imprecise a proportion estimate may be under a specified sampling model. The following examples use 95% Wilson score intervals. Let \(z=1.96\), the standard normal critical value for a two-sided 95% interval. The interval is calculated from \(x\), \(n\), and \(\hat p=x/n\):

\text{center} = \frac{\hat p+z^2/(2n)}{1+z^2/n}, \qquad \text{half-width} = \frac{z}{1+z^2/n} \sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}.

\text{Wilson interval} = \text{center}\ \pm\ \text{half-width}.

The interval is bounded between 0 and 1. Unlike the simple Wald interval, which uses \(\hat p\pm z\sqrt{\hat p(1-\hat p)/n}\), the Wilson interval behaves sensibly when event counts are small, including when the observed proportion is 0 or 1. All interval endpoints below are rounded to four decimal places.

Assumptions and scope: The usual binomial interpretation treats the subgroup size \(n\) as fixed and the \(n\) outcomes as independent Bernoulli observations with a common event probability \(p\). The Wilson interval is obtained by inverting an approximate score test, so its coverage is not an exact guarantee for every possible sampling design. If observations are clustered, weights are used, or the subgroup is not representative of the population being discussed, a simple binomial interval may not describe the relevant uncertainty. State the limitation rather than applying the formula automatically.

Why the Wilson Formula Has This Form

The Wilson interval is found by asking which candidate values \(p_0\) would not be rejected by a two-sided score test at the chosen confidence level. Under the binomial model, the score statistic is

\frac{\hat p-p_0}{\sqrt{p_0(1-p_0)/n}}.

The candidate \(p_0\) is retained when the absolute value of this statistic is no greater than \(z\). Squaring both sides and multiplying by the positive denominator gives

n(\hat p-p_0)^2 \le z^2p_0(1-p_0).

Set \(q=z^2/n\), divide by \(n\), and collect terms in \(p_0\). The inequality becomes

(1+q)p_0^2-(2\hat p+q)p_0+\hat p^2 \le 0.

The retained values lie between the roots of this quadratic. The midpoint of its roots is \((\hat p+q/2)/(1+q)\), which gives the stated center. The distance from the midpoint to either root is \(\sqrt{q\hat p(1-\hat p)+q^2/4}/(1+q)\). Substituting \(q=z^2/n\) gives the stated half-width. Thus, the Wilson formula follows by inverting the score test rather than by treating the estimated standard error as fixed at \(\hat p\).

Worked Examples

Worked Example: A Large Estimate Difference with Sparse Support

In a hypothetical validation study, subgroup A has 18 events among 120 cases, while subgroup B has 4 among 20. Calculate each observed proportion and 95% Wilson interval. Then state what the figures support. Use \(z=1.96\), so \(z^2=3.8416\).

Method. Calculate \(\hat p=x/n\), then substitute \(x\), \(n\), and \(\hat p\) into the Wilson formulas. The common mistake to avoid is reporting the difference between the two point estimates as if it were an established subgroup difference.

Subgroup A. The observed proportion is \(\hat p=18/120=0.15\). The denominator is \(1+3.8416/120=1.032013\). The center is \((0.15+3.8416/240)/1.032013=(0.1660067)/1.032013\approx0.1609\). The quantity under the square root is \(0.15(0.85)/120+3.8416/[4(120)^2]=0.0010625+0.00006669=0.00112919\). The half-width is approximately \((1.96/1.032013)\sqrt{0.00112919}=0.0638\). The interval is approximately \(0.1609\pm0.0638\), or 0.0970 to 0.2247.

Subgroup B. The observed proportion is \(\hat p=4/20=0.20\). The denominator is \(1+3.8416/20=1.19208\). The center is \((0.20+3.8416/40)/1.19208=0.2483\). The quantity under the square root is \(0.20(0.80)/20+3.8416/[4(20)^2]=0.008+0.002401=0.010401\). The half-width is approximately \((1.96/1.19208)\sqrt{0.010401}=0.1677\). Using the unrounded center and half-width, the interval is approximately 0.0807 to 0.4160.

The point estimates differ by \(0.20-0.15=0.05\), or 5 percentage points. But subgroup B’s interval is wide: its data are consistent with a substantial range of possible event proportions. The responsible conclusion is that the observed estimates differ, while subgroup B’s estimate is imprecise. These calculations alone do not establish that the underlying subgroup proportions differ.

Worked Example: A High Rate Based on Six Cases

A hypothetical operational review records 3 events among 6 cases in subgroup C and 60 among 200 cases in subgroup D. Calculate both observed proportions and their 95% Wilson intervals. Explain why the observed 20-percentage-point difference should be reported cautiously.

Method. Apply the Wilson formula separately to each subgroup; do not pool their counts when the purpose is to describe subgroup-specific estimates. The common mistake is to treat a high proportion based on six cases as comparably precise to one based on hundreds.

Subgroup C. Here, \(\hat p=3/6=0.50\), and the denominator is \(1+3.8416/6=1.640267\). The center is \((0.50+3.8416/12)/1.640267=0.5000\). The square-root term is \(0.50(0.50)/6+3.8416/[4(6)^2]=0.0416667+0.0266778=0.0683445\). The half-width is approximately \((1.96/1.640267)\sqrt{0.0683445}=0.3124\). The interval is therefore 0.1876 to 0.8124.

Subgroup D. Here, \(\hat p=60/200=0.30\), and the denominator is \(1+3.8416/200=1.019208\). The center is \((0.30+3.8416/400)/1.019208\approx0.3038\). The square-root term is \(0.30(0.70)/200+3.8416/[4(200)^2]=0.00105+0.00002401=0.00107401\). The half-width is approximately \((1.96/1.019208)\sqrt{0.00107401}=0.0630\). The interval is approximately 0.2407 to 0.3668.

The observed proportions are \(0.50\) and \(0.30\), a difference of 0.20. Subgroup C’s interval is especially wide because only six cases were observed. The estimates should be reported with their counts and uncertainty; the point difference alone is not enough to support a confident claim about the underlying proportions. Whether a difference is statistically supported requires an appropriate comparison, not just separate interval calculations.

Worked Example: No Observed Events Does Not Mean No Risk

In a hypothetical validation set, subgroup E has zero events among 12 cases. Another subgroup, F, has 8 events among 80 cases. Calculate the observed proportions and 95% Wilson intervals. What conclusion about subgroup E is justified?

Method. Use the Wilson formula even when \(\hat p=0\); this is one reason to prefer it to the simple Wald interval. The common mistake is to report zero events as proof that the event probability is zero.

Subgroup E. The observed proportion is \(0/12=0\). The Wilson center is \((0+3.8416/24)/(1+3.8416/12)=0.1600667/1.320133\approx0.12125\). The half-width is also approximately \(0.12125\), giving an interval of 0.0000 to 0.2425. The interval includes positive event probabilities despite no observed events.

Subgroup F. The observed proportion is \(\hat p=8/80=0.10\). The denominator is \(1+3.8416/80=1.04802\). The center is \((0.10+3.8416/160)/1.04802=0.1183\). The square-root term is \(0.10(0.90)/80+3.8416/[4(80)^2]=0.001125+0.00015006=0.00127506\). The half-width is approximately \((1.96/1.04802)\sqrt{0.00127506}=0.0668\). Using unrounded values, the interval is approximately 0.0515 to 0.1851.

The data show no events in the 12 observed cases in subgroup E; they do not show that the underlying probability is zero. The interval makes clear that the estimate is imprecise. A report should state the count and the subgroup size, rather than presenting “0%” without qualification.

How to Write a Cautious Subgroup Finding

A useful report distinguishes what was observed from what can reasonably be inferred. The observed rate is a fact about the validation sample. An interval describes uncertainty under its assumptions. A conclusion about a broader population needs additional support: the sample must be relevant to that population, and the method must reflect the sampling design. A numerical interval does not repair a biased or unrepresentative sample.

When subgroup counts are small, use wording that is both specific and restrained. For example: “The observed event proportion was 20% (4 of 20 cases); the 95% Wilson interval was approximately 8.1% to 41.6%, so this estimate is imprecise.” This is more informative than “the subgroup rate was 20%” and more defensible than asserting a large or established difference.

Do not use overlap between two separate confidence intervals as a definitive test of a difference. Separate intervals describe uncertainty around separate proportions; they are not, by themselves, a confidence interval for the difference. If a decision requires comparing proportions, use an appropriate method for that comparison and its assumptions. If the data are too limited for a reliable comparison, state that plainly.

The same discipline applies to metrics other than proportions. Report the subgroup sample size alongside an error summary, and avoid implying precision that the data cannot support. A small subgroup may produce a volatile estimate even when a metric is calculated correctly. Where there are very few events or cases, it may be more useful to describe the result as exploratory, gather more evidence, or avoid a strong subgroup conclusion.

Define the denominator.
State who or what is included in the subgroup and how many observations it contains.

Show the count behind the metric.
For an event proportion, report both the number of events and the total number of cases, not only the percentage.

Describe uncertainty with an appropriate method.
For a proportion under the binomial assumptions, a Wilson interval is one useful option. Check whether the assumptions match the data.

Separate observation from conclusion.
Say what the validation data show, then qualify what they do not establish. Do not promote an unstable difference into a firm claim.

A compact report can therefore include the subgroup definition, count, observed metric, uncertainty, and a one-sentence limitation. For instance, “Among 20 cases in subgroup B, 4 had the event (20%; 95% Wilson interval 8.1% to 41.6%); the small denominator makes this estimate imprecise.” The interval should not be presented as the only limitation: sampling bias, changes in the evaluated population, and the number of subgroup comparisons may also affect interpretation.

Key takeaway: A subgroup percentage without its denominator can be misleading. Report counts as well as rates, use an uncertainty method that fits the data, and describe sparse results as limited evidence rather than established differences.

Check Your Understanding

For each situation, distinguish the observed result from what the limited subgroup data can support.

A hypothetical subgroup has 5 events in 25 cases. Give its observed proportion and identify the two counts a report should include. (Answer: 20%; report 5 events and 25 total cases.)
A subgroup has zero events among 10 cases. Using a 95% Wilson interval with \(z=1.96\), give the interval’s upper endpoint to four decimal places and state whether zero risk is established. (Answer: upper endpoint 0.2775; no, zero observed events do not establish zero risk.)
Subgroup G has 24 events among 160 cases, and subgroup H has 3 among 15. Calculate the observed proportions and their difference, then state whether the point estimates alone establish a population difference. (Answer: 15.0% and 20.0%, a 5.0-percentage-point difference; no.)
A report states, “The event rate was 50% in subgroup J,” based on 3 events among 6 cases. Give a more informative version that includes the Wilson interval to four decimal places and a caveat. (Answer: 3 of 6 cases had the event (50.0%; 95% Wilson interval 0.1876 to 0.8124); the estimate is imprecise.)
A subgroup estimate is based on a clustered sample, but the analyst reports an ordinary binomial Wilson interval. Identify the key concern and the appropriate reporting response. (Answer: independence may fail, so the simple interval may not represent uncertainty; use a method that accounts for the sampling design or qualify the interval’s limitations.)

What You'll Learn