Why Calculate Several Statistics per Group?
A single summary statistic rarely describes a group completely. A mean gives an average, but can be pulled by unusually large or small values. A median describes the middle observation, but does not show how many records support it. A count provides that context. Calculating these statistics together gives a compact description of each group and helps you understand what each reported number represents.
In R, dplyr's group_by() identifies the groups, and summarise() calculates one or more results within each group. For example, a hypothetical service team might want the number of cases and the mean and median resolution time for each intake channel. The goal here is to compute and read those columns accurately, not to treat a summary as a complete explanation of why groups differ.
The Grouped Summarise Pattern
A basic grouped summary has three parts: identify the data frame, declare the grouping variable, and request scalar statistics in summarise(). Here is the general pattern, using a data frame named records with a numeric column named amount and a grouping column named segment.
R code: records %>% group_by(segment) %>% summarise(n = n(), mean_amount = mean(amount), median_amount = median(amount), .groups = "drop")
The result has one row per observed value of segment and one column for each requested statistic. In this pattern, n() counts rows in the current group. The names before the equals signs, such as mean_amount, become the output column names. Setting .groups = "drop" makes the result ungrouped, which is useful when later code should not continue operating group by group.
Each expression in summarise() is evaluated within a group. This is why the same call to mean(amount) produces a separate mean for every segment rather than one mean for the whole data frame.
Confirm that each value represents the category whose rows should be summarized, and that the column has an appropriate class.
Include a row count when useful, and give each calculated column a clear name that identifies the statistic and variable.
Compare the row count with the number of non-missing values used by a numeric statistic. They may differ.
Use the count to understand the amount of data behind a mean or median, and interpret each statistic according to what it measures.
Why the Statistics Are Calculated Within Groups
Suppose group \(g\) contains observed values \(x_{1g}, x_{2g}, \ldots, x_{m_g g}\). The mean calculated for that group is
This formula follows directly from applying the ordinary arithmetic mean to the observations belonging to \(g\), rather than to all rows. Because group_by(segment) partitions the rows by segment, summarise(mean_amount = mean(amount)) applies that calculation separately to each partition. The count from n(), however, is \(n_g\), the number of rows, which is not necessarily \(m_g\): missing values may be present.
For a group with sorted observed values \(x_{(1)g} \leq \cdots \leq x_{(m_g)g}\), the median is \(x_{((m_g+1)/2)g}\) when \(m_g\) is odd. When \(m_g\) is even, it is the average of the two middle values, \(\bigl(x_{(m_g/2)g}+x_{(m_g/2+1)g}\bigr)/2\). R's median() uses this standard definition.
Worked Example: Three Statistics by Intake Channel
A hypothetical support unit records resolution time, in days, for seven cases. The Web cases took 2, 4, 6, and 8 days; the Phone cases took 3, 7, and 9 days. Produce the row count, mean, and median for each channel, then verify the output.
Method. Group by channel, then request n(), mean(), and median(). There are no missing resolution times, so each row count is also the number of values used in its mean and median.
R code: cases %>% group_by(channel) %>% summarise(n = n(), mean_days = mean(days), median_days = median(days), .groups = "drop")
For Web, \(n=4\). The mean is \((2+4+6+8)/4=20/4=5\) days. The sorted values are 2, 4, 6, 8, so the median is \((4+6)/2=5\) days.
For Phone, \(n=3\). The mean is \((3+7+9)/3=19/3=6.3333\ldots\) days. The sorted values are 3, 7, 9, so the median is 7 days.
| channel | n | mean_days | median_days |
|---|---|---|---|
| Web | 4 | 5 | 5 |
| Phone | 3 | 6.3333... | 7 |
The calculation checks: the displayed counts total \(4+3=7\), matching the seven cases in the data. The mean and median columns both report days, but summarize the values in different ways. A common mistake would be to calculate only one mean over all seven cases and assume it represents each channel; grouping ensures that each output row uses only its own channel's observations.
Missing Values: Rows and Usable Observations
By default, mean() and median() return a missing result when their input includes an NA. To calculate from the observed values, use na.rm = TRUE. That option removes missing values from that calculation; it does not remove the corresponding rows from the data frame or change what n() counts.
When missing values are possible, include both the row count and a non-missing count, such as n_valid = sum(!is.na(amount)). This makes the basis of a mean or median visible. If a group has no observed values, its valid count is zero; mean(amount, na.rm = TRUE) returns NaN and median(amount, na.rm = TRUE) returns NA. Those outputs are not estimates of zero.
Worked Example: Separate Row Counts from Valid Counts
A hypothetical repair desk records estimated repair amounts in dollars. The App group has values 100, NA, 300, and 500; the Desk group has values 200, 400, and NA. Calculate the row count, non-missing count, mean, and median for each group.
Method. Count all rows with n(), count observed amounts with sum(!is.na(amount)), and add na.rm = TRUE to each numeric summary.
R code: repairs %>% group_by(source) %>% summarise(n_rows = n(), n_valid = sum(!is.na(amount)), mean_amount = mean(amount, na.rm = TRUE), median_amount = median(amount, na.rm = TRUE), .groups = "drop")
For App, there are four rows, but only three observed amounts. The mean is \((100+300+500)/3=900/3=\$300\). The observed values in order are 100, 300, 500, so the median is \(\$300\).
For Desk, there are three rows and two observed amounts. The mean is \((200+400)/2=600/2=\$300\). With two values, the median is their average: \((200+400)/2=\$300\).
| source | n_rows | n_valid | mean_amount | median_amount |
|---|---|---|---|---|
| App | 4 | 3 | $300 | $300 |
| Desk | 3 | 2 | $300 | $300 |
The row counts sum to 7, while the valid counts sum to 5, matching the two missing amounts. A common mistake is to interpret n() as the denominator for the mean after removing missing values. Here, for instance, dividing the App total of 900 by 4 would give 225, but that would incorrectly treat the missing amount as if it were an observed zero.
Read Counts and Means Together
The count is important when interpreting a collection of group means. If group sizes differ, the arithmetic average of the group means generally is not the mean of all observations. The overall mean can be recovered from group counts and means by weighting each group mean by its valid count:
To justify the formula, the group mean satisfies \(m_g\bar{x}_g=\sum_{j=1}^{m_g}x_{jg}\). Adding these group totals includes every observed value exactly once. Dividing their sum by the total number of observed values gives the overall mean. This calculation requires the counts to refer to the same non-missing values used for the group means. If you use n() despite missing outcome values, the weights will not be appropriate.
Worked Example: Recovering an Overall Mean Correctly
A hypothetical equipment log reports readings from two shifts. Shift A has readings 2 and 6. Shift B has readings 8, 10, 10, and 12. From the grouped counts and means, calculate the overall mean. Also explain why simply averaging the two group means does not give the answer.
Method. Calculate each group summary, then use its count as the weight when combining means. There are no missing readings.
For Shift A, \(n_A=2\), the mean is \((2+6)/2=4\), and the median is \((2+6)/2=4\). For Shift B, \(n_B=4\), the mean is \((8+10+10+12)/4=40/4=10\), and the median is \((10+10)/2=10\).
The weighted overall mean is \((2\times4+4\times10)/(2+4)=(8+40)/6=48/6=8\). Checking from the original six readings gives \((2+6+8+10+10+12)/6=48/6=8\), as required.
The unweighted average of the group means is \((4+10)/2=7\), not 8. It gives each shift equal weight even though Shift A has two observations and Shift B has four. The group counts show why that shortcut fails. The group medians also do not determine the overall median by averaging them: the six sorted readings are 2, 6, 8, 10, 10, 12, so the overall median is \((8+10)/2=9\), not \((4+10)/2=7\).
Common Mistakes and Checks
- Confusing rows with observed values. n() counts rows, including rows whose numeric outcome is missing. Report a valid-value count separately when missing values affect a statistic.
- Forgetting to remove missing values explicitly. Without na.rm = TRUE, a missing numeric value can make the calculated mean or median missing. Do not assume R silently ignores it.
- Averaging group means without weights. The ordinary average of the group means is appropriate only when the groups have equal valid counts, or in other special cases where it happens to match. In general, use the weighted formula.
- Treating a group median as enough to find the overall median. The overall median depends on the ordering and positions of all observations, not just the median within each group.
- Giving unclear output names. Names such as mean_amount and n_valid communicate more than generic names such as x and count.
A useful final check is to confirm that row counts add to the number of input rows represented in the output, and that valid counts add to the number of non-missing values for the summarized variable. If a grouping value itself is missing, dplyr ordinarily treats those rows as a group; decide whether that is meaningful for the task rather than silently assuming they were omitted. Careful grouped summaries are a basic exploratory tool aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, calculate the requested grouped statistics and check how the counts relate to the values used.
- A hypothetical parcel team records handling times in hours. Zone A has values 4, 6, and 8; Zone B has values 1 and 9. Find each zone's row count, mean, and median. (Answer: Zone A: n = 3, mean = 6, median = 6; Zone B: n = 2, mean = 5, median = 5.)
- A hypothetical inspection log has East values 12, NA, 18, and 30, and West values 10, 20, and NA. Using missing-value removal for the numeric statistics, give each group's row count, valid count, mean, and median. (Answer: East: n_rows = 4, n_valid = 3, mean = 20, median = 18; West: n_rows = 3, n_valid = 2, mean = 15, median = 15.)
- A hypothetical sensor records Morning values 5, 7, 9, and 15, and Late values 4, 10, and 14. Find the count, mean, and median for each group. (Answer: Morning: n = 4, mean = 9, median = 8; Late: n = 3, mean = 28/3 = 9.3333..., median = 10.)
- A hypothetical test bench records Batch A values 3 and 5, and Batch B values 7, 9, 11, and 13. Find the two group means and counts, then calculate the overall mean. (Answer: Batch A: n = 2, mean = 4; Batch B: n = 4, mean = 10; overall mean = (2 × 4 + 4 × 10)/6 = 8, not the unweighted average of group means, which is 7.)
- A hypothetical desk log has group X values 2 and 4, group Y values NA and NA, and group Z value 6. With missing values removed for the mean and median, report each group's row count, valid count, mean, and median. (Answer: X: n_rows = 2, n_valid = 2, mean = 3, median = 3; Y: n_rows = 2, n_valid = 0, mean = NaN, median = NA; Z: n_rows = 1, n_valid = 1, mean = 6, median = 6.)