Why Filter Rows?
An exploratory question often concerns only part of a data set: policies issued in one region, claims above a threshold, or records meeting two conditions at once. Filtering rows lets you inspect that relevant subset without discarding or altering the other observations in the source data frame. This is a practical way to make a targeted summary answer a clearly stated question.
The key idea is to create a logical condition that is evaluated separately for every row. R then uses the resulting sequence of TRUE and FALSE values to decide which rows to return. A filter should reflect the question precisely: in particular, distinguish “above” from “at least,” and decide how records with missing values should be treated.
One Condition at a Time
Suppose a data frame named policies contains a column called region. The expression policies$region == "West" compares every value in that column with the text "West" and returns a logical vector. If the first three regions are West, East, and West, the first three results are TRUE, FALSE, and TRUE. The comparison does not change policies; it describes which rows meet the condition.
Square brackets select rows and columns using the form data[rows, columns]. In data[condition, ], the condition is in the row position and the blank after the comma means “keep all columns.” This matters when the purpose is to inspect all the information for selected observations, rather than to select a smaller set of columns as well.
Common comparison operators include == for equal to, != for not equal to, > for greater than, >= for greater than or equal to, < for less than, and <= for less than or equal to. Use ==, not =, to test equality. A single = is commonly used to assign a value or name an argument; it is not the equality comparison used in a row condition.
Worked Example: Inspecting Policies in One Region
A hypothetical portfolio extract has six rows. The policy IDs are P11, P12, P13, P14, P15, and P16, in that order. Their regions are West, East, West, North, West, and South. Select the West policies and report how many are selected.
Method. Compare each region with "West", then place that logical vector in the row position of the data-frame index.
R expression: policies[policies$region == "West", ]
Step 1: Evaluate the condition by row. The six comparisons produce TRUE, FALSE, TRUE, FALSE, TRUE, and FALSE. In order, these correspond to P11, P12, P13, P14, P15, and P16.
Step 2: Select the TRUE rows. The selected policy IDs are P11, P13, and P15. The blank column position retains all columns for those three rows.
Step 3: Count the selected observations. There are three TRUE results, so three policies are returned. The check is consistent with the selected IDs: P11, P13, and P15 are exactly the three policies marked West.
Common mistake avoided. Testing policies$region = "West" does not perform an equality comparison. Use == for the row condition.
Combining Conditions
A targeted subset may need to satisfy more than one requirement. In base R, the vectorized AND operator & returns TRUE for a row only when both corresponding conditions are TRUE. The vectorized OR operator | returns TRUE when at least one corresponding condition is TRUE. Each operator works across the full logical vectors, row by row.
For example, a filter for records in the North region with age above 60 uses an AND condition: region equals "North" and age is greater than 60. A filter for records in the North region or with age above 60 uses OR instead. Those statements define different groups, so the operator must match the wording of the question.
Use & and | for conditions applied across rows. The operators && and || evaluate only the first element of each vector, so they are not substitutes for & and | in a data-frame row filter. This is an important distinction: a row condition needs a result for every row, not a single result based on the first row.
Worked Example: Applying a Strict Age Threshold
A hypothetical claims table contains five claimants with ages 42, 61, 67, 60, and 73. Their paid claim amounts, in the same order, are $700, $1,200, $1,800, $900, and $1,500. Select claimants older than 60 and calculate the mean paid amount for the selected rows.
Method. Filter on the numeric condition age > 60, inspect which rows it selects, and then calculate the mean from the corresponding paid amounts.
R expression: claims[claims$age > 60, ]
Step 1: Evaluate the age condition. The comparisons are 42 > 60 (FALSE), 61 > 60 (TRUE), 67 > 60 (TRUE), 60 > 60 (FALSE), and 73 > 60 (TRUE). Thus the mask is FALSE, TRUE, TRUE, FALSE, TRUE.
Step 2: Identify the selected claimants and payments. The selected ages are 61, 67, and 73, and their paid amounts are $1,200, $1,800, and $1,500. The claimant aged exactly 60 is excluded because the condition is strictly greater than 60.
Step 3: Calculate the targeted mean. The total selected payment is $1,200 + $1,800 + $1,500 = $4,500. There are three selected rows, so the mean is $4,500 / 3 = $1,500.
Check. The selected amounts are $1,200, $1,500, and $1,800; their middle value is $1,500, which agrees with their mean. This check also confirms that the payment of $900 for the age-60 claimant was not included.
Common mistake avoided. Replacing > with >= would include the claimant aged exactly 60 and change the result. Use the operator that matches the stated threshold.
Missing Values in Conditions
A comparison involving a missing value, represented by NA in R, usually produces NA rather than TRUE or FALSE. This is not the same as a row that definitely fails the condition: R does not know whether the missing value would meet the test. In base R, a logical row index containing NA can produce a returned row whose entries are NA. Such a row is not an observed record and can mislead a later inspection.
When the question requires a known value, state that requirement in the filter. For a numeric condition such as age > 60, use !is.na(age) & age > 60. The first part tests that age is not missing; the second tests the threshold. When age is NA, !is.na(age) is FALSE, so the combined result is FALSE and that row is not selected.
For compound conditions, consider missingness in every variable used in a comparison. The behavior of AND and OR depends on the values of their component results: for example, FALSE & NA is FALSE, while TRUE | NA is TRUE. But a result can still be NA when neither known component determines the answer. Explicit missing-value checks make the intended rule easier to see and reduce the risk of unintended NA entries in the filtered result.
Worked Example: Combining Region and Payment Conditions
A hypothetical claims extract has six rows. The region values are North, South, North, East, South, and North. The paid amounts are $900, $2,400, $1,700, $2,100, NA, and $2,600, respectively. Select records that are in the North region or have a known paid amount above $2,000. Report the selected row numbers and count.
Method. Define the North-region condition and the known-high-payment condition separately, then join them with vectorized OR. The nonmissing-value check ensures that an unknown payment is not treated as a known payment above $2,000.
R expression: claims[(!is.na(claims$region) & claims$region == "North") | (!is.na(claims$paid) & claims$paid > 2000), ]
Step 1: Evaluate the North condition. There are no missing regions. The results for rows 1 through 6 are TRUE, FALSE, TRUE, FALSE, FALSE, and TRUE.
Step 2: Evaluate the known-payment condition. The payments are compared with $2,000 after checking that each is not missing. The results are FALSE, TRUE, FALSE, TRUE, FALSE, and TRUE. Row 5 is FALSE because its payment is missing, not because its unknown value has been assumed to be below the threshold.
Step 3: Apply OR row by row. Combining the two sequences gives TRUE, TRUE, TRUE, TRUE, FALSE, and TRUE. Rows 1, 2, 3, 4, and 6 are therefore selected.
Step 4: Check the count. The selected row numbers are 1, 2, 3, 4, and 6: five rows. Each selected row satisfies at least one stated condition: rows 1, 3, and 6 are North, while rows 2 and 4 have known payments above $2,000. Row 5 meets neither condition on the available information.
Common mistake avoided. A bare comparison paid > 2000 returns NA for row 5. Filtering without making the missing-value rule explicit can leave an NA result in the index. Here the rule is clear: a missing paid amount does not satisfy the known-payment condition.
Why the Filter Selects Those Rows
The indexing rule follows directly from how a logical row index is interpreted. Suppose a data frame has \(n\) rows and a condition returns \(c_1, c_2, \ldots, c_n\), where each \(c_i\) is TRUE or FALSE. Position \(i\) in the condition corresponds to row \(i\) in the data frame. R returns row \(i\) exactly when \(c_i\) is TRUE. Therefore, the selected data frame contains precisely the original rows whose conditions are true, in their original order.
This explains both the usefulness and the limits of the operation. The condition selects rows; it does not sort them, and it does not make a missing value known. Also, evaluating data[condition, ] creates a filtered result without replacing the source data frame. Assigning the result to a new name keeps both objects available. Assigning it back to the original name replaces that name’s current result, so preserve a separate source object when you need to return to the complete data.
Checks Before a Targeted Summary
A filter can be syntactically valid and still answer the wrong question. Before calculating a summary from selected rows, verify the condition against the wording, inspect a few identifying values, and check the selected count. In particular, pay attention to the following distinctions:
- Strict versus inclusive thresholds. “Above 60” means > 60; “at least 60” means >= 60.
- AND versus OR. “Meets both conditions” calls for &; “meets either condition” calls for |.
- Known versus missing values. Decide whether a row with a missing comparison value should be excluded, and write that rule explicitly when needed.
- Rows versus columns. In data[condition, ], the condition selects rows and the blank retains all columns.
- Vectorized versus single-result operators. Use & and | to combine row-wise conditions, not && and ||.
Filtering is useful because it makes an exploratory calculation specific to the population named in the question. It does not by itself establish that the selected records are representative of a wider population or that a difference between subsets is causal. For this step, the priority is narrower: define the subset correctly, preserve the source data, and ensure that the returned rows match the stated selection rule. This row-filtering practice is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, determine the selected observations and apply the stated rule carefully.
- A hypothetical table has IDs A, B, C, D and regions Central, West, Central, East. Write the base R row-filter expression to select Central records, then state the selected IDs and count. (Answer: use records[records$region == "Central", ]; IDs A and C, count 2.)
- A table has ages 48, 55, 56, 70 and exposure amounts 100, 200, 300, 400. Select records aged at least 56. What are the selected exposures and their mean? (Answer: exposures 300 and 400; mean 350.)
- A hypothetical file has six rows with claim status values Open, Closed, Open, Closed, Open, Closed, and paid amounts 500, 2,300, 1,700, 900, 2,100, 1,200. Select rows that are Open and have paid amounts above 1,000. Which rows are selected? (Answer: rows 3 and 5.)
- A condition is built for amount > 1,000, and the amounts are 800, NA, and 1,400. What results from the raw comparison, and how should the filter be written to select only known amounts above 1,000? (Answer: FALSE, NA, TRUE; use !is.na(data$amount) & data$amount > 1000.)
- A six-row data frame has logical condition results TRUE, FALSE, TRUE, FALSE, FALSE, TRUE. How many rows does data[condition, ] return, and does this expression replace data? (Answer: 3 rows; no, the expression returns a filtered result without replacing data unless the result is assigned back to the name data.)