Why Repeated Records Need a Time Context
A data set may contain several rows for the same policy, customer, vehicle, or facility. Those rows are not necessarily duplicates: each may describe that entity at a different observation time. A monthly policy record, for example, can represent a different prediction opportunity from a quarterly policy record, even if both rows carry the same policy identifier.
To interpret such data, identify not only which entity a row concerns, but also when and over what interval the row was created. The schedule affects how many records an entity contributes, what information can be attached to each record, and the period meant by an outcome label. A record-level percentage is therefore meaningful only in relation to the records and time intervals that form its denominator.
What the Observation Schedule Specifies
A schedule can be regular, such as one record at the start of each calendar month, or irregular, such as a record each time an inspection occurs. When describing it, state the timing and frequency as precisely as the data allow. “Repeatedly observed” is not enough: monthly measurements and measurements taken only at renewal create different record sequences.
The schedule also determines whether a row describes a point in time or a period. A balance measured on April 1 is a snapshot at that date. A claim indicator defined as “any claim during April” describes an interval. The row’s date alone does not establish which meaning applies; that must come from the data definition.
- Entity: the unit that can recur across rows, such as one policy.
- Observation time: the date on which the row is indexed or recorded.
- Observation interval: the period summarized by a measurement, if it is a period rather than a snapshot.
- Outcome window: the period in which the outcome is assessed, stated relative to the observation time when appropriate.
- Schedule: the rule determining which observation times or events produce records.
These distinctions are consistent with the earlier material on unit of analysis, prediction index dates, predictor availability, and outcome horizons. In a predictive data set, a row should refer to a defined prediction occasion, with predictors available at that occasion and an outcome assigned to its stated future window. Repeating an entity identifier does not, by itself, tell us that two rows refer to the same prediction occasion.
Worked Example: Monthly and Quarterly Outcome Records
A hypothetical insurer follows four policies for six months. For monthly records, each row indicates whether a service request occurred in that month. For quarterly records, each row indicates whether at least one request occurred in the three-month quarter. Requests occurred for Policy A in months 1 and 4, Policy B in month 2, and Policy D in months 1 and 5. Policy C had no requests. Calculate and interpret the positive-record proportions for the two schedules.
Method. Count the rows generated by each schedule, then count positive rows using the outcome definition for that schedule. The number of policies is four in both cases; the number and meaning of the records differ.
Monthly schedule. Each policy has six monthly records, so there are \(4 \times 6=24\) records. The five request-months are A1, A4, B2, D1, and D5. Each produces one positive monthly record. Thus the proportion of positive monthly records is \(5/24=0.2083\), or 20.83% when expressed as a percentage.
Quarterly schedule. Each policy has two quarterly records, so there are \(4 \times 2=8\) records. In months 1–3, Policies A, B, and D each have at least one request, giving three positive records. In months 4–6, Policies A and D each have at least one request, giving two positive records. There are \(3+2=5\) positive quarterly records. The proportion is \(5/8=0.625\), or 62.5%.
Check. Both calculations account for all four policies over all six months: \(24\) monthly records and \(8\) quarterly records. The quarterly proportion is not evidence that the underlying request activity became more frequent. A positive quarterly row covers three months and only asks whether any request occurred during that interval; a positive monthly row covers one month.
Common mistake avoided: Treating the two proportions as estimates of the same quantity. Their denominators count different observation intervals, and their positive labels use different time windows. The schedule and outcome definition must accompany any comparison.
Why the Record-Level Denominator Matters
A useful way to make the interpretation precise is to define \(Z_{ij}\) as an indicator equal to 1 when the outcome is positive for entity \(i\) at observation \(j\), and 0 otherwise. If there are \(N\) entities and each has \(m\) scheduled records, the proportion of positive records is
This is an average over records, not an average over entities. To see why, suppose every scheduled record has the same probability \(p\) of being positive. By linearity of expectation, the expected total number of positive records is
Dividing by the fixed number of records, \(Nm\), gives an expected record-level proportion of \(p\). This justification requires the same positive-outcome probability at each scheduled record and a fixed schedule with \(Nm\) records. It does not require independence for this expectation calculation. If observation frequencies vary, or the probabilities differ across times or entities, the record-level average instead combines the probabilities of the records that actually appear. It does not automatically give every entity equal weight.
A record-level outcome can also refer to a defined interval rather than a single event. For example, “any incident during the next 30 days” and “any incident during the next 90 days” are different labels. A longer window gives more time in which an event can qualify, so the positive-record proportions need not match even if the underlying entities and observation dates are otherwise similar.
Worked Example: Records Indexed at the Start of Each Quarter
A hypothetical service program creates one record for each of three customers at the start of each quarter: January 1, April 1, and July 1. Each record has a positive outcome if the customer requests assistance during the following calendar quarter. Customer A requests assistance during the first and third quarters, Customer B during the second quarter, and Customer C has no requests. Find the number of records, the number of positive records, and the positive-record proportion. State what one row represents.
Method. Count scheduled customer-quarter opportunities first. Then assign each request to the record whose following calendar quarter contains it.
There are three customers and three scheduled index dates per customer, so the total is \(3 \times 3=9\) records. Each row represents one customer at one quarterly index date, with an outcome defined over the next calendar quarter.
Customer A has positive outcomes for the records indexed January 1 and July 1, giving two positive records. Customer B has a positive outcome for the record indexed April 1, giving one. Customer C has none. The positive-record total is \(2+1+0=3\).
The positive-record proportion is \(3/9=0.3333\), or 33.33% when expressed as a percentage. The denominator is nine customer-quarter records, not three customers. The result describes the proportion of the scheduled quarterly opportunities with a request in the corresponding following quarter.
Check. Each customer contributes exactly three records, and each of the nine records has one defined following-quarter outcome. The three stated positive customer-quarters therefore produce three positive records.
Common mistake avoided: Calling \(3/9\) the percentage of customers who requested assistance. The calculation counts positive records; the customer-level question would use a different count and denominator.
Irregular Schedules and Unequal Record Counts
Not all repeated records arrive on a fixed calendar schedule. A record may be created at each site inspection, customer contact, or policy renewal. In that case, entities can contribute different numbers of records. A row-level percentage then gives more influence to entities with more recorded occasions, because each of their records enters the numerator or denominator as another observation.
That arithmetic does not establish why some entities have more records. It may reflect a scheduled design, an operational rule, or simply the pattern of events recorded. The data definition should distinguish these possibilities. Before interpreting a rate, check that the denominator counts the relevant opportunities and that records from different entities refer to comparable intervals or events.
Worked Example: Inspection Records Across Five Sites
A hypothetical program observes five sites for one year. It records one row per inspection, and the row is positive if a specified safety issue is found at that inspection. Site 1 has one inspection with a positive finding; Site 2 has two inspections with no positive findings; Site 3 has three inspections with one positive finding; Site 4 has four inspections with one positive finding; and Site 5 has ten inspections with one positive finding. Calculate the positive-inspection proportion and the proportion of sites with at least one positive inspection.
Method. For the record-level proportion, total the inspections and positive findings. For the site-level proportion, count sites with one or more positive findings and divide by the number of sites.
The number of inspections is \(1+2+3+4+10=20\). The positive findings total \(1+0+1+1+1=4\). Therefore, the positive-inspection proportion is \(4/20=0.2000\), or 20.00%.
Sites 1, 3, 4, and 5 each have at least one positive finding; Site 2 does not. Thus \(4\) of the \(5\) sites had at least one positive inspection, giving \(4/5=0.8000\), or 80.00%.
Check. The inspection counts sum to 20, and the positive counts sum to 4. The site-level count includes four sites with positives and one without, totaling all five sites.
The two results answer different questions. The first describes inspections; the second describes sites over the year. Site 5 contributes ten of the twenty inspection records, while Site 1 contributes one. Consequently, the inspection-level result weights those sites differently from a calculation that gives each site one annual outcome.
Common mistake avoided: Reporting 20% as “the percentage of sites with a safety issue.” Its denominator is inspections, not sites. The stated annual site-level calculation is 80%; neither percentage should be substituted for the other without naming its unit.
A Practical Interpretation Check
For every repeated-observation data set, write a one-sentence interpretation of a typical row before summarizing its outcomes. Include the entity, the observation time or event, and the outcome window. For instance: “This row describes one site at one recorded inspection, and its outcome indicates whether the specified issue was found at that inspection.” If the outcome is measured over a future interval, state the interval relative to the row’s index date.
Then verify the counts implied by the schedule. If every one of \(N\) entities is recorded \(m\) times, there should be \(Nm\) rows, unless a stated exception explains otherwise. With an irregular schedule, sum the entity-specific record counts instead. This check can reveal accidental omissions, duplicate records, or a mistaken assumption about how frequently observations were made.
Find the entity identifier and determine which rows belong to the same entity.
Describe when a record is created and whether the schedule is regular or event-based.
Distinguish a snapshot, a past-period summary, and an outcome measured over a stated window.
Say whether a summary counts records, time intervals, or entities, and interpret it using that unit.
The main lesson is not that one schedule is always preferable. The lesson is that a row has a temporal meaning, and changing the schedule can change both that meaning and the denominator of a summary. Clear definitions of the entity, record timing, and outcome window make repeated-observation data interpretable and keep record-level statements from being mistaken for entity-level statements. This treatment is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify what the records represent and interpret the requested count or rate using the correct denominator.
- A hypothetical monitoring program records each of six machines once at the start of each of four months. Three records indicate a temperature alert. How many records are there, and what proportion of records show an alert? (Answer: 24 records; \(3/24=0.1250\), or 12.50%.)
- Five customers each have two quarterly records. A positive label means at least one service call in the following quarter. Four of the ten records are positive. What does \(4/10\) describe, and why is it not automatically the proportion of customers who called? (Answer: It is the proportion of customer-quarter records followed by a service call; the same customer can have two positive records or none.)
- A hypothetical study records one row per monthly account review. Three accounts contribute 2, 4, and 6 reviews, respectively. There are 3 positive review records. Find the total number of records and the record-level positive proportion. (Answer: 12 records; \(3/12=0.2500\), or 25.00%.)
- Four facilities are each observed for a full year. The schedule produces 1, 2, 3, and 4 inspections at the four facilities, respectively; there are 2 positive inspection records, and both occur at the facility with four inspections. What is the positive-inspection proportion, and what proportion of facilities had at least one positive inspection? (Answer: \(2/10=20.00\%\) of inspection records; \(1/4=25.00\%\) of facilities.)
- Two schedules follow the same eight entities for six months. Schedule A creates one record per entity per month; Schedule B creates one record per entity per three-month quarter. State the total records under each schedule and one reason their positive-record proportions may differ even when they cover the same six-month follow-up. (Answer: Schedule A has \(8 \times 6=48\) records and Schedule B has \(8 \times 2=16\); their outcome windows differ, so a quarterly positive label can cover more time than a monthly one.)