Monitoring Outcomes with Delayed Labels

Why Delayed Labels Change Monitoring

A model may generate predictions today, while the outcomes needed to assess those predictions arrive later. In a hypothetical claims setting, a claim could be reported weeks after an incident, and its final status might take longer to establish. A recent cohort can therefore have fewer recorded positive outcomes than an older cohort even if their eventual outcome rates are alike. Comparing the raw recorded rates can confuse incomplete reporting with a genuine difference in outcomes.

Monitoring should make the observation window explicit. For each cohort, record when its predictions were made, the date through which outcomes were observed, and how much time had elapsed for labels to arrive. This does not turn incomplete information into certain information. It provides a transparent estimate, conditional on the reporting assumptions, and signals when evidence is still provisional.

Definition: A cohort is a group of predictions or cases defined by a common starting period or operational rule. The age of a cohort is the elapsed time between that starting point and the date through which outcomes are observed. The maturity fraction \(c(a)\) is the estimated proportion of all eventual positive outcomes that would have been reported by cohort age \(a\). It lies between 0 and 1; a small value indicates that many positive outcomes may still be unreported.

For this tutorial, let \(N\) be the number of eligible cases in a cohort, \(O(a)\) the number of positive outcomes recorded by age \(a\), and \(U\) the eventual number of positive outcomes. The outcome is treated as binary, and a case is included in \(N\) whether or not its label has arrived. The maturity fraction concerns the reporting of eventual positive outcomes, not the fraction of all cases with any label at all.

The Basic Maturity Adjustment

If the maturity fraction is known or estimated from comparable historical cohorts, a simple adjustment is to divide the observed positive count by that fraction. The corresponding estimated positive rate uses the full cohort size as its denominator. This estimates the eventual total, rather than merely the outcomes already reported.

\widehat{U}(a)=\frac{O(a)}{c(a)}, \qquad \widehat{r}(a)=\frac{\widehat{U}(a)}{N} =\frac{O(a)}{N\,c(a)}.

The estimate is meaningful only if the maturity fraction is suitable for the cohort and age being assessed. In particular, historical reporting patterns should be comparable in their outcome definition, data capture, operational process, and relevant population. The adjustment also assumes that an eventual positive outcome has the same chance of being reported by age \(a\) across the cases to which the estimate is applied. A maturity fraction estimated from a different process may systematically overstate or understate the eventual count.

Assumptions and limits: The adjustment treats \(c(a)\) as the fraction of eventual positive outcomes reported by age \(a\), assumes the cohort’s reporting pattern is adequately represented by the historical data used to estimate \(c(a)\), and assumes the cohort denominator \(N\) is known. If \(c(a)=0\), the adjustment is undefined. If \(c(a)\) is very small, dividing by it can amplify small count errors and produce an unstable estimate. A maturity-adjusted count is an estimate, not a count of confirmed outcomes.

Why the Division Works

The adjustment follows from the expected number of reported positives under a simple reporting model. Suppose there are \(U\) eventual positive cases, and each has probability \(c(a)\) of having been reported by age \(a\). If reporting indicators are independent conditional on the eventual positive cases, then \(O(a)\) has a binomial distribution with size \(U\) and probability \(c(a)\). Its expected value is \(U c(a)\). Solving this expectation relationship for \(U\) gives the adjustment.

E[O(a)\mid U]=U c(a) \quad\Longrightarrow\quad \frac{E[O(a)\mid U]}{c(a)}=U.

Thus, when \(c(a)\) is known, \(O(a)/c(a)\) is unbiased for \(U\) under this model: its conditional expectation is \(U\). This justification does not guarantee that an individual estimate equals the eventual count. Nor does it account for uncertainty in an estimated maturity fraction. If the reporting process changes, or if historical cohorts are not comparable, the expectation relationship may fail. In those settings, a precise-looking adjusted value can still be misleading.

Worked Adjustments

Worked Example: A Recent Cohort with Partial Reporting

A hypothetical service has 1,000 eligible cases in a recent monthly cohort. By the monitoring date, 40 positive outcomes have been recorded. Historical cohorts indicate that 0.50 of eventual positive outcomes are typically reported by this cohort age. Estimate the eventual positive count and rate. Also compare the estimate with the raw recorded rate.

Method. Apply the maturity adjustment to the observed positive count, then divide the estimated count by all 1,000 eligible cases. This avoids treating unreported outcomes as confirmed negatives.

Step 1: Estimate the eventual positive count. Here \(O(a)=40\) and \(c(a)=0.50\), so \(40/0.50=80\). The estimated eventual count is 80 positive outcomes.

Step 2: Calculate the adjusted rate. Divide the estimated count by the cohort size: \(80/1{,}000=0.08\), or 0.0800 to four decimal places.

Step 3: Calculate the raw recorded rate. The raw rate is \(40/1{,}000=0.04\), or 0.0400 to four decimal places. It counts only the outcomes recorded so far, so it is not directly comparable with a fully mature rate.

Check. If 0.50 of 80 eventual positive outcomes have been reported, the expected reported count is \(80(0.50)=40\), matching the observed count. The adjusted estimate is 80 positives and a rate of 0.0800; the raw recorded rate is 0.0400. The common mistake is to report the raw rate as if every case already had a final label.

Worked Example: Comparing Cohorts at Different Ages

Two hypothetical cohorts each contain 800 eligible cases. Cohort A has 32 recorded positive outcomes at an age with maturity fraction 0.40. Cohort B has 24 recorded positive outcomes at an age with maturity fraction 0.20. Estimate the eventual positive count and rate for each cohort.

Method. Adjust each count using its own maturity fraction before comparing the rates. A common age is not required if the maturity estimates are appropriate for each cohort’s actual age.

Cohort A. The estimated eventual count is \(32/0.40=80\). Dividing by 800 gives \(80/800=0.10\), or 0.1000 to four decimal places.

Cohort B. The estimated eventual count is \(24/0.20=120\). Dividing by 800 gives \(120/800=0.15\), or 0.1500 to four decimal places.

Check. The implied reported counts are \(80(0.40)=32\) for A and \(120(0.20)=24\) for B, matching the data. The adjusted rates are 0.1000 and 0.1500. The raw rates, \(32/800=0.0400\) and \(24/800=0.0300\), would suggest the reverse ordering; that contrast is not a sound comparison of eventual rates because the cohorts have different maturity. The common mistake is to use one cohort’s maturity fraction for both.

Worked Example: Pooling Cohorts of Different Sizes

A hypothetical monitoring report combines two cohorts. Cohort A contains 600 cases, has 18 recorded positives, and has maturity fraction 0.60. Cohort B contains 400 cases, has 18 recorded positives, and has maturity fraction 0.30. Calculate the pooled adjusted rate.

Method. First estimate each cohort’s eventual count, then add those counts and divide by the combined number of eligible cases. Do not take an unweighted average of the cohort rates when the cohort sizes differ.

Step 1: Adjust cohort A. Its estimated count is \(18/0.60=30\). Its adjusted rate is \(30/600=0.05\), or 0.0500 to four decimal places.

Step 2: Adjust cohort B. Its estimated count is \(18/0.30=60\). Its adjusted rate is \(60/400=0.15\), or 0.1500 to four decimal places.

Step 3: Pool using the full exposure denominator. The combined estimated count is \(30+60=90\), and the combined cohort size is \(600+400=1{,}000\). Therefore the pooled adjusted rate is \(90/1{,}000=0.09\), or 0.0900 to four decimal places.

Check. The exposure-weighted rates give the same result: \((600/1{,}000)(0.05)+(400/1{,}000)(0.15)=0.03+0.06=0.09\). The unweighted average, \((0.05+0.15)/2=0.10\), is not the pooled rate because it gives equal weight to cohorts with different sizes. The common mistake is to average rates without considering their denominators.

A Practical Monitoring Process

A useful monitoring record makes the adjustment auditable. It should retain the observed count and the maturity-adjusted estimate separately, document the maturity fraction and its source, and state when the cohort’s age was measured. The following sequence helps prevent an estimate from being mistaken for a finalized outcome result.

Define the cohort and outcome.
Specify the eligible cases, cohort start date, positive outcome, and denominator. Keep the definition consistent with the intended monitoring question.

Set the data cut-off and cohort age.
State the date through which labels are observed and calculate elapsed time from the cohort’s start point. Comparisons without a clear cut-off can mix different amounts of reporting time.

Choose and check the maturity fraction.
Estimate it from sufficiently developed historical cohorts with comparable definitions and processes. Check whether reporting practices or data capture changed.

Show observed and adjusted results together.
Report \(O(a)\), \(c(a)\), \(\widehat{U}(a)\), and \(\widehat{r}(a)\). Label adjusted values as estimates and identify recent cohorts as provisional.

Revisit the estimate as labels arrive.
Refresh observed outcomes and the applicable maturity estimate at later cut-offs. Preserve earlier snapshots so readers can see how the estimate changed as evidence accumulated.

Interpretation, Uncertainty, and Common Traps

A maturity adjustment addresses one narrow issue: some eventual positive outcomes have not yet been reported. It does not establish that a predictive model is performing well or poorly, and it does not repair a changed outcome definition, biased observation process, or unsuitable comparison population. The report should distinguish confirmed observed outcomes from estimated eventual outcomes, and explain the assumptions behind any adjustment.

The maturity fraction itself is often estimated rather than known. It can vary across periods or operational settings, and a small historical sample may make it imprecise. As a result, \(\widehat{U}(a)\) has uncertainty from both the observed count and the maturity estimate. Dividing by a small \(c(a)\) magnifies the effect of either source: for example, a change of one observed positive changes the estimated count by \(1/c(a)\). Do not present a maturity-adjusted number with a level of certainty unsupported by the data.

When historical cohorts show materially different reporting patterns, a single pooled maturity fraction may not be defensible. Investigate whether the difference reflects elapsed time, outcome definition, reporting channel, case mix, or a process change. If the monitoring data do not support a reliable maturity estimate, report the observed count and explicitly state that the eventual outcome rate is not yet estimable with confidence. That is more informative than applying an inappropriate adjustment.

Exam trap: Dividing observed positives by the cohort size gives the raw recorded rate, not an estimate of the eventual positive rate when labels are incomplete. The maturity fraction applies to the positive count: first calculate \(O(a)/c(a)\), then divide by \(N\). Also avoid confusing a maturity fraction with the fraction of cases already labeled unless those quantities have been shown to be equivalent for the stated outcome and process.

Check Your Understanding

For each problem, treat the stated maturity fraction as appropriate for the cohort and assume the cohort denominator is complete. Give rates to four decimal places.

A cohort has 300 eligible cases. By the cut-off, 21 positive outcomes are recorded, and the maturity fraction is 0.70. Estimate the eventual positive count and rate. (Answer: 30 positives; rate 0.1000.)
A cohort has 1,200 cases, 36 recorded positives, and maturity fraction 0.30. Calculate the adjusted count and rate. (Answer: 120 positives; rate 0.1000.)
A cohort has 400 cases, with 12 recorded positives at maturity fraction 0.40. Calculate the adjusted rate. (Answer: 30 estimated positives; rate 0.0750.)
Two cohorts have 500 and 1,500 cases. Their recorded positive counts are 20 and 30, and their maturity fractions are 0.50 and 0.25, respectively. Calculate the pooled adjusted rate. (Answer: 160 estimated positives out of 2,000 cases; rate 0.0800.)
For a cohort of 1,000 cases, 50 positives are recorded when the applicable maturity fraction is 0.50. At a later cut-off, 72 are recorded and the applicable maturity fraction is 0.80. Calculate each cut-off’s adjusted count and rate, and explain why they differ. (Answer: First cut-off, 100 positives and rate 0.1000; later cut-off, 90 positives and rate 0.0900. The estimates differ because the observed count and maturity estimate changed; the later adjusted estimate is not required to equal the earlier one.)

What You'll Learn