Outcome Definitions and Ascertainment

Why Outcome Definitions Need More Than a Name

Terms such as “infection,” “relapse,” or “hospitalization” sound concrete, but they can refer to different events in different data systems. A laboratory-confirmed infection is not necessarily the same endpoint as an infection diagnosis in a billing record. A hospitalization may mean any admission, an admission for a prespecified cause, or an admission recorded in one particular health system. If the endpoint is not specified in operational terms, another analyst may not know which observations count.

In an emulated trial, the outcome definition should be fixed independently of how favorable it makes one treatment strategy appear. It should also make clear what the records can and cannot reveal. A recorded outcome is evidence about an event, not automatically a perfect record of the event itself.

Let \(Z_i\) denote the treatment strategy to which person \(i\) is classified for the comparison, and let \(Y_i\) indicate whether the prespecified event occurred within the outcome window. Let \(Y_i^*\) denote the outcome as classified from the available source. When ascertainment is complete, \(Y_i^*=Y_i\) for every person; when it is imperfect, some events may be missed or some non-events may be classified as events.

Build an Operational Definition

A useful specification answers three questions in enough detail that two analysts would apply it in the same way.

• What counts? State the event criteria, including any required diagnostic evidence, procedure, or adjudication rule. If a code list is used, identify the code set and whether one or repeated records are required.
• When does it count? Give the start and end of the measurement window relative to the trial’s time zero. State how the event date is determined and how an event on a boundary date is handled.
• Where is it measured? Name the data source, such as a linked hospital registry, electronic health records, claims, or scheduled study assessments. State whether sources are combined and how conflicts are resolved.

The follow-up window and endpoint timing are established in the earlier material on follow-up windows and trial endpoints. Here the emphasis is on making the event and its ascertainment rule explicit within that window. A database’s last available date, for example, is not itself evidence that no event occurred after that date.

Two endpoints can share a label but differ in criteria or capture. For instance, “stroke” identified by a discharge diagnosis code may include some cases that fail a clinical adjudication rule, while an adjudication rule may miss events never recorded in the linked hospitals. Neither source is automatically the correct choice in every setting. The choice should follow the clinical question and the data available, and the resulting endpoint should be described honestly.

How Unequal Ascertainment Changes Recorded Risks

Ascertainment may differ by strategy if participants receive different amounts or types of medical contact, if clinicians look for an outcome more often in one group, or if one strategy is more likely to lead to care outside the data system. Classification can also differ when reviewers know the treatment received. A comparison of recorded outcomes then mixes differences in event occurrence with differences in the chance that an event is observed or classified.

The following derivation isolates one important form of this problem. Within strategy group \(z\), define sensitivity as \(Se_z=P(Y^*=1\mid Y=1,Z=z)\) and specificity as \(Sp_z=P(Y^*=0\mid Y=0,Z=z)\). Write the event risk as \(p_z=P(Y=1\mid Z=z)\) and the recorded risk as \(q_z=P(Y^*=1\mid Z=z)\). Assume these probabilities refer to the same target population and window, and that the stated sensitivity and specificity apply to its events and non-events.

The first term counts true events correctly recorded as events. The second counts non-events incorrectly recorded as events. This identity is a direct partition by the true event status; it does not require sensitivity or specificity to be high.

Subtracting the two recorded risks gives a useful diagnostic decomposition. Since \(q_z=(Se_z+Sp_z-1)p_z+1-Sp_z\),

Thus, even if the true risks are equal, the recorded risks can differ when classification properties differ by strategy. Conversely, similar recorded risks do not establish similar true risks. A simple special case is perfect specificity, \(Sp_0=Sp_1=1\): then \(q_z=Se_zp_z\), so differential sensitivity alone can change the recorded comparison.

Incomplete capture is not the only issue. If the outcome is unknown for some people, the observed risk among those with recorded outcomes may differ from the group risk even when the outcome classification itself is accurate for observed people. To see the distinction, let \(R=1\) indicate that the outcome is ascertained. If ascertainment depends on event status within a strategy group, then among those with known outcomes,

This follows by applying Bayes’ rule to the event and non-event cases among \(R=1\). The observed-case risk equals \(p_z\) if the probability of ascertainment is the same for events and non-events within that group. If that condition fails, simply omitting unknown outcomes can shift the risk. This equation is a warning about interpretation, not a general correction method.

Assessing Whether Ascertainment Differs

Assessment starts with the process that produces the recorded outcome, not just a comparison of final event counts. Compare the strategies on features that can affect capture: frequency of clinical contact, availability of outside-network records, use of diagnostic tests, referral patterns, and completeness of linkage. Check whether the outcome criteria and source coverage are the same by group, and whether adjudicators could infer strategy from the records.

When feasible, a validation sample can be reviewed using a reference procedure applied in both groups. For example, records from both groups can undergo blinded clinical review, allowing sensitivity and specificity to be estimated separately. Such estimates are only informative for the populations and definitions represented in the validation sample; a small or selectively reviewed sample may not reveal important differences.

A related but distinct check is to tabulate the fraction of people with an outcome that is unknown, by strategy. That identifies a potential difference in completeness, but equal fractions do not guarantee equal ascertainment: the unknown outcomes could be concentrated among events in one group and among non-events in the other. Likewise, equal recorded event rates do not establish equal sensitivity or specificity.

Common Pitfall and Practical Limits

Wrong reasoning: “The same claims algorithm was applied in both groups, so ascertainment must be comparable.” Correction: applying the same rule does not ensure equal sensitivity or specificity. Participants in one group may be more likely to receive tests, be admitted to a linked hospital, or have their records captured. The algorithm can be identical while the information supplied to it differs.

The primary alternative to a routine-data endpoint may be a clinical adjudication process, a registry, or a different combination of sources. These approaches can improve event definition or capture, but they also require resources and may have their own errors. A code-based endpoint is efficient and reproducible when its performance is adequate; adjudication can be more clinically specific but may be subjective, incomplete, or costly. There is no universally superior source independent of the event and setting.

Diagnostics cannot prove that ascertainment is identical. If there is no validation information, no source able to detect events outside the routine system, and no plausible basis for comparing capture, the degree of differential error may remain unknown. State the operational definition and the known limitations rather than interpreting recorded outcomes as error-free events. Detailed methods for unequal observation and follow-up require separate treatment.