Outcome Measurement Under Unequal Observation

Why Unequal Observation Matters

An emulated trial can define the same follow-up window and outcome for every treatment strategy while still observing those outcomes differently. One strategy may involve frequent clinic visits and routine laboratory testing; another may involve less contact or more care outside the data system. A recorded difference in outcomes can then reflect a difference in health, a difference in observation, or both.

The issue is not merely missing values. An event may occur between visits and never be recorded, a test may be ordered more often for one group, or a diagnosis may be captured only when care occurs in a particular system. The outcome definition and ascertainment plan established in the earlier material on Outcome Definitions and Ascertainment must therefore be considered together: the target outcome is what the trial seeks to compare, while ascertainment describes how the data reveal it.

For a binary endpoint, the strategy-specific target risk is \(P(Y^a=1)\), where \(Y^a\) is the outcome that would occur under strategy \(a\). This tutorial focuses on whether the data ascertain that outcome comparably across strategies. It does not redefine the endpoint or change the causal estimand.

How Observation Can Distort a Risk

First consider a simple setting in which \(Y\) is measured perfectly whenever \(R=1\), but some outcomes are unobserved. Within strategy \(a\), the observed-case risk is \(P(Y=1\mid A=a,R=1)\). This equals the target risk only under conditions that make observed participants representative with respect to \(Y\), possibly after conditioning on measured history.

A useful identity makes the source of distortion explicit. Write \(r_a=P(R=1\mid A=a)\), \(p_{1a}=P(Y=1\mid A=a,R=1)\), and \(p_{0a}=P(Y=1\mid A=a,R=0)\). By the law of total probability,

Thus, an observed-case comparison substitutes \(p_{1a}\) for a mixture of risks among observed and unobserved people. Equal observation rates across strategies do not guarantee unbiased comparison: the unobserved outcome risks may differ, and the observed people can be selected differently. Conversely, unequal observation rates need not bias estimates if observation is independent of outcome within strategy.

Adjustment for Measured Predictors of Observation

Observation may be related to outcome through measured history. For example, symptoms recorded before the endpoint measurement may predict both a later visit and the endpoint. Let \(H\) denote the measured history available before the observation decision. A conditional observation assumption says that, within strategy and history, whether the outcome is observed carries no further information about its value.

Under these conditions, the full-data risk can be recovered by averaging observed conditional risks over the history distribution for the strategy. Equivalently, inverse-probability weighting gives observed people more weight when their history makes observation less likely. For a fixed strategy \(a\), let \(\pi_a(H)=P(R=1\mid A=a,H)\).

To prove the identity, condition on \(A=a,H\). Since \(\pi_a(H)\) is fixed given this history,

Because \(R\) is binary and \(Y\) and \(R\) are conditionally independent given \(A,H\), \(E[RY\mid A=a,H]=E[R\mid A=a,H]E[Y\mid A=a,H]=\pi_a(H)E[Y\mid A=a,H]\). Positivity permits division by \(\pi_a(H)\), so the conditional expectation equals \(E[Y\mid A=a,H]\). Taking expectation over \(H\) conditional on \(A=a\) yields the stated identity. \(\square\)

In a sample, an unnormalized weighted estimate for strategy \(a\) is the average of \(R_iY_i/\hat\pi_a(H_i)\) among people following that strategy, with the denominator chosen to represent the corresponding strategy population. A normalized weighted mean divides the sum of weighted observed outcomes by the sum of weights among observed people. These are finite-sample estimators of the same target under the assumptions; the normalization does not make incorrect observation probabilities correct.

Visit Schedules and Imperfect Detection

Some outcomes are not simply missing when no visit occurs. They may be missed or detected late. For example, an event defined by a laboratory threshold is detected only if a test is performed, while a diagnosis code may appear only after healthcare contact. In these cases, distinguish the target event from the recorded indicator. Let \(Y^\ast\) denote whether the data record the event. A model for \(R\), the chance of measurement, cannot by itself correct errors in \(Y^\ast\) when a measurement occurs.

A scheduled-visit outcome and a continuously recorded event can therefore require different ascertainment models. If the target is status at a fixed time, a test taken near that time may be needed. If the target is whether an event occurred at any point in a window, visit-only records can miss events between visits or locate onset imprecisely. Changing the endpoint to “event recorded at a visit” may make it observable, but it changes what is being estimated.

Diagnostics, Alternatives, and Limits

Compare observation processes across strategies before interpreting outcome contrasts. Useful checks include visit frequency and timing, proportion with an outcome measurement, type of contact (scheduled, symptom-driven, or external care), testing rates, and the availability of records across care settings. Examine these measures over follow-up and across measured histories related to both the outcome and observation. Large differences are warning signs, not proof of bias; similar rates do not prove comparable detection.

The main analytic alternatives answer different problems:

• Observed-case analysis uses only recorded outcomes. It is simplest, but generally needs observation to be independent of outcome within each strategy, or conditional on variables properly handled in the analysis.
• Standardization by measured history estimates the outcome risk within history strata and averages those risks over the target history distribution. It relies on the same key conditional exchangeability and positivity conditions as observation weighting.
• Inverse-probability weighting for observation reweights observed outcomes by the inverse probability of being observed given strategy and history. It can be convenient when the observation model is credible, but extreme weights signal weak positivity and can make estimates unstable.
• Validation or measurement-error approaches use additional information on missed events or test accuracy. These are needed when the recorded outcome is an imperfect measure, rather than merely unavailable for some participants.

A practical pitfall is to assume that equal follow-up duration means equal ascertainment. It does not: one strategy can generate more tests or more visits during the same window. Another is to weight by observation probability and conclude that all differential measurement has been corrected. The weighting identity requires outcome exchangeability conditional on measured history and correct measurement when observed; it does not repair unmeasured drivers of visits, false negatives, false positives, or an outcome definition that is only indirectly captured.

When observation probability is zero for a relevant history, no reweighting can recover its outcome risk from observed data alone. When probabilities are merely very small, estimated weights can be highly variable. Report the observation pattern, the conditioning variables, the weight distribution if weighting is used, and the limitations of the data capture system. This issue is distinct from the timing and causes of administrative end of follow-up or loss to follow-up, addressed separately.