Decision-Relevant Estimands for Dosing Evaluation

From a Dosing Question to an Estimand

A dosing policy can be judged by several different outcomes. It might increase the chance of reaching a therapeutic target, change the risk of toxicity, and require more visits or laboratory tests. Those outcomes do not automatically reduce to one measure of “success.” Before evaluating a policy, specify what population the conclusion concerns, which policies are compared, what outcome is measured, and over what period.

The estimand is the precisely defined quantity the evaluation is intended to learn about. It is not the observed result in a particular sample, and it is not a performance statistic for a prediction model alone. A dosing policy includes the rule that turns available information into a dose recommendation, along with any implementation features included in the evaluation, such as safety overrides. The estimand concerns outcomes under that policy as a whole.

For notation, let \(\pi_1\) denote the policy being evaluated and \(\pi_0\) the comparator. Let \(Y_i(\pi)\) be the outcome that person \(i\) would have under policy \(\pi\), and let \(E_P\) denote an average over the stated target population \(P\). The two population-level outcomes, \(E_P[Y_i(\pi_1)]\) and \(E_P[Y_i(\pi_0)]\), are not generally both observable for the same person. Their contrast is a policy effect on the chosen outcome.

The sign of the contrast should match the outcome’s meaning. For a favorable outcome such as reaching a therapeutic target, define benefit as the policy-minus-comparator difference. For an undesirable event such as toxicity, define excess harm in the same direction. For a burden measure such as the number of visits, a policy-minus-comparator difference describes the additional burden.

A positive value is favorable for the benefit contrast, but unfavorable for excess harm and burden. Always state that direction alongside the result. These three contrasts form a set of decision-relevant outcomes; they need not be combined into a single score. Combining them requires an explicit, defensible way to value different outcomes, as in the earlier material on expected utility.

Specify Each Part Before Calculating

Population. Describe who is represented and in what clinical setting. “Patients receiving the drug” may be too broad if the intended use is limited to adults beginning treatment at a particular care setting. Eligibility criteria and a defined setting help make the target population interpretable. The population also determines how outcomes are averaged: a result for one population need not apply to another.

Policies and comparison. Describe both strategies sufficiently clearly that their actions can be distinguished. For example, one policy might use a model-based dose recommendation with a specified safety override; the comparator might use a standard dosing algorithm. If the override, monitoring plan, or information available to the policy is part of the intervention, specify it. A comparison that leaves important policy components ambiguous does not define a reproducible estimand.

Outcome and summary. Define what is measured, how it is summarized, and which direction is favorable. An outcome might be the proportion reaching a target by day 30, the risk of a specified toxicity by day 90, or the mean number of dose-related visits over six months. These are different estimands even if they are collected in the same study. Include units for continuous outcomes and define the event for binary outcomes.

Follow-up. State when outcome ascertainment begins and ends. A 30-day risk and a 90-day risk answer different questions. For outcomes that can change over time, the assessment window and summary rule matter: “target attainment” could mean attainment at a scheduled visit, attainment at any time in the first month, or sustained attainment. Name the intended definition rather than relying on a broad label.

Estimands Depend on Follow-Up and What Happens After a Recommendation

A policy evaluation needs a clear rule for events that happen after assignment. For example, a person may stop the medication, switch treatment, or receive a clinician-directed dose change. One possible question is the effect of assignment to each policy, allowing subsequent care to proceed as it does in practice. For that question, outcomes are followed and attributed to the originally assigned policy even after discontinuation. This is often called a treatment-policy strategy for handling such events.

Other questions are possible, but they answer different things. Excluding people who discontinue, or treating their outcomes as missing at discontinuation, changes the analysis target and may introduce bias if discontinuation is related to prognosis or toxicity. An estimand should make clear whether the intended outcome reflects the full policy pathway or a narrower hypothetical scenario. The chosen approach must match the clinical decision being evaluated.

Benefit, Harm, and Burden Are Distinct Outcomes

Treatment burden is not merely a less serious form of harm. It may include extra appointments, blood draws, dose adjustments, or time spent managing treatment. Choose a measure that can be defined and collected consistently, then report its units. A mean difference in visits has units of visits per person over the stated period; a difference in time has units such as hours per person. Avoid calling either one “burden” without saying what was counted.

These examples illustrate why a useful evaluation commonly reports a vector of estimands: one or more benefit outcomes, one or more harms, and relevant burdens. A single composite can hide trade-offs—for example, a favorable target-attainment contrast alongside an unfavorable toxicity contrast. A composite is interpretable only when its components and their relative weights are specified and justified.

Justification and Conditions for Interpretation

The basic policy contrast follows directly from the target population averages. For a favorable binary outcome, each person’s outcome under a policy is 1 if the event occurs within the stated window and 0 otherwise. The mean of these zero-one outcomes is the population event probability. Subtracting the comparator probability from the policy probability therefore gives the difference in population probabilities:

In a study, the corresponding arm-specific proportions estimate these population probabilities. Their difference estimates the defined contrast when the study represents the target population and the study design and outcome ascertainment support that interpretation. For a randomized study, randomization makes assignment independent of potential outcomes on average; with well-defined policies, adequate follow-up, and analysis consistent with the assignment estimand, the arm comparison can estimate the causal effect of assignment to those policies.

In nonrandomized data, causal interpretation additionally requires that relevant differences between people receiving each policy are adequately controlled, and that both policies are represented for the covariate patterns in the target population. These conditions are substantive, not guaranteed by writing down an estimand. Missing outcome data also require attention: if outcome availability differs by policy or prognosis, a complete-case comparison may not estimate the stated population contrast without further assumptions or methods.

Diagnostics, Alternatives, and Common Pitfalls

Before interpreting results, check whether the study actually matches the estimand. Confirm that eligibility and setting identify the intended population; policy components are described; the outcome definition and units are consistent; and follow-up covers the declared horizon. Check how discontinuation, switching, death, and missing measurements were handled. If the analysis omits people or events that the estimand includes, the reported calculation may target something else.

Absolute differences, relative risks, and mean differences are alternative summaries of policy contrasts, not interchangeable answers. An absolute risk difference is often directly interpretable as a change in event probability; a relative measure expresses the comparison on a ratio scale. State the chosen scale and, where useful, report more than one. Do not confuse a precisely estimated contrast with a clinically important one, or a population average with the experience of every patient. Uncertainty intervals describe sampling uncertainty around an estimate; they do not repair a poorly specified estimand or biased study design.

• Pitfall: reporting “the policy worked” from a model metric. A policy-level estimand concerns outcomes under the full dosing strategy, not prediction performance alone. Define and measure the clinical outcome under each policy.
• Pitfall: omitting the horizon. “Toxicity risk” is incomplete without an event definition and time window. State, for example, serious toxicity by day 90.
• Pitfall: changing the denominator after discontinuation. Excluding discontinuers when the target is the effect of assignment to a policy can change the question and bias the comparison. Follow outcomes in the population specified by the estimand or explain the alternative target and assumptions.
• Pitfall: treating benefit, harm, and burden as one self-evident score. These outcomes have different meanings and often different units. Report them separately unless a defensible weighting scheme is specified.

The estimand also has limits: it is specific to its population, comparator, outcome definition, and horizon. A result for one setting or follow-up period does not automatically establish the effect in another. Clear specification makes those limits visible and allows readers to judge whether the result is relevant to the dosing decision at hand.