Eliciting Benefit-Risk Preferences from Stakeholders

Why Elicit Preferences Before Defining Dose Utility?

A dose-finding objective can include efficacy, toxicity, and the burden of receiving treatment, but choosing those outcomes does not by itself specify how they should be valued. A patient may consider a modest efficacy gain worthwhile only if the chance of a serious adverse event remains low. A clinician may focus on preventing that event, while another stakeholder may emphasize symptom relief or time spent receiving treatment. Structured elicitation makes these judgments explicit before they are encoded in a dose utility.

The aim is not to ask stakeholders to calculate a utility function. It is to collect clear, interpretable evidence about what matters, which tradeoffs are acceptable, and where opinions differ. The questions should describe treatment consequences in a consistent clinical context, and the responses should be recorded with enough detail to distinguish a firm preference from an uncertain or conditional one.

Earlier material on constructing joint benefit-risk utility functions establishes why a dose decision may need to account for more than one outcome. Here the emphasis is on how to ask stakeholders about those outcomes. The resulting evidence can inform which attributes to include, how to describe them, and which tradeoffs deserve further evaluation. It should not be presented as though a small set of answers establishes a single preference shared by all patients.

Designing Questions That Stakeholders Can Answer

Start with the decision the utility will support. Specify the disease, treatment setting, relevant patient population, comparator, and time horizon. For example, a question about the chance of nausea during the first treatment cycle is not interchangeable with a question about persistent nausea over a year. Similarly, “more effective” is too vague unless the respondent knows what outcome improves, by how much, and over what period.

Then define the attributes that will appear in the scenarios. Efficacy might be expressed as the chance of symptom improvement by a stated time; toxicity might be the chance of a named adverse event with a clear severity and duration; treatment burden might include infusion time, visit frequency, or daily dosing. Use units and absolute risks where possible. Keep the wording understandable, and do not imply that a modeled outcome is certain or guaranteed.

A useful sequence moves from comprehension to comparison. First ask the respondent to explain, in their own words, what each outcome means. Next ask which outcomes matter most in the stated decision. Then use carefully varied scenarios to explore tradeoffs. Finally, ask how certain the respondent is and what information, assumption, or personal circumstance influenced the answer. Open-ended follow-up questions are valuable because a choice alone rarely explains its rationale.

Avoid leading language. “Would you accept a dangerous treatment for a chance of benefit?” steers the respondent through emotionally loaded wording. A more neutral version describes the outcomes concretely and asks which option they would choose, or whether they regard either as preferable. Avoid combining several changes in one comparison: if efficacy, toxicity, and visit burden all change at once, the reason for a preference is hard to identify.

Useful Elicitation Formats

• Direct attribute questions. Ask which outcomes are important, what makes an outcome unacceptable, and what treatment burden is manageable. These questions help identify relevant attributes and concerns, but answers such as “toxicity matters most” do not specify how much toxicity a person would trade for a particular benefit.
• Controlled pairwise comparisons. Present two profiles that differ in one attribute, or in a clearly defined tradeoff between two attributes. Ask which is preferred, whether they seem equally desirable, and how confident the respondent is. Pairwise questions are often easier to understand than asking for numerical weights.
• Threshold or boundary probes. Hold the benefit description fixed and vary a harm, or hold the harm fixed and vary a benefit. Ask at which point the respondent would switch preference, or whether there is a range where neither option is clearly favored. This reveals a boundary or interval, not necessarily a precise point.
• Acceptability questions. Ask whether a profile would be acceptable, unacceptable, or acceptable only under specified conditions. Follow up on the conditions. Acceptability is a useful clinical judgment, but it is not automatically a hard safety constraint or a complete ordering of all possible options.
• Open-ended explanation. Ask what drove the response and whether it depends on personal circumstances, such as disease severity, prior experience, or caregiving responsibilities. These explanations can expose missing attributes and important sources of variation.

Ranking several profiles can provide a quick view of broad priorities, but a rank order alone says little about the size or stability of differences in preference. It should be treated as one form of preference evidence rather than as a complete account of tradeoffs. Converting ranks into numerical utility weights is a separate task.

Locating a Preference Boundary

When a respondent’s preference changes as one attribute varies, a sequence of questions can locate a boundary more clearly than a single comparison. For example, keep the expected symptom-improvement rate fixed and vary the severe-event risk. The respondent can indicate which profile they prefer at each level. A boundary is the level or range where the respondent switches from favoring one profile to favoring the other, or reports that neither is clearly preferred.

This approach relies on an important assumption: over the range tested, the respondent’s preference changes in an orderly direction as the varying attribute changes. If increasing harm risk while holding everything else fixed makes an option more desirable for some responses, the answers may reflect misunderstanding, random response, a changed interpretation, or a genuine non-monotone preference that requires investigation. Do not force a single threshold if the data do not support one.

Comparing Stakeholder Groups and Checking Response Quality

Patients, clinicians, caregivers, and policy stakeholders may answer from different roles and experiences. Elicit and summarize these perspectives separately before considering how they should inform a decision. A clinician’s judgment about the consequences of toxicity is not the same as a patient’s experience of that toxicity. Likewise, a group average can conceal a minority for whom a treatment burden is especially important.

Use a common core of questions when comparing groups, while allowing appropriate follow-up questions for each group. Record who responded, the context of the interview, which profiles were shown, and the wording and order of questions. If respondents see multiple comparisons, order can influence later answers: a severe scenario presented first may make subsequent risks seem less concerning. Pilot the questions with people similar to the intended respondents and revise confusing terms before the main elicitation.

Useful diagnostics include frequent requests for clarification, inconsistent choices across near-identical scenarios, missing responses, extreme answers that appear to reflect a misunderstanding, and large changes when wording or order changes. Examine these signals by respondent as well as by stakeholder group. A response can be valid but uncertain; it should not be discarded simply because it differs from the majority. Instead, retain uncertainty and explain how it affects interpretation.

From Elicited Judgments to a Defensible Utility Specification

Preference evidence helps the analysis team make transparent modeling decisions. It can identify which efficacy, toxicity, and burden outcomes should be represented; highlight outcomes that require clinically meaningful categories; and show where a single summary may obscure important differences. It can also reveal that stakeholders disagree about the decision itself, not merely about a numerical input.

Elicitation does not replace clinical evidence. Respondents describe how they value stated consequences; trial and model data describe how likely those consequences are at each dose. Keep the two kinds of evidence distinct. Likewise, an elicited statement that a particular profile is unacceptable should prompt careful examination, but the team must state separately whether it is being used as a preference, a decision rule, or a formal safety constraint.

For traceability, document the stakeholder groups, recruitment context, question wording, attribute definitions, time horizon, scenario values, response options, follow-up explanations, and any revisions made after piloting. Report uncertainty and disagreement rather than hiding them behind a single consensus label. A defensible utility specification makes clear which preference evidence informed its structure and where judgments remain unresolved.

Common Pitfalls

• Wrong reasoning: “Most respondents ranked efficacy first, so efficacy should receive the largest numerical utility weight.”
  Correction: A ranking indicates ordering, not the magnitude of tradeoffs. Ask targeted comparisons and retain the distinction between stated priority and a calibrated utility weight.
• Wrong reasoning: “The respondent accepted a 10% event risk, so 10% is safe for every patient.”
  Correction: The answer applies to the respondent and profile described. A preference judgment is not a population safety conclusion or a universal clinical limit.
• Wrong reasoning: “The two groups disagree, so one group must have misunderstood.”
  Correction: Check comprehension, but also allow that preferences, experiences, and decision roles genuinely differ. Report the difference and its uncertainty rather than forcing agreement.
• Wrong reasoning: “A profile with more efficacy and more toxicity tells us which outcome drove the choice.”
  Correction: Unless the comparison isolates the attributes or includes follow-up, the response reflects the combined profile and cannot identify the reason on its own.
• Wrong reasoning: “A single boundary is precise because the respondent chose one side at each question.”
  Correction: Choices can be uncertain, context-dependent, or inconsistent. Report the tested range and confidence, and do not claim greater precision than the responses justify.