Why Check the Preference Model?
A benefit-risk ranking can be calculated precisely and still be poorly supported. The elicited preferences may contain contradictions, the chosen value function may fail to represent those preferences, or one respondent may have disproportionate influence on the aggregate. These are different problems: a respondent can make internally consistent choices that a proposed value function fits poorly, and an overall model can fit reasonably well while one respondent strongly affects which treatment ranks first.
The checks in this tutorial concern the preference inputs and their implications for a ranking. They do not establish that treatment outcomes are estimated correctly, nor do they prove that a preference model is the only defensible one. The interpretation depends on the attributes, elicitation tasks, value scales, and population represented.
For a treatment \(T\) with attributes \(x_1,\ldots,x_K\), an additive model commonly has the form \(U(T)=\sum_{k=1}^K w_k v_k(x_k)\). Here \(v_k\) maps an attribute to a value scale and \(w_k\) represents its contribution to the aggregate, under the model's chosen scaling and weight conventions. The score is meaningful only relative to those conventions. A higher score ranks higher within the specified model; it is not, by itself, evidence of a causal effect or a universally preferred treatment.
Check Logical Consistency Before Measuring Fit
A basic check is dominance. If option A is at least as good as option B on every relevant attribute and strictly better on at least one, a respondent who prefers B has given a dominance-inconsistent answer, assuming the attributes and their directions were understood and no omitted consideration changes the comparison. Another check is transitivity: if a respondent prefers A to B and B to C, a strict preference for C over A forms a cycle.
These checks identify answers to inspect; they do not reveal why they occurred. A cycle can result from response error, fatigue, changing interpretation, framing, or genuine context-dependent preferences. Before treating it as evidence against a respondent, check the task wording, attribute direction, response recording, and whether the choices were made under comparable conditions. A stated tie or near-indifference should not be silently converted into a strict preference.
Worked Example: Finding a Dominance Violation and a Cycle
Suppose a hypothetical elicitation presents three treatment profiles. Higher benefit value and lower harm burden are preferred. On a common scale, A has benefit 8 and harm burden 2; B has benefit 7 and harm burden 2; and C has benefit 6 and harm burden 3. The respondent chooses B over A, A over C, and C over B.
A dominates B: A has higher benefit (8 > 7) and the same harm burden (2 = 2). Choosing B over A therefore violates dominance under the stated attribute interpretation. A is also preferred to C by the elicited answer, while C is preferred to B and B to A. These three choices form the cycle \(A \succ C \succ B \succ A\), so they are not transitive.
The appropriate conclusion is that these answers warrant review, not that the respondent must be excluded. If the recorded choice B over A was a data-entry error, correction is appropriate only after verification. If the responses are confirmed, the analysis should report the inconsistency and assess whether plausible ways of handling it change the treatment comparison.
Repeated or mirrored tasks can provide additional checks: materially different answers to the same trade-off may suggest instability, while consistent answers across different wordings support—but do not prove—reliability. A simple count of violations can summarize a response set, but its meaning depends on how many opportunities there were for violations and how difficult or repetitive the tasks were.
Assess Whether the Value Function Fits
A value function is often elicited at selected attribute levels and then used to assign values at other levels. To assess fit, compare elicited values with values predicted by the proposed function on the same scale. For observations \(y_i\) and fitted values \(\hat{y}_i\), the residual is \(e_i=y_i-\hat{y}_i\). Residuals show direction and size of local disagreement; a summary such as root mean squared error (RMSE) gives a typical discrepancy in the units of the elicited value scale.
RMSE gives equal weight to squared residuals and is sensitive to large discrepancies. It is not a universal pass/fail criterion: its scale, task design, and intended use matter. Inspect the residuals as well. A systematic pattern—for example, underprediction at both ends and overprediction in the middle—can indicate that a linear function is not capturing curvature. A small average error can conceal such a pattern, especially with few elicitation points.
Worked Example: Evaluating a Linear Value Function
In a hypothetical assessment, an attribute is rescaled from 0 to 1. The proposed linear value function is \(v(x)=x\). Elicited values at \(x=0,\frac{1}{3},\frac{2}{3},1\) are respectively \(0,0.40,0.75,1\). The linear model predicts \(0,\frac{1}{3},\frac{2}{3},1\). Assume the elicited and predicted values use the same scale and that these four tasks are the available fit checks.
The residuals are \(0\), \(0.40-\frac{1}{3}=0.06667\), \(0.75-\frac{2}{3}=0.08333\), and \(0\). The endpoint values agree, while both interior elicited values exceed the linear predictions. Thus the discrepancies have a pattern consistent with curvature rather than being evenly scattered around zero.
Using the unrounded residuals, the squared-error sum is \(0.06667^2+0.08333^2\), more exactly \((0.40-\frac{1}{3})^2+(0.75-\frac{2}{3})^2=0.01139\). Therefore,
The fit discrepancy is about 0.0534 value units on this scale. That number alone does not establish that the model is unacceptable; it signals the size of the typical discrepancy for these elicitation points. The interior pattern gives a reason to compare the linear form with a plausible curved or piecewise-linear form, while retaining the same scale and checking predictions at points not used to define the alternative.
A more flexible function will usually fit its elicitation points at least as closely as a restricted function, but fitting those points is not sufficient evidence of better predictive performance. With few elicitation tasks, extra flexibility can follow response noise. Where the assessment permits, reserve some tasks for validation or elicit additional points in regions where competing functions differ. Check that the fitted function respects the attribute's direction and any justified endpoint constraints; monotonicity is appropriate only when higher attribute levels have been defined as uniformly better.
Check Whether a Respondent Drives the Ranking
Influence analysis asks how much the result changes when a respondent's preference input is perturbed or omitted. A useful first diagnostic is leave-one-out analysis: recompute the aggregate preference input and treatment scores after omitting each respondent in turn, then compare score differences and rankings with the full analysis. This is a sensitivity analysis, not a formal test that a respondent is an outlier or should be removed. If respondents represent distinct patient groups, omission may also change the target population rather than merely reduce noise.
Worked Example: A Respondent Changes Which Treatment Ranks First
Consider three hypothetical respondents whose elicited preference parameter values are \(w=0.55,\ 0.65,\ 0.95\). For illustration, the treatment-score difference is defined as \(D=U(A)-U(B)=0.50w-0.35\); the \(0.50w\) term is the benefit contribution under this model, and \(-0.35\) is the fixed net contribution from the other modeled attributes. Positive \(D\) ranks A above B; negative \(D\) ranks B above A. Assume equal respondent weighting and no other uncertainty for this calculation.
With all respondents, the mean preference parameter is \(\bar{w}=(0.55+0.65+0.95)/3=2.15/3=0.7167\). Using the unrounded mean, \(D=0.50(2.15/3)-0.35\approx0.00833\). The aggregate score therefore ranks A slightly above B.
Omitting the respondent with \(w=0.95\) gives \(\bar{w}=(0.55+0.65)/2=0.60\), and \(D=0.50(0.60)-0.35=-0.0500\), so B ranks above A. For comparison, omitting \(w=0.55\) gives \(\bar{w}=0.80\), \(D=0.0500\), and A remains above B; omitting \(w=0.65\) gives \(\bar{w}=0.75\), \(D=0.0250\), and A remains above B.
The respondent with \(w=0.95\) is influential for this ranking: omitting that respondent reverses the ordering, while omitting either other respondent does not. This result does not show that the high value is erroneous. It shows that the treatment ordering is sensitive to that input and should be reported as such, with the respondent's elicitation checked and the full-data and sensitivity results presented.
Influence should be evaluated on the decision-relevant output, not only on the parameter itself. A large change in a weight may have little impact if treatments are similar on the corresponding attribute; a small change can alter the ranking when the score difference is already close to zero. Report score differences alongside ranks, because a rank change between nearly tied options has a different practical interpretation from a large change in their score separation.
Putting the Checks Together
A defensible review follows a transparent sequence. First verify task coding, scale direction, and the intended preference model. Then identify logical inconsistencies, quantify and inspect value-function residuals, and assess respondent influence on the treatment scores. If a concern is found, compare prespecified, substantively plausible alternatives—for example, correcting a verified recording error, using a different justified function, or presenting results with and without an influential respondent. Do not choose whichever analysis produces the preferred treatment ranking.
Confirm attribute definitions, directions, units, response coding, and the scale on which elicited values and fitted values are compared.
Review dominance, cycles, repeated-task agreement, residual magnitudes, and residual patterns. Distinguish a verified data error from a confirmed but inconsistent judgment.
Recalculate treatment scores under defensible alternative preference inputs and leave-one-out analyses, then report both the score changes and any ordering changes.
These diagnostics have limits. Elicited answers are imperfect measurements of preferences; observed inconsistency does not uniquely identify its cause. RMSE depends on the chosen scale and elicitation design, and a good in-sample fit does not guarantee accurate predictions at unelicited attribute levels. Leave-one-out results can be unstable in a small sample and can be misleading if respondents are intentionally sampled to represent different perspectives. No single threshold for residual size or influence applies to every assessment.
The main alternative to a single fitted preference model is not to ignore fit, but to compare plausible model forms or report a range of preference scenarios. A single model gives a concise ranking conditional on its assumptions. Sensitivity analyses make that conditionality visible. If rankings agree across reasonable models and respondent perturbations, the preference-based ordering is more robust to the examined choices; if they differ, the disagreement is a substantive result to communicate rather than a nuisance to conceal.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A respondent prefers A to B and B to C, but C to A. What inconsistency is present, and what should be concluded?
- For elicited values \(0.20, 0.50, 0.90\), suppose fitted values are \(0.25, 0.45, 0.80\). Calculate the RMSE.
- Why should residuals be inspected in addition to RMSE?
- A leave-one-out analysis changes the score difference but not which treatment ranks first. Is the respondent influential for the ranking?
- Why does a treatment ranking that changes under one plausible value function require cautious interpretation?
Solutions. (1) The choices form a cycle, violating transitivity. Review the tasks and responses; the cycle alone does not establish why it occurred or justify automatic exclusion. (2) Residuals are \(-0.05,0.05,0.10\), so \(\operatorname{RMSE}=\sqrt{(0.0025+0.0025+0.0100)/3}=\sqrt{0.005}=0.07071\) value units. (3) Different residual patterns can have the same RMSE; a systematic pattern may indicate model misspecification. (4) The respondent affects the score difference, but does not change the ordering in this analysis. Whether that influence matters depends on the size and decision relevance of the change. (5) The ordering depends on the selected representation of preferences, so report the alternative results and avoid presenting the ranking as robust to that modeling choice.