Weighting Evidence From Unequally Reliable Models

Why Evidence Components Should Not Have Equal Influence

An integrated development argument may draw on clinical exposure-response analyses, mechanistic models, biomarker-outcome relationships, and trial simulations. Their predictions can differ, but even when they address the same quantity, the components need not be equally informative. One may perform well in relevant external validation; another may be weakly validated, uncertain, or based on a population unlike the one informing the decision.

The task here is to assign a defensible relative influence to components when evaluating an integrated evidence argument. This is not a universal rule for combining numerical predictions. A weight should explain why one component deserves more or less consideration for a particular target decision. It is conditional on the target outcome, population, time horizon, and evidence available.

The earlier material on adjudicating discordant model predictions motivates first identifying what differs and whether the predictions concern the same target. Weighting is appropriate only after that target is stated clearly. If components address different estimands, populations, or time points, their apparent disagreement may not be comparable evidence about one quantity.

A Structured Influence Score

A useful starting point is to assess four dimensions separately: validation performance, relevance to the target context, uncertainty in the component’s target prediction, and dependence on evidence already represented elsewhere. Keeping these dimensions visible makes it easier to explain a weight and to find out which judgment drives it.

For a concrete operationalization, let \(L_j\) be a predictive validation loss for component \(j\), and let \(L_{\mathrm{ref}}\) be the loss from a pre-specified reference prediction evaluated on the same outcome scale and validation cases. Define \(q_j=1/(1+L_j/L_{\mathrm{ref}})\). Lower loss gives a larger factor. The reference must be stated: choosing a convenient or weak reference after seeing results can make performance appear stronger than it is. A proper predictive scoring rule, such as a suitable continuous-outcome score, should be selected for the prediction being assessed.

Set \(r_j\) using explicit criteria for similarity to the target population, treatment, outcome definition, and time horizon. For example, a rubric may assign values near 1 when validation and target settings are closely aligned and lower values when important differences remain. Set \(u_j\) to a measure of uncertainty for the component’s prediction on the target scale, including relevant parameter and model-discrepancy uncertainty. This is not the same as individual-level variability. With a pre-specified reference uncertainty \(u_{\mathrm{ref}}\), one possible bounded precision factor is \(p_j=u_{\mathrm{ref}}^2/(u_{\mathrm{ref}}^2+u_j^2)\). This gives less influence to a more uncertain prediction, all else equal.

The dependence discount \(d_j\) represents how much distinct information the component contributes, given evidence already counted. It can be assigned using a documented assessment of shared data, shared assumptions, or common model structure. It is not a substitute for representing dependence in an integrated simulation. Earlier material on preserving dependence, parameter covariance, and preventing double counting is relevant here: correlated components should not be treated as independent confirmations.

This framework assumes that components can be assessed against a common target; validation losses and prediction uncertainties are meaningfully comparable after appropriate scaling; the relevance rubric is specified for that target; and dependence is considered explicitly. If these conditions are not met, a single set of weights may conceal rather than resolve the mismatch. The factor formulas and rubric should be fixed before reviewing the final ranking where feasible, and the results should be checked under reasonable alternatives.

How Normalization Works

The normalized weights sum to one whenever the total score is positive. This property makes the relative influence allocation easy to inspect, but it does not turn the allocation into probabilities. The justification is algebraic: divide each nonnegative score by the same positive total, then sum the resulting ratios.

Thus, a component’s weight depends on its score relative to the other components included. Adding a new component can change all normalized weights even when the original scores remain unchanged. A weight of 0.6 means that this scoring framework assigns 60% of the normalized influence allocation to that component; it does not mean there is a 60% probability the component is correct or that its prediction should automatically contribute 60% to a pooled prediction.

Diagnostics, Alternatives, and Limitations

A numerical score can look more objective than the judgments behind it. Diagnose the weighting before relying on it:

• Check validation alignment. Examine whether validation cases represent the target population, treatment, outcome, and time horizon. Good performance in a different setting does not establish target-context reliability.
• Inspect uncertainty meaning. Verify that uncertainty measures refer to comparable target predictions and include relevant knowledge uncertainty. A narrow interval alone does not establish accuracy.
• Audit dependence. Trace shared data, parameters, structural assumptions, and interfaces. Two model components may not constitute two independent confirmations.
• Run sensitivity analyses. Recalculate the ranking under plausible relevance ratings, uncertainty references, validation benchmarks, and dependence discounts. If small reasonable changes reverse the ranking, report that instability.
• Look for unrepresented evidence. A factor set cannot repair omitted model discrepancy, poor validation, or a mismatch in the target estimand. A low score should prompt examination, not automatic deletion.

Equal weighting is easy to explain but assumes equal relative influence without distinguishing performance, relevance, uncertainty, or overlap. Performance-only weighting ignores transport to the target context. Inverse-variance weighting is most defensible for estimates of the same quantity when they are unbiased and appropriately independent; it does not by itself handle relevance or shared evidence. Expert elicitation can make contextual judgments explicit, but should record the rubric and uncertainty in those judgments. Model averaging is a different task: it concerns combining predictions across models, whereas the weights here are a framework for assessing relative evidence influence and do not prescribe a pooled prediction.

The scoring framework also has limitations. Its factors may be correlated: for example, poor context relevance can already be reflected in validation performance, and uncertainty may partly reflect model discrepancy. Multiplying factors can then penalize the same weakness twice. The numeric result can also depend on scale choices and rubric anchors. Report the component factors alongside the normalized weights, explain why each factor was chosen, and treat a weight as a summary of a documented argument rather than as a measurement free of judgment.