Worked Benchmark With Complementary Omics Signal

Why Complementary Signal Calls for a Careful Comparison

Two omics blocks can each help predict an outcome for information the other block does not contain. That situation is different from having two blocks that largely repeat the same signal: combining distinct information may help, but the gain depends on how the integration strategy uses it. A useful benchmark therefore needs to specify the outcome mechanism, the predictions made by each strategy, and the evaluation metric.

This tutorial works through a hypothetical benchmark with two binary block summaries, \(X_A\) and \(X_B\), and a binary outcome \(Y\). The values are deliberately simple so that every score can be checked by hand. They illustrate a possible comparison, not a claim about a real study or a guarantee that one integration strategy will always win.

The last qualification matters. If \(X_B=X_A\) for every observation, then knowing \(X_B\) after \(X_A\) adds no information, regardless of whether a fitted formula assigns both variables nonzero coefficients. A support condition rules out this particular failure. Here we use the stronger, simple condition that \(X_A\) and \(X_B\) are independent Bernoulli variables with probability \(1/2\) for each value. All four combinations occur with positive probability, and each summary varies conditional on the other.

Strategies and Benchmark Assumptions

The three strategies differ in where they combine information. Early integration concatenates the two block summaries and fits a joint prediction rule. Intermediate integration first represents each block separately, then combines the retained representations in a joint prediction rule. Late integration fits a predictor for each block and combines their predicted probabilities. These are broad strategy descriptions; a benchmark must state the actual representations, model classes, and probability-combination rule it uses.

For this worked comparison, the early model has access to the pair \((X_A,X_B)\) and can assign a prediction to each of its four possible combinations. The intermediate strategy separately retains the binary indicator from each block and uses a joint downstream rule with access to both indicators; it can likewise assign a prediction to each combination. Late integration averages the two single-block predicted probabilities with equal weights. This deliberately controlled setup isolates what happens when full joint information is retained versus when predictions are averaged. It is not a claim that these models have equal flexibility in every application.

Assume the test observations are independent draws from the stated target distribution, the test outcomes are not used to fit or choose predictions, and all strategies are evaluated on the same observations. The fitted predictions below are fixed before evaluation. These assumptions make the score comparison interpretable as a held-out comparison in this hypothetical setting; a real benchmark also needs a defensible sampling and evaluation design.

Changing either block from 0 to 1 raises the conditional probability of the outcome by 0.40, whichever value the other block takes. For example, changing \(X_A\) from 0 to 1 changes the probability from 0.10 to 0.50 when \(X_B=0\), and from 0.50 to 0.90 when \(X_B=1\). The same comparisons hold for \(X_B\). These conditional contrasts, along with the independent, non-degenerate support, verify that both blocks carry predictive information not determined by the other.

Worked Example: Set Up the Evaluation Table

Deriving and Calculating the Brier Scores

For binary outcomes, the Brier score is the mean squared difference between each observed outcome and its predicted probability. For \(n\) test observations with outcomes \(y_i\) and predictions \(\hat p_i\), it is

The score ranges from 0 upward, with lower values indicating better probability predictions on the evaluated observations. It is a proper scoring rule: in expectation, the probability forecast that minimizes squared error is the true conditional outcome probability. To see why, let \(p=P(Y=1\mid X)\) and consider a fixed forecast \(q\). Since \(Y\) is binary, \(E(Y\mid X)=p\) and \(E(Y^2\mid X)=p\). Expanding the squared error gives

The term \(p(1-p)\) is the outcome’s conditional variability, which cannot be removed by choosing a forecast. The extra squared term is zero only when \(q=p\). This identity also helps separate irreducible outcome variation from prediction error.

What the Comparison Shows—and What It Does Not

The score differences can be stated directly in Brier-score units. Early integration improves on either single-block predictor by \(0.2100-0.1700=0.0400\); the late average improves on either by \(0.2100-0.1900=0.0200\). Early integration improves on late integration by \(0.1900-0.1700=0.0200\). These are descriptive differences on this hypothetical test set. They are not, by themselves, evidence that the same gains would occur in a new population.

The example also clarifies why complementary information does not imply that every fusion rule will fully benefit from it. The late average uses both block-specific predictions, but averaging discards which combination of the two block values produced those predictions. In the middle cells, for instance, the late average is 0.50; in this benchmark that agrees with the conditional outcome rate. In the corner cells, however, its predictions of 0.30 and 0.70 are less extreme than the joint probabilities of 0.10 and 0.90. The resulting loss in those cells is enough to make its overall Brier score higher than the joint strategy’s.

This is not a universal ranking. A different outcome mechanism, training sample, feature representation, model class, or probability-combination rule could change the results. In a real comparison, use the same target population, evaluation observations, and scoring definition across strategies; make model flexibility and tuning opportunities comparable enough that a performance difference can be interpreted; and report uncertainty rather than treating a small observed rank difference as decisive. The earlier material on measuring block contributions, strategy-by-scenario interactions, and Monte Carlo uncertainty provides useful framing for those comparisons.

Diagnostics and Common Pitfalls

• Check support, not just coefficients. If one block is constant or determined by the other, apparent contributions in a formula may not represent distinct information. Verify that relevant combinations occur and that conditional outcome probabilities vary with each block.
• Check the prediction rule and arithmetic. State exactly how late predictions are combined. Confirm each cell’s number of cases and controls before summing squared errors; a wrong multiplicity changes the score.
• Inspect cell-level results. A single overall score can hide a strategy that helps in some combinations and harms in others. Compare predictions and errors by block combination, especially when a gain appears unexpectedly large.
• Do not equate a lower score with a causal effect. These are predictive comparisons. A block’s added predictive information does not establish that changing that block would change the outcome.
• Do not interpret this table as a population certainty. It is one deliberately balanced hypothetical test set. In actual data, sampling variation and uncertainty in fitted predictions matter; the printed decimals alone do not quantify that uncertainty.

A common mistake is to reason, “Both blocks have nonzero coefficients, so each must add predictive information after the other is known.” This fails when the block summaries are dependent in a way that makes one determined by the other, or when the observed support contains no comparisons that distinguish their contributions. The correction is to examine the joint support and conditional outcome probabilities, as done here, before describing the signal as complementary.