Why the Data Generator Must Be Separate From the Method
A method comparison is only interpretable if it is clear what data were generated and what each method was asked to do. In a multi-omics setting, one subject may contribute measurements in several feature blocks and one outcome. A simulation therefore needs to describe both the joint distribution of those measurements and the process that produces the outcome. The integration methods are then applied to data from that specified process; they are not part of the process itself.
This distinction matters because a simulation can favor a method before analysis begins. For example, if the generator uses a particular model’s assumptions to produce data, that method may be especially well matched to the simulation. That result can still be informative, but it does not establish that the method is generally superior. A useful benchmark makes its assumptions explicit and states which target—such as prediction of new subjects—is being evaluated.
Write Down the Subject-Level Data Structure
Let \(n\) be the number of subjects and \(K\) the number of omics blocks. For subject \(i\), write \(X_i^{(k)}\) for the feature vector in block \(k\), where \(i=1,\ldots,n\) and \(k=1,\ldots,K\). Block \(k\) has \(p_k\) features, so \(X_i^{(k)}\) is a vector of length \(p_k\). Write \(Y_i\) for the outcome and \(C_i\) for any additional subject-level covariates included in the design.
In a complete, matched simulation, the same subject index \(i\) refers to the measurements and outcome belonging together. The full data for that subject can be represented as \((C_i,X_i^{(1)},\ldots,X_i^{(K)},Y_i)\). “Matched” here describes the subject-level alignment; it does not imply that all blocks have identical features or that their values are generated by the same distribution.
A precise specification must also say whether the simulation includes incomplete blocks, measurement error, or a sampling mechanism. If the goal is a complete-data benchmark, state that all \(K\) blocks and the outcome are observed for every subject. If missingness is included, specify which values can be missing and how that event is generated. Leaving the matter unstated makes the benchmark ambiguous.
Specify a Modular Generative Model
A convenient way to organize a DGP is to state its joint distribution using a factorization. One general form is
This factorization is a specification of the order and conditional distributions used to generate a subject’s data. The middle term is a joint conditional distribution: it must allow whatever dependence among blocks the chosen DGP intends to include. It does not assert that blocks are independent. If blocks are generated independently conditional on \(C_i\), that is an additional assumption and should be stated, not inferred from the notation.
The outcome model is another conditional distribution. Its form depends on the outcome and the scientific or methodological question. A simulation might specify a continuous outcome using a conditional mean and an error distribution. Other outcome types require a corresponding probability model. In every case, specify the parameters, units or coding, and random variation. The DGP should be complete enough that a reader could implement it without guessing.
At the subject level, a common simulation convention is to draw independent subjects from the stated joint distribution. This means the subject-level data vectors are identically distributed and independent across \(i\). That convention is not automatic: repeated measurements, families, sites, or batches may induce dependence across subjects. If the benchmark includes such structure, it belongs in the DGP.
Worked Example: Generating Two Blocks and a Continuous Outcome
Consider a hypothetical complete dataset with \(n=3\) subjects and \(K=2\) blocks, each containing one feature. For this illustration, let the block features be fixed by the following chosen draws: \(X_i^{(1)}=(0.5,-1,1.5)\) and \(X_i^{(2)}=(1,0,-0.5)\). Let the outcome follow
with independent errors across subjects. Suppose the realized errors are \((0.1,-0.2,0)\). These values are hypothetical draws, not measurements from a study.
For subject 1, substitution gives \(Y_1=1+2(0.5)-1+0.1=1.1\). For subject 2, \(Y_2=1+2(-1)-0-0.2=-1.2\). For subject 3, \(Y_3=1+2(1.5)-(-0.5)+0=4.5\). Thus the three matched rows are \((0.5,1,1.1)\), \((-1,0,-1.2)\), and \((1.5,-0.5,4.5)\), in the order \(X^{(1)},X^{(2)},Y\).
The specification says that the outcome error is normal with mean 0 and variance 1, so its standard deviation is 1 outcome unit. It also says the displayed block values are fixed inputs for this small arithmetic illustration. To make a full repeated simulation, a feature-generation distribution would additionally be needed; without one, these feature values alone do not define how new block data are generated.
Check the Model’s Implications Before Comparing Methods
A DGP should imply coherent distributions and scales. As a simple check, suppose \(X_i^{(1)}\) and \(X_i^{(2)}\) are independent, each has mean 0 and variance 1, and the outcome follows the same linear equation as in the example, with independent \(\eta_i\sim N(0,1)\). These are assumptions for this calculation, not properties guaranteed by the equation alone.
The conditional mean given the two feature values is \(E(Y_i\mid X_i^{(1)},X_i^{(2)})=1+2X_i^{(1)}-X_i^{(2)}\). The marginal mean is therefore 1. Independence of the feature variables and the error gives the marginal variance:
The standard deviation is \(\sqrt{6}\approx2.449\) outcome units. If the two features were correlated, this calculation would need a covariance term; independence is doing substantive work here.
Worked Example: Checking a Variance Calculation
Use a hypothetical model \(Y_i=3+X_i^{(1)}+2X_i^{(2)}+\eta_i\), with mean-zero features, \(\operatorname{Var}(X_i^{(1)})=1\), \(\operatorname{Var}(X_i^{(2)})=4\), and \(\operatorname{Var}(\eta_i)=1\). Assume the two features and the error are mutually independent.
The mean outcome is \(E(Y_i)=3\), because all three random terms have mean 0. Under the stated independence assumption, the variance is
Thus the outcome standard deviation is \(\sqrt{18}\approx4.243\) outcome units. The coefficient 2 multiplies a feature with variance 4, so its contribution to outcome variance is \(2^2(4)=16\), not \(2(4)=8\). If the features were correlated, this answer would not follow without adding their covariance contribution.
Keep the Generator Fixed While Comparing Methods
Once the DGP is specified, a comparison should distinguish its components from the analysis pipeline. The DGP may use parameters and random draws to produce subjects. A method may then transform the observed blocks, fit an estimator, and return an output such as a prediction. These are different operations. If one method receives a different simulated dataset, has access to information unavailable to another, or is evaluated against a different target, the results no longer isolate the intended method comparison.
For a prediction benchmark, a clear design is to generate training and test subjects under the same stated DGP, fit each method using training data only, and score predictions on the held-out test outcomes. Any preprocessing that estimates quantities from data—such as centering or feature selection—must be learned from training data and then applied to test data. Otherwise, information from the test set has entered the fitted pipeline.
The benchmark should also state what is held constant across methods: the generated subjects, available blocks, outcome labels in training, evaluation subjects, and scoring rule. Method-specific tuning can differ when justified, but the tuning procedure and its information budget should be described. Repeated simulation replicates help reveal variability due to random draws; they do not repair an incomplete or mismatched DGP.
Worked Example: Comparing Predictions on the Same Simulated Subjects
Suppose three hypothetical test subjects have generated outcomes \(Y=(1,2,4)\). Two methods produce predictions \(A=(1.5,1,5)\) and \(B=(0,2,3)\). For this illustration, compare mean squared prediction error, defined for \(m\) test subjects as \(m^{-1}\sum_{i=1}^{m}(Y_i-\hat{Y}_i)^2\).
For method A, the squared errors are \((1-1.5)^2=0.25\), \((2-1)^2=1\), and \((4-5)^2=1\). Its mean squared error is \((0.25+1+1)/3=0.75\) outcome-units-squared.
For method B, the squared errors are \((1-0)^2=1\), \((2-2)^2=0\), and \((4-3)^2=1\). Its mean squared error is \((1+0+1)/3=0.6667\) outcome-units-squared. On these three test subjects, B has the smaller mean squared error by \(0.75-0.6667=0.0833\) outcome-units-squared. This small hypothetical comparison does not establish that B performs better in general; it illustrates the calculation. Both methods were evaluated against the same generated outcomes, which makes the contrast interpretable for this test set.
Diagnostics, Failure Modes, and Alternatives
Before running a large benchmark, simulate a small dataset and inspect whether it behaves as the specification intends. Check block dimensions, ranges, means, variances, and the matching of subject indices. Compare simulated summaries with quantities implied by the model. If the generator is stochastic, repeat this check across several draws: a single small sample may differ from its theoretical expectation by chance.
Common warning signs include unintended block independence, a feature scale inconsistent with the stated parameters, outcomes with implausible ranges, and subject rows that have become misaligned during data handling. For a benchmark involving training and test sets, check that all transformations are fitted without test-set information. A discrepancy between the mathematical specification and the implementation is a simulation error, not a property of an integration method.
There are alternatives to a fully specified parametric generator. One can resample whole matched subject rows from an existing dataset, preserving the observed joint configuration in those rows. This avoids choosing every feature distribution parametrically, but it inherits the source dataset’s limitations and does not generate new combinations beyond those represented. A parametric DGP can explore controlled assumptions and parameter settings, but its conclusions depend on those assumptions. Neither approach is universally preferable; the choice depends on the target question and the data available.
A further limitation is that “realistic” is not a single statistical property. A generator can match marginal means and variances while missing dependence, tail behavior, or important subject structure. Conversely, an elaborate DGP may be difficult to understand or validate. Report the assumptions that matter for the comparison, and treat performance as conditional on the simulated scenarios rather than as a universal ranking of methods.
Common Pitfalls
- Confusing the generator with the fitted method. A method is not the DGP merely because it is used to analyze the generated data. State each separately.
- Assuming independence without saying so. Separate marginal distributions do not define the joint distribution of blocks. Specify the dependence structure.
- Generating each block independently by accident. Independent random draws may be a deliberate assumption, but they should not arise from an unexamined implementation shortcut.
- Allowing test data to influence fitting. Estimating preprocessing or selection steps using test data makes a held-out evaluation optimistic.
- Overgeneralizing from one DGP. A method’s performance under one set of distributions and parameters is evidence about that setting, not proof of universal superiority.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a matched dataset with two blocks and one outcome, what does the shared subject index \(i\) identify?
- Does writing \(P(X_i^{(1)},X_i^{(2)}\mid C_i)\) imply that the two blocks are conditionally independent? Explain.
- Suppose \(Y=2+3X+\eta\), with \(E(X)=0\), \(\operatorname{Var}(X)=2\), and independent \(\eta\) with mean 0 and variance 1. Find \(E(Y)\) and \(\operatorname{Var}(Y)\).
- Why should centering parameters for a prediction pipeline be estimated using training data only?
Solutions. (1) It identifies one subject’s aligned measurements in both blocks and that subject’s outcome. (2) No. It denotes a joint conditional distribution; conditional independence would require the additional factorization \(P(X_i^{(1)}\mid C_i)P(X_i^{(2)}\mid C_i)\). (3) \(E(Y)=2+3(0)+0=2\). By independence, \(\operatorname{Var}(Y)=3^2(2)+1=19\). (4) If test observations help estimate the centering parameters, information from the evaluation set enters the fitted pipeline, so the reported test performance no longer represents prediction using training data alone.