Building Semi-Synthetic Benchmark Data

Why Use Semi-Synthetic Benchmark Data?

A fully synthetic benchmark gives the investigator direct control over its generating model, but realistic multi-omics measurements can be difficult to reproduce. An observed cohort has authentic marginal distributions, feature dependencies, and technical irregularities, yet its true biological signals are usually not completely known. Semi-synthetic data combine observed measurements with deliberately controlled additions or alterations. The aim is to retain specified aspects of realism while creating a target whose injected signal is known.

This approach is useful only when “known” is used carefully. A researcher may know exactly which features were shifted, or which coefficients were used to generate an outcome. That does not make every association in the resulting data known: the source cohort can already contain associations, batch effects, or other structure. A benchmark should therefore distinguish the signal imposed by construction from the structure that came along with the observed data.

Specify What Must Be Preserved

Let \(X^{(m)}\) be the observed data matrix for modality \(m\), with \(n\) rows for samples and \(p_m\) columns for features. If the modalities are matched, row \(i\) in each matrix refers to the same sample. A semi-synthetic construction produces modified matrices \(X^{*(m)}\), and possibly a generated outcome or label. Before modifying anything, specify the properties the benchmark is intended to retain: for example, each feature’s empirical distribution, within-modality covariance, cross-omics covariance, matched sample structure, or technical batch patterns.

These properties are not interchangeable. A permutation of the rows of one modality leaves that modality’s feature values and empirical distributions unchanged. Applying the same permutation to every modality also retains the original pairing of samples, and thus retains cross-omics relationships. Permuting each modality separately retains its separate empirical distributions but generally changes those relationships. Adding a signal can change one or more of the preserved properties, so the construction must say which are intended to remain exact and which may change.

A useful general form for adding a known signal to one modality is \(X^*=X+y\beta^T\), where \(y\) is a specified \(n\)-vector and \(\beta\) is a \(p\)-vector of feature effects. Row \(i\) receives the shift \(y_i\beta^T\). Entries of \(\beta\) equal to zero mark features not directly modified. The construction itself is controlled, but its total observed pattern includes both the original \(X\) and the added term.

What an Additive Signal Does to Covariance

An additive signal is simple to implement, but it need not leave the covariance structure unchanged. To see why, let \(X_c\) denote the column-centered version of \(X\), and let \(y_c\) be \(y\) centered by its sample mean. Centering \(X^*=X+y\beta^T\) gives \(X_c^*=X_c+y_c\beta^T\). The sample covariance matrix of the modified features is therefore

The equality follows by multiplying out the matrix product and grouping the four terms; the first is \(X_c^TX_c/(n-1)=S_X\). The two middle terms depend on the association between the original features and the injection pattern \(y\), and the last term is the covariance contribution from the injected signal itself. Thus, even if the original features have a particular covariance, a group-dependent shift can alter it. A constant shift applied to every row is different: centering removes it, so it does not change sample covariance.

This calculation is a practical warning, not a reason to avoid injection. It clarifies what the operation guarantees. If preserving covariance exactly is essential, a signal construction that changes covariance may not meet the benchmark’s objective. If injecting a biologically interpretable association is the objective, changed covariance may be an intended consequence and should be reported as such.

Generate an Outcome Without Altering the Omics Matrices

A different construction keeps observed omics values fixed and generates an outcome from them. For a continuous outcome, one possible specification is \(z=\gamma_0+X^{(1)}\gamma_1+X^{(2)}\gamma_2+\varepsilon\), where the coefficient vectors and intercept are specified by the benchmark and \(\varepsilon\) is a generated noise vector. In this design the omics distributions and their observed cross-omics covariance are unchanged, because the matrices themselves are not modified. The known outcome-generating coefficients define the imposed relationship.

That guarantee is conditional on using the matrices as given: the generated outcome can still be associated with features that have zero coefficients because those features correlate with features that do have nonzero coefficients. A coefficient of zero means “no direct term in the specified outcome equation,” not “no marginal association in the constructed data.” If adding noise, state its distribution and how it is generated; the outcome then contains both the specified signal and the noise realization.

Resampling and Matched Multi-Omics Structure

Resampling can create a benchmark cohort while retaining observed measurement patterns. If rows are sampled with replacement, use the same sampled row indices across all modalities when sample matching is part of the intended structure. The resulting cohort may contain duplicate source samples, so it does not preserve every original feature distribution exactly; rather, it samples from the source cohort’s empirical distribution. Independent row resampling across modalities breaks the original sample-level pairing.

Assumptions, Diagnostics, and Alternatives

Semi-synthetic data do not require the observed cohort to be a random sample from a named population for the arithmetic of a transformation to be valid. They do require that the source data and the construction be suitable for the benchmark’s stated target. Samples must be correctly matched before joint operations; feature scales and preprocessing must be understood before interpreting an additive shift; and any generated labels, outcomes, or noise must follow the written construction. If one source sample contributes to both development and evaluation data, performance can also reflect familiarity with that source sample rather than generalization to new samples.

Useful diagnostics compare the source and modified data on the properties the construction claims to preserve. Check feature-wise empirical distributions, within-modality covariance, cross-omics covariance, sample pairing, and batch summaries as relevant. Separately verify that the injected support, group shifts, or outcome coefficients match the intended specification. A discrepancy is not automatically an error: it may be the expected result of injection. The key is whether it agrees with the stated preservation target.

A fully synthetic benchmark offers stronger control over the whole generating mechanism, but can fail to represent important empirical features. Direct resampling retains observed values but supplies little new ground truth unless labels or signals are added. Semi-synthetic construction occupies the middle ground: it can provide controlled targets on an observed-data backbone, but inherits unmodeled structure and depends on the source cohort. No construction simultaneously guarantees complete realism, complete knowledge of truth, and preservation of every observed feature.

A common mistake is to say, “The selected feature is associated with the generated label, so the association is entirely the known signal.” The correction is to distinguish the injected contribution from the pre-existing association, as in the group-contrast example. Another mistake is to claim that separate modality distributions are preserved and infer that their joint structure is preserved too. The permutation example shows why those are distinct claims.