Normalizing Kernel Matrices Before Fusion

Why Normalize Before Kernel Fusion?

A kernel matrix records pairwise similarities among the same \(n\) patients. When kernels from different omics views are added, their numerical scales matter: an equally weighted sum does not necessarily give the views equal influence. If one matrix has entries far larger than another, it can dominate the sum even when the assigned weights are identical. Normalizing first is one way to make the scales more comparable, but the choice of normalization also determines which features of the matrices are preserved.

The earlier material on centering and normalizing kernel matrices introduces normalization as a general idea. Here the focus is the distinction among three choices used before fusion: trace normalization, Frobenius-norm normalization, and diagonal normalization. The first two rescale an entire matrix by one number. The third rescales rows and columns separately, so it can change pairwise similarities relative to one another.

Definition: Let \(K\) be a symmetric \(n\times n\) kernel matrix. Its trace is \(\operatorname{tr}(K)=\sum_{i=1}^n K_{ii}\), and its Frobenius norm is \(\|K\|_F=\sqrt{\sum_{i=1}^n\sum_{j=1}^n K_{ij}^2}\). Trace normalization is \(K/\operatorname{tr}(K)\), when the trace is positive. Frobenius-norm normalization is \(K/\|K\|_F\), when the norm is positive. For diagonal normalization, define \(D=\operatorname{diag}(K_{11},\ldots,K_{nn})\) and set \(K_{\mathrm{diag}}=D^{-1/2}KD^{-1/2}\), provided every \(K_{ii}>0\).

For a nonzero positive semidefinite (PSD) kernel, the trace and Frobenius norm are positive, and the diagonal entries are nonnegative. Diagonal normalization therefore requires the additional condition that no diagonal entry is zero. A zero diagonal means that this formula cannot define a unit diagonal for that patient; it must be investigated rather than silently divided by zero. Throughout, fusion means a weighted sum \(\sum_{m=1}^M w_mK_m\), with nonnegative weights \(w_m\). If each \(K_m\) is PSD, this sum is PSD as well.

What Each Normalization Controls

Trace normalization sets the sum of diagonal entries to one. For PSD kernels those entries are nonnegative, so the trace is a measure of the total self-similarity on the diagonal. It does not set the off-diagonal entries to a common size or fix the Frobenius norm. Frobenius normalization sets the square root of the sum of squared entries to one; it controls the overall matrix magnitude, counting off-diagonal entries as well as diagonal entries.

Both are global rescalings. If \(c>0\), then the normalized versions of \(cK\) equal those of \(K\), and every entry of \(K\) is multiplied by the same positive constant during normalization. Thus a global rescaling preserves the relative pattern of entries and, in particular, the ratio \(K_{ij}/K_{ii}\) wherever that ratio is defined.

Diagonal normalization instead converts each entry to \(K_{ij}/\sqrt{K_{ii}K_{jj}}\). Its diagonal entries become one, and its off-diagonal entries express similarity relative to the two patients’ self-similarities. When \(K\) is a Gram matrix of feature vectors, this is the cosine similarity between the corresponding vectors. Unlike global rescaling, diagonal normalization can change the relative magnitudes of different off-diagonal entries.

Worked Example: A Large Scale Difference in Two Kernels

Consider two hypothetical PSD kernels for two patients:

K_A=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}, \qquad K_B=\begin{pmatrix}100&10\\10&100\end{pmatrix}.

Their eigenvalues are \(1.5,0.5\) for \(K_A\) and \(110,90\) for \(K_B\), so both are PSD. The traces are \(2\) and \(200\). Trace normalization gives

\frac{K_A}{\operatorname{tr}(K_A)} =\begin{pmatrix}0.5&0.25\\0.25&0.5\end{pmatrix}, \qquad \frac{K_B}{\operatorname{tr}(K_B)} =\begin{pmatrix}0.5&0.05\\0.05&0.5\end{pmatrix}.

For example, the normalized diagonal entries are \(1/2=0.5\) for \(K_A\) and \(100/200=0.5\) for \(K_B\). The normalized off-diagonal entries are \(0.5/2=0.25\) and \(10/200=0.05\). Trace normalization removes the hundredfold difference in diagonal scale, while retaining the difference in relative off-diagonal similarity.

With equal raw weights, \(K_A+K_B\) has diagonal entries \(101\) and off-diagonal entries \(10.5\). The contribution from \(K_B\) is much larger in absolute size: its diagonal entry is 100 times that of \(K_A\), and its off-diagonal entry is 20 times as large. Consequently, the sum’s scale is largely set by \(K_B\), notwithstanding equal weights. By contrast, the equal-weight sum of the trace-normalized matrices has diagonal entries \(1.0\) and off-diagonal entries \(0.30\). The views now have the same trace, though their similarity patterns remain different.

Frobenius and Diagonal Normalization in Practice

Worked Example: Calculating Frobenius and Diagonal Normalizations

Use the two kernels in the previous calculation. For \(K_A\), \(\|K_A\|_F=\sqrt{1^2+0.5^2+0.5^2+1^2}=\sqrt{2.5}\approx1.581\). For \(K_B\), \(\|K_B\|_F=\sqrt{100^2+10^2+10^2+100^2}=\sqrt{20200}\approx142.1\). Therefore,

\frac{K_A}{\|K_A\|_F} \approx\begin{pmatrix}0.6325&0.3162\\0.3162&0.6325\end{pmatrix}, \qquad \frac{K_B}{\|K_B\|_F} \approx\begin{pmatrix}0.7036&0.07036\\0.07036&0.7036\end{pmatrix}.

For instance, \(100/\sqrt{20200}\approx0.7036\) and \(10/\sqrt{20200}\approx0.07036\). Each resulting matrix has Frobenius norm one, but the normalized diagonal entries are not identical: the Frobenius norm accounts for both diagonal and off-diagonal values. This differs from trace normalization, which made both diagonal entries exactly \(0.5\).

For diagonal normalization, the diagonal entries of \(K_A\) are both 1, so \(K_A\) is unchanged. For \(K_B\), each diagonal entry is 100 and each off-diagonal entry is 10. Thus the normalized off-diagonal entry is \(10/\sqrt{100\cdot100}=0.1\), giving

(K_A)_{\mathrm{diag}}= \begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}, \qquad (K_B)_{\mathrm{diag}}= \begin{pmatrix}1&0.1\\0.1&1\end{pmatrix}.

These have matching diagonals and display the difference in relative pairwise similarity directly. They do not, however, have unit trace or unit Frobenius norm.

Worked Example: Diagonal Normalization Can Change Relative Similarities

Consider a hypothetical three-patient Gram matrix constructed from feature vectors \((2,0)\), \((1,1)\), and \((0,3)\):

K=\begin{pmatrix}4&2&0\\2&2&0\\0&0&9\end{pmatrix}.

It is PSD because it is a Gram matrix, and its diagonal entries \(4,2,9\) are all positive. Its trace is \(15\), so trace normalization divides every entry by 15; for example, the first off-diagonal entry becomes \(2/15\approx0.1333\). Frobenius normalization instead uses \(\|K\|_F=\sqrt{4^2+2^2+2^2+2^2+9^2}=\sqrt{109}\approx10.44\); the same entry becomes \(2/\sqrt{109}\approx0.1916\).

Diagonal normalization divides the first off-diagonal entry by \(\sqrt{4\cdot2}\), giving \(2/\sqrt{8}\approx0.7071\). The full result is

K_{\mathrm{diag}}= \begin{pmatrix} 1&1/\sqrt{2}&0\\ 1/\sqrt{2}&1&0\\ 0&0&1 \end{pmatrix}.

Here the first two patients’ original similarity of 2 is interpreted relative to their self-similarities, rather than simply shrunk by one common factor. In general, diagonal normalization can alter the relative off-diagonal entries because each pair is divided by a different product of diagonal terms.

Why the PSD Guarantee Is Retained

The normalization choices are also relevant to the mathematical validity of the fused kernel. Trace and Frobenius normalization divide by a positive scalar, so they preserve positive semidefiniteness. Diagonal normalization also preserves it when all diagonal entries are positive. To see why, let \(z\) be any real vector and \(A=D^{-1/2}\). Then

z^\mathsf{T}K_{\mathrm{diag}}z =z^\mathsf{T}AKA z =(Az)^\mathsf{T}K(Az)\geq0,

because \(K\) is PSD and \(Az\) is a real vector. Therefore \(AKA\) is PSD. If normalized kernels are combined with nonnegative weights, their weighted sum is PSD: for every \(z\),

z^\mathsf{T}\left(\sum_{m=1}^M w_mK_m\right)z =\sum_{m=1}^M w_m z^\mathsf{T}K_mz\geq0.

This argument depends on nonnegative weights. Arbitrary negative weights do not generally preserve positive semidefiniteness.

Choosing a Normalization

Choice	What is fixed	What is preserved or changed	Useful caution
Trace	Trace equals 1	One global scale factor preserves relative entry pattern	Does not equalize off-diagonal scale or Frobenius norm
Frobenius norm	Frobenius norm equals 1	One global scale factor preserves relative entry pattern	Large entries contribute quadratically to the norm
Diagonal	Every diagonal entry equals 1	Pairwise entries are rescaled according to both patients’ diagonals	Requires strictly positive diagonal entries and can alter relative similarities

Trace normalization is a direct choice when matching total self-similarity is meaningful. Frobenius normalization is natural when the target is equal total squared matrix magnitude. Diagonal normalization is suitable when comparisons relative to each patient’s self-similarity are desired, as with cosine-like similarity. These are different targets, not interchangeable formulas for the same operation.

A practical check is to report each view’s trace, Frobenius norm, and diagonal range before and after normalization. Also inspect representative off-diagonal values. If the raw magnitudes differ greatly, an equal-weight raw sum may largely reflect the scale of one view. After normalization, confirm that the chosen criterion was achieved and that no invalid diagonal or unexpected numerical values remain. Normalization does not establish that views carry comparable biological information, and it cannot correct a poorly constructed kernel.

Common Mistake and Limitations

Mistake: “The weights are equal, so the views contribute equally.” Equal coefficients only give equal multipliers. If the matrices differ greatly in scale, their numerical contributions to the sum can still be very different. Compare matrix scales or normalize according to a stated objective before interpreting an equal-weight fusion.
Mistake: Treating diagonal normalization as just another global rescaling. Its divisors vary by patient pair, so it can change relative off-diagonal similarities. Use it only when that relative-to-diagonal interpretation is intended.
Mistake: Assuming unit trace, unit Frobenius norm, and unit diagonal all describe the same normalization. Each condition controls a different quantity; verify the defining quantity directly.
Limitation: Normalization can reduce scale imbalance, but it does not select fusion weights or guarantee that a view is scientifically relevant. It may also remove magnitude information that was meaningful in the original kernel.

The choice should therefore be documented as part of the analysis: state the formula, verify its conditions, and explain which aspect of the kernels is being made comparable. The normalized matrices and the resulting fusion answer a modeling question shaped by that choice.

Check Your Understanding

For each question, give the calculation or reasoning that supports your answer.

A nonzero PSD kernel has trace \(12\). What is its trace after trace normalization? Solution: It is \(12/12=1\), by definition.
A PSD kernel has diagonal entries \(4\) and \(9\), and its corresponding off-diagonal entry is \(3\). What is that entry after diagonal normalization? Solution: \(3/\sqrt{4\cdot9}=3/6=0.5\).
Does Frobenius normalization necessarily make every diagonal entry equal? Solution: No. It divides all entries by the same norm; it sets the sum of squared entries to one, not the individual diagonal values.
Why does a nonnegative weighted sum of PSD kernels remain PSD? Solution: For every vector \(z\), its quadratic form is the sum \(\sum_m w_m z^\mathsf{T}K_mz\). Each term is nonnegative because \(w_m\geq0\) and \(K_m\) is PSD, so the sum is nonnegative.
Two views receive equal raw weights, but one kernel’s entries are about 100 times larger. Does equal weighting ensure equal numerical contribution? Solution: No. The coefficient is equal, but the larger entries can make that kernel’s contribution much larger. A scale assessment and an explicitly chosen normalization may be needed.