Deriving Kernel Alignment Weight Estimates

Why derive the weights?

A weighted sum of kernels is a way to represent patient similarity using several omics views. The weights determine how strongly each view enters that sum. Rather than choosing them only by inspection, one can estimate weights by asking how closely a weighted combination of centered view kernels matches a specified centered target kernel. This turns the choice into a least-squares problem with explicit constraints.

The target $T$ might be a prespecified similarity pattern or a reference kernel defined for the analysis. In an unsupervised application it need not be a clinical outcome. The resulting weights quantify how the views combine to approximate that target; they are not causal effects, nor automatically percentages of biological contribution. The earlier material on kernel alignment and unsupervised kernel weighting motivates comparing centered kernels; here the focus is the derivation of the constrained least-squares estimate.

Definition: For $n$ matched subjects, let $K_1,\ldots,K_M$ be centered $n\times n$ kernel matrices, and let $T$ be a centered target matrix of the same size. The simplex-constrained least-squares weights minimize $$ L(w)=\left\|T-\sum_{m=1}^{M}w_mK_m\right\|_F^2 $$ subject to $w_m\geq 0$ for every $m$ and $\sum_{m=1}^{M}w_m=1$. Here $w=(w_1,\ldots,w_M)^\top$, and $\|A\|_F^2=\sum_{i,j}A_{ij}^2$ is the squared Frobenius norm.

Centering means applying $H=I_n-\frac{1}{n}\mathbf{1}\mathbf{1}^\top$ on both sides of each original kernel: $K_m=HK_m^{\mathrm{raw}}H$, and similarly for $T$ when it is obtained from an uncentered target. Thus the row and column sums are zero. The matrices must be indexed by the same subjects in the same order; otherwise their entrywise comparisons do not represent matching subject pairs. The least-squares derivation itself does not require the matrices to be positive semidefinite, but kernel applications commonly use centered positive semidefinite kernels. The constraints ensure a convex combination: weights are nonnegative and total one.

Expanding the objective

Use the Frobenius inner product $\langle A,B\rangle_F=\sum_{i,j}A_{ij}B_{ij}$. Define $Q_{ij}=\langle K_i,K_j\rangle_F$ and $c_i=\langle K_i,T\rangle_F$. Expanding the squared norm gives

L(w)=\langle T,T\rangle_F-2\sum_{i=1}^{M}w_i\langle K_i,T\rangle_F +\sum_{i=1}^{M}\sum_{j=1}^{M}w_iw_j\langle K_i,K_j\rangle_F =\|T\|_F^2-2c^\top w+w^\top Qw.

The matrix $Q$ is positive semidefinite: for any vector $a$, $a^\top Qa=\|\sum_i a_iK_i\|_F^2\geq0$. Consequently, $L(w)$ is a convex quadratic function. Minimizing it over the simplex has a global solution. If $Q$ is positive definite, the objective is strictly convex and the minimizing weight vector is unique. If $Q$ is singular, distinct weight vectors can produce the same combined kernel, so the weights may not be identifiable even when the fitted combination is.

Derivation: For two views, set $w_1=w$ and $w_2=1-w$, so $0\leq w\leq1$. Write $D=K_1-K_2$. The combined kernel is $K_2+wD$, and $$ L(w)=\|T-K_2-wD\|_F^2 =\|T-K_2\|_F^2-2w\langle D,T-K_2\rangle_F+w^2\|D\|_F^2. $$ If $D\ne0$, differentiating gives $L'(w)=-2\langle D,T-K_2\rangle_F+2w\|D\|_F^2$. The unconstrained minimizer is therefore $w^*=\langle D,T-K_2\rangle_F/\|D\|_F^2$. Because the constrained domain is the interval $[0,1]$, the estimate is $\hat w_1=\min(1,\max(0,w^*))$, with $\hat w_2=1-\hat w_1$. If $D=0$, both views are the same matrix and every pair of weights summing to one gives the same fit.

For more than two views, the simplex constraints generally make a single closed-form expression inconvenient. The Karush–Kuhn–Tucker conditions describe the solution. With multipliers $\lambda$ for the sum constraint and $\mu_m\geq0$ for the nonnegativity constraints, stationarity is $2Qw-2c+\lambda\mathbf{1}-\mu=0$. The solution must also satisfy $\mathbf{1}^\top w=1$, $w_m\geq0$, and $\mu_mw_m=0$ for every $m$. A zero weight may have a positive multiplier; a positive weight must have multiplier zero. In practice, a quadratic-programming solver can impose these conditions while minimizing the convex objective.

Worked estimates

Worked Example: Two Centered Kernels Fit a Target Exactly

Consider three hypothetical subjects and the centered feature vectors $u=(1,-1,0)^\top/\sqrt{2}$ and $v=(1,1,-2)^\top/\sqrt{6}$. They are orthogonal and each sums to zero. Define the positive semidefinite centered kernels $K_1=uu^\top$ and $K_2=vv^\top$. Their squared Frobenius norms are both one, and $\langle K_1,K_2\rangle_F=(u^\top v)^2=0$. Let the target be $T=0.75K_1+0.25K_2$. All matrices refer to the same three subjects and are centered.

For a candidate weight $w$, the fused matrix is $wK_1+(1-w)K_2$. The residual is $(0.75-w)K_1+(w-0.75)K_2$. Orthogonality under the Frobenius inner product gives $$ L(w)=(0.75-w)^2+(w-0.75)^2=2(w-0.75)^2 =2w^2-3w+1.125. $$

Differentiating yields $L'(w)=4w-3$. Setting this equal to zero gives $\hat w_1=0.75$; the simplex constraint then gives $\hat w_2=1-0.75=0.25$. Both estimates satisfy the nonnegativity constraints. Substitution gives $L(0.75)=2(0)^2=0$, so these weights reproduce the target exactly. The constant term matters: the expanded squared-error objective is nonnegative, and the displayed expression equals zero at its minimizer.

Worked Example: A Boundary Estimate When One View Overshoots

Use the same hypothetical centered, orthogonal kernels $K_1$ and $K_2$, each with squared Frobenius norm one, but now set the target to $T=2K_1$. This target is centered and positive semidefinite. For $0\leq w\leq1$, orthogonality gives $$ L(w)=\|2K_1-wK_1-(1-w)K_2\|_F^2=(2-w)^2+(1-w)^2 =2w^2-6w+5. $$

The unconstrained derivative is $4w-6$, which is zero at $w=1.5$. That value is outside the allowable interval. The constrained minimum is at the nearest endpoint, $w_1=1$, $w_2=0$. Its residual sum of squares is $L(1)=2-6+5=1$. The target asks for twice $K_1$, while a convex combination cannot scale $K_1$ above one in this example. Thus the boundary estimate is the best permitted approximation, not an exact reconstruction.

Worked Example: Three Orthogonal Views

For four hypothetical subjects, take three mutually orthogonal, centered unit vectors $u_1,u_2,u_3$, and define $K_m=u_mu_m^\top$. These rank-one kernels are centered and positive semidefinite, with $\langle K_i,K_j\rangle_F=1$ when $i=j$ and $0$ otherwise. Suppose $T=0.6K_1+0.3K_2+0.1K_3$. Then $Q=I_3$, $c=(0.6,0.3,0.1)^\top$, and $\|T\|_F^2=0.36+0.09+0.01=0.46$.

The objective is $L(w)=0.46-1.2w_1-0.6w_2-0.2w_3+w_1^2+w_2^2+w_3^2$. The candidate weights $(0.6,0.3,0.1)$ are nonnegative and sum to one. Substituting gives $0.46-0.72-0.18-0.02+0.36+0.09+0.01=0$. Since a squared norm cannot be negative, this feasible zero-error solution is a global minimum. In this constructed example the three estimated weights exactly recover the target mixture.

Diagnostics, alternatives, and limitations

A fitted weight vector alone does not show whether the target was well approximated. Report or inspect the residual norm $\|T-\sum_m\hat w_mK_m\|_F$, preferably alongside $\|T\|_F$ or the objective value on a clearly stated scale. Check that the weights are nonnegative and sum to one, and confirm that the kernel matrices use the same subject ordering and centering convention. A solver returning a small numerical violation of a constraint may reflect numerical tolerance; a substantial violation suggests an implementation or constraint-specification problem.

Several warning signs concern interpretation as well as computation:

Nearly identical views: if $K_i$ and $K_j$ are nearly equal, reallocating weight between them changes the fitted matrix very little. Individual estimates can be unstable even when the fused kernel is stable.
Weights at zero or one: a boundary solution can be correct, as in the second example, but it is worth checking whether the target is poorly matched or a view is on a much larger scale.
Large residual: the target may not be representable by a convex combination of the supplied views. A valid optimizer cannot remove that structural mismatch.
Scale sensitivity: the least-squares objective is based on matrix magnitudes. Rescaling one kernel can change the estimated weights even if its qualitative similarity pattern remains similar. Examine scales and document preprocessing; do not interpret weights as comparable contributions without considering this dependence.

Equal weights are simple and transparent, but they do not use the target to adapt the combination. Unconstrained least squares removes the nonnegativity and sum-to-one restrictions; it can fit more flexibly, but can assign negative weights or produce coefficients that are not a convex mixture. Nonnegative least squares permits nonnegative weights without requiring their sum to be one, so its total scale can vary. The simplex-constrained rule specifically answers the question of which convex combination best approximates $T$. These are different estimands, not merely different numerical algorithms.

Key takeaway: Centered-kernel least squares becomes a convex quadratic program with $Q_{ij}=\langle K_i,K_j\rangle_F$ and $c_i=\langle K_i,T\rangle_F$. The simplex constraints make the estimated combination interpretable as a convex mixture, while the fit, identifiability, and scale must still be assessed separately.

Check Your Understanding

Try each question, then compare with the brief solutions below.

For two views, suppose $\|K_1-K_2\|_F^2=4$ and $\langle K_1-K_2,T-K_2\rangle_F=1$. Find both constrained weights.
What does $Q_{ij}$ measure, and why is $Q$ positive semidefinite?
If two view kernels are identical, can their separate weights be identified from this objective?
A fitted simplex weight vector is valid, but its residual norm is large. Does that alone imply the optimizer failed?
Why can rescaling a kernel change the estimated simplex weights?

Solutions. 1. The unconstrained first weight is $1/4=0.25$, already in $[0,1]$; therefore $\hat w_1=0.25$ and $\hat w_2=0.75$. 2. $Q_{ij}$ is the Frobenius inner product, measuring matrix alignment in the least-squares expansion. For any $a$, $a^\top Qa=\|\sum_i a_iK_i\|_F^2\geq0$, so $Q$ is positive semidefinite. 3. No. Moving weight between identical kernels leaves the combined matrix unchanged, so the separate weights are not identifiable. 4. No. The optimizer may have found the best feasible combination even though the target is poorly represented by the available views. 5. Rescaling changes the magnitudes and inner products in the least-squares objective, so the best convex coefficients can change even if the kernel's qualitative pattern is similar.

Deriving Kernel Alignment Weight Estimates

What You'll Learn