Constrained Regression for Cell-Type Proportions

Why Constrain the Regression?

A spatial spot can contain more than one cell type, so its marker expression may be approximated by a mixture of reference profiles. If the reference profiles are on a compatible scale, the mixture weights have a direct interpretation: they are nonnegative proportions that add to one. Ordinary least squares does not enforce either property. It can therefore return negative weights or weights whose sum is not one, even when the goal is to estimate composition.

This tutorial considers a marker-based reference treated as fixed and an expression vector for one spot. It focuses on the optimization problem, its boundary behavior, and checks of the fitted residuals. The model is an approximation: a mathematically valid set of proportions does not by itself establish that the reference profiles describe the spot well.

Definition: Let $G$ be the number of selected marker measurements and $K$ the number of reference cell types. Write $y\in\mathbb{R}^G$ for the spot’s marker-expression vector and $A\in\mathbb{R}^{G\times K}$ for the reference matrix, whose $k$th column contains the expression profile for cell type $k$. The unknown proportion vector $p\in\mathbb{R}^K$ is constrained to the simplex $$ \Delta_K=\{p:p_k\geq 0\text{ for all }k,\ \mathbf{1}^{\mathsf T}p=1\}. $$ The constrained least-squares estimate minimizes $\|y-Ap\|_2^2$ over $p\in\Delta_K$.

The working mixture model is $y=Ap+\varepsilon$, where $\varepsilon$ represents deviations from the reference mixture, including measurement noise and possible reference mismatch. The scale and preprocessing of $y$ must be compatible with those of the columns of $A$; otherwise the linear mixture itself is not a meaningful comparison. This formulation assumes the spot’s expression is adequately represented by a mixture of the supplied profiles, without an unmodeled offset or other component. It does not assume the residuals are independent, Gaussian, or equally variable merely because least squares is used.

The simplex constraint is substantive. It says the analyzed spot is represented as a composition among the $K$ included types. If the data contain an omitted cell type, ambient contribution, or a baseline term, forcing all expression into the listed types may produce poor residuals or misleading proportions. Adding such terms changes the model and the interpretation of the proportions; it should not be done implicitly.

Optimization and Its Boundary Conditions

For computation, it is convenient to minimize half the residual sum of squares:

\widehat p=\underset{p\in\Delta_K}{\arg\min}\ \frac{1}{2}(y-Ap)^{\mathsf T}(y-Ap).

Because the simplex is closed and bounded and the objective is continuous, at least one minimum exists. The objective is convex, so every local minimum is a global minimum. The fitted vector $A\widehat p$ is a convex combination of the reference columns: it lies in their convex hull. The proportion vector need not be unique, however. If reference profiles are redundant or nearly indistinguishable, different proportions can give the same or almost the same fitted expression.

The optimality conditions explain why estimates can land at zero. Introduce a multiplier $\lambda$ for the sum constraint and nonnegative multipliers $\mu_k$ for the lower bounds. With $r=y-Ap$, the Karush–Kuhn–Tucker conditions are:

A^{\mathsf T}(Ap-y)+\lambda\mathbf{1}-\mu=0,\qquad \mathbf{1}^{\mathsf T}p=1,\qquad p_k\geq 0,\quad \mu_k\geq 0,\quad \mu_kp_k=0.

For every positive component $p_k$, complementary slackness gives $\mu_k=0$. For a component at zero, $\mu_k$ can be positive: the corresponding cell type would not improve the objective if its proportion were increased from that boundary while the other proportions were adjusted to keep their sum equal to one. Thus a zero estimate is a permissible boundary solution, not automatically a computational error or proof that the type is absent from the tissue.

Justification by active set: Suppose the positive components have index set $S$, and the columns in $A_S$ are linearly independent. With those components treated as free and their sum fixed at one, stationarity gives $A_S^{\mathsf T}A_Sp_S-A_S^{\mathsf T}y+\lambda\mathbf{1}=0$. Hence $$ p_S=(A_S^{\mathsf T}A_S)^{-1}(A_S^{\mathsf T}y-\lambda\mathbf{1}),\qquad \lambda=\frac{\mathbf{1}^{\mathsf T}(A_S^{\mathsf T}A_S)^{-1}A_S^{\mathsf T}y-1} {\mathbf{1}^{\mathsf T}(A_S^{\mathsf T}A_S)^{-1}\mathbf{1}}. $$ This solves the equality-constrained problem on that face of the simplex. It is a valid simplex solution only if its components are positive and the KKT inequalities for excluded components also hold. An optimizer can search across faces; simply solving the unconstrained regression and clipping negative estimates to zero does not generally satisfy these conditions.

When the constrained minimizer is in the interior, the residual is orthogonal to every allowable change in composition. In particular, for each pair of cell types $j,k$, $r^{\mathsf T}(A_j-A_k)=0$, where $A_j$ denotes column $j$. At a boundary, this equality need only hold for changes among active types; the corresponding one-sided inequalities apply to excluded types. These conditions are useful for understanding the solution, but a small residual norm and appropriate residual patterns are still needed to assess whether the reference explains the observed markers.

Worked Examples

Worked Example: An Interior Two-Type Estimate

Suppose three marker measurements have reference profiles $A_1=(8,2,5)^{\mathsf T}$ and $A_2=(2,8,5)^{\mathsf T}$. The hypothetical spot vector is $y=(5.6,4.2,5.4)^{\mathsf T}$. All entries are on the same arbitrary normalized-expression scale. Let $t$ be the proportion of type 1; the sum constraint makes the type 2 proportion $1-t$, with $0\leq t\leq1$.

The fitted marker values are $2+6t,\ 8-6t,\ 5$. Therefore the objective is

$$ Q(t)=(3.6-6t)^2+(-3.8+6t)^2+0.4^2. $$

Differentiating gives $Q'(t)=-12(3.6-6t)+12(-3.8+6t)=144t-88.8$. Setting this to zero gives $\widehat t=88.8/144=0.6167$, which is inside $[0,1]$. Thus $\widehat p=(0.6167,0.3833)$. The fitted values are $(5.7,4.3,5.0)$, so the residual vector, observed minus fitted, is $(-0.1,-0.1,0.4)$. The residual sum of squares is $0.1^2+0.1^2+0.4^2=0.1800$ in squared normalized-expression units. The third marker is identical in the two references and cannot distinguish their proportions; its residual contributes to fit but not to the estimate of $t$.

Worked Example: A Boundary Solution

Use three hypothetical two-marker reference profiles $A_1=(8,2)^{\mathsf T}$, $A_2=(2,8)^{\mathsf T}$, and $A_3=(5,5)^{\mathsf T}$, with observed spot vector $y=(9,1)^{\mathsf T}$. These profiles and observations share a scale. At the vertex $p=(1,0,0)$, the fitted vector is $(8,2)$, the residual is $(1,-1)$, and the sum of squared residuals is $1^2+(-1)^2=2$.

To check whether moving away from this vertex can improve fit, consider a small increase $\delta$ in type 2, taking the same amount from type 1. The fitted vector changes in direction $A_2-A_1=(-6,6)$. The directional derivative of the squared-error objective at the vertex is $-2(y-A_1)^{\mathsf T}(A_2-A_1)=-2(-12)=24$, which is positive. For a move toward type 3, the direction is $A_3-A_1=(-3,3)$, and the derivative is $-2(-6)=12$, also positive. Every feasible direction from this vertex is a nonnegative combination of those two directions, so its directional derivative is nonnegative. Since the objective is convex, this vertex is a global minimum. The constrained estimate is $(1,0,0)$, with residual sum of squares 2. It is a boundary solution; by itself, it does not establish that the spot contains only type 1.

Worked Example: Reading a Residual Pattern

Consider four hypothetical markers with two reference profiles $A_1=(10,2,6,4)^{\mathsf T}$ and $A_2=(2,10,6,4)^{\mathsf T}$, and spot vector $y=(7.2,4.8,6.5,4.2)^{\mathsf T}$. Again, all values are assumed to be comparable normalized expression. For type 1 proportion $t$, the fitted values are $(2+8t,10-8t,6,4)$. The objective is

$$ Q(t)=(5.2-8t)^2+(-5.2+8t)^2+0.5^2+0.2^2. $$

The first two terms are minimized at $t=5.2/8=0.65$, which satisfies the constraints. Thus $\widehat p=(0.65,0.35)$, and the fitted values are $(7.2,4.8,6,4)$. The residuals are $(0,0,0.5,0.2)$, giving residual sum of squares $0.5^2+0.2^2=0.2900$. The last two markers have identical expression in both references, so they provide no information about the mixture proportion. Their positive residuals nonetheless show that the fitted mixture misses those measurements. A shared offset, omitted contribution, or reference mismatch could cause such a pattern; residuals alone do not identify its cause.

Assumptions, Diagnostics, and Alternatives

The optimization requires a fixed reference matrix and a spot vector expressed on a compatible scale. Interpreting the solution as proportions additionally requires the reference columns to represent comparable cell-type profiles for the spot’s tissue and processing context, and the mixture approximation to be suitable for the selected markers. If these conditions fail, the optimizer still returns a best-fitting point on the simplex, but that point may not be a reliable biological composition.

Check the fit and the geometry, not just whether the output sums to one:

Inspect the residuals by marker. Large residuals, repeated signs within a marker group, or a trend with marker abundance can signal systematic lack of fit. Residuals should be interpreted on the measurement scale and with marker-specific variability in mind; a raw residual of a given size may matter differently for low- and high-expression markers.
Check the residual sum of squares and fitted values. A low objective is relative to the scale and number of markers, so compare it with the measurement context and the pattern of residuals. A valid simplex estimate can still fit poorly if the observed vector lies far from the convex hull of the reference profiles.
Check identifiability. If reference columns are identical, or nearly collinear in their contrasts, proportions may not be uniquely determined or may change substantially under small data perturbations. A solver returning one solution does not demonstrate that it is well identified.
Check boundary estimates and solver conditions. Exact or near-zero components can be genuine optima, but they can also arise from weakly distinguishable references or mismatch. Confirm feasibility, objective value, and the appropriate one-sided optimality conditions. Small numerical negative values near zero may be rounding artifacts; substantial negatives violate the constraint.

Constrained least squares is not a probabilistic model for the observed counts. It supplies a deterministic best fit under a squared-error criterion; the objective alone does not provide a probability statement or quantify uncertainty in the proportions. If errors have mean zero and covariance $\sigma^2 I$, then, for a fixed design matrix of full column rank, the unconstrained least-squares estimator has the Gauss–Markov best-linear-unbiased-estimator property. Equal variances without uncorrelated errors are not enough for that result. The simplex-constrained estimator is generally not an unconstrained linear estimator, so the Gauss–Markov statement is not an efficiency claim for it.

Approach	Constraint or criterion	Important consequence
Unconstrained least squares	Minimizes squared error without restricting $p$	Can produce negative weights or a sum different from one
Nonnegative least squares without a sum constraint	Requires $p_k\geq0$, but not $\mathbf{1}^{\mathsf T}p=1$	Estimates nonnegative weights, not necessarily proportions
Simplex-constrained least squares	Requires nonnegative weights summing to one	Produces a composition among included types, but can fit poorly under reference mismatch
Weighted least squares	Minimizes a weighted squared-error criterion, potentially with simplex constraints	Can reflect differing marker variances, but requires defensible weights and a compatible error model

The main limitation is that the constraints encode the intended interpretation; they do not repair a bad reference or an inadequate mixture model. Further, ordinary squared error gives larger-scale residuals greater influence. Weighting can change that influence, but arbitrary weights can distort the fit. With a fixed design and full rank, uncorrelated homoscedastic errors yield the familiar Gauss–Markov result for unconstrained least squares; that condition is not implied merely by using this constrained procedure.

Common Mistake

Common pitfall: “The estimates sum to one and are nonnegative, so the fit must be good and a zero component means that cell type is absent.” The correction is that feasibility follows from the optimization constraints, not from agreement between the reference and observed markers. Examine residuals, reference distinguishability, and boundary optimality. A boundary estimate may reflect the best fit within the supplied references, including when the reference set is incomplete or poorly matched.

Align the measurements.
Confirm that spot expression and reference profiles use compatible marker definitions and scales, and state the mixture model being fitted.

Solve on the simplex.
Minimize squared residual error subject to nonnegative proportions summing to one; do not replace this step by clipping unconstrained coefficients.

Evaluate the result.
Record fitted values, marker residuals, objective value, and whether the solution lies on a boundary; check identifiability and reference compatibility.

Check Your Understanding

Try each problem, then compare with the solutions below.

For two cell types, explain why the sum-to-one constraint allows the proportions to be represented by a single number $t$.
A three-type fit is $(0.4,0.6,0)$. Is the zero component automatically evidence that the third type is absent? What should be checked?
Suppose two reference columns are identical. What does that imply about identifying their separate proportions from these markers?
Why does clipping a negative unconstrained coefficient to zero, followed by rescaling the remaining coefficients to sum to one, not generally solve the constrained least-squares problem?

Solutions. (1) If the first proportion is $t$, the second must be $1-t$; nonnegativity requires $0\leq t\leq1$. (2) No. It is a boundary optimum under the supplied reference and data; examine residuals, reference distinguishability, and whether the one-sided optimality conditions hold. The fitted zero may reflect mismatch or weak information rather than true absence. (3) Only the sum of the two corresponding proportions affects the fitted marker vector, so their separate proportions cannot be identified from those markers alone. (4) Clipping and rescaling generally changes the fitted vector without minimizing squared error on the simplex; the resulting point need not satisfy the constrained problem’s stationarity and boundary conditions.