Factorized Losses and Marginal Calibration

Why Factorization Does Not Mean Independence

A structured label can contain several dependent components: a sequence of tags, a set of attributes, or a vector of binary outcomes. A factorized surrogate assigns a separate loss to each component and adds those losses. This can make its conditional risk straightforward to analyze, but it does not make the components statistically independent. The central question is what information the factorized objective recovers, and whether that information is enough for the task’s decision rule.

We focus on a binary vector \(Y=(Y_1,\ldots,Y_d)\), where \(Y_i\in\{0,1\}\), and condition on a feature value \(X=x\). Write \(\eta_i(x)=P(Y_i=1\mid X=x)\) for the \(i\)th conditional marginal. Dependence among coordinates is encoded by the joint conditional distribution of \(Y\); the collection of marginals \((\eta_1(x),\ldots,\eta_d(x))\) generally does not specify that joint distribution.

Logistic Loss Recovers the Marginals

For predicted probabilities \(r=(r_1,\ldots,r_d)\in[0,1]^d\), the factorized logistic loss is

Use the usual extended-value conventions: a term with zero multiplier is zero, while assigning probability zero to an event with positive probability gives infinite loss. The conditional risk is the expectation of this loss given \(X=x\).

Proof. By linearity of expectation, the conditional risk is the sum of the coordinatewise conditional risks. For coordinate \(i\), only the marginal probability \(\eta_i=\eta_i(x)\) enters:

For \(0<\eta_i<1\), differentiating with respect to \(r_i\in(0,1)\) gives

The denominator is positive, so the derivative is negative for \(r_i<\eta_i\) and positive for \(r_i>\eta_i\). Thus \(r_i=\eta_i\) is the unique minimizer. Substitution gives the minimum \(-\eta_i\log\eta_i-(1-\eta_i)\log(1-\eta_i)\). If \(\eta_i=0\), the risk is minimized at \(r_i=0\); if \(\eta_i=1\), it is minimized at \(r_i=1\), with the same entropy formula interpreted by continuity. Since the total risk is a sum and each term depends only on its own \(r_i\), its minimizers are precisely the coordinatewise minimizers just found. No step uses the joint probabilities of \(Y\), so dependence does not alter the result. \(\square\)

When Marginals Are Sufficient for the Task

Marginal calibration is especially useful when the task loss is additive across coordinates. The action need not range over every binary vector: it may be restricted to a feasible set \(A\subseteq\{0,1\}^d\). What matters is that the loss of an action \(a\in A\) on outcome \(y\) can be written as a sum of coordinate losses \(c_i(a_i,y_i)\).

Proof. Finite additivity of expectation gives

The expression contains no joint probabilities such as \(P(Y_i=1,Y_j=1\mid X=x)\). Therefore the marginals determine the risk of every action in \(A\). Since \(A\) is finite and nonempty, at least one action attains the minimum of these risks; comparing them yields a Bayes action. This proves the claim, including when \(A\) imposes constraints on the output. \(\square\)

For Hamming loss, \(L(a,y)=\sum_i \mathbf{1}\{a_i\ne y_i\}\), each coordinate’s conditional error is \(\eta_i\) when predicting zero and \(1-\eta_i\) when predicting one. If every binary vector is allowed, minimizing each term separately gives \(a_i=1\) when \(\eta_i>1/2\), \(a_i=0\) when \(\eta_i<1/2\), and either value at a tie. If actions are constrained, the theorem still says marginals suffice for additive Hamming risk, but the decision must minimize total risk over the feasible set. Independent thresholding is not automatically a feasible or optimal procedure.

Constraints and Coordinatewise Guarantees

The distinction between recovering marginals and making a feasible structured prediction matters whenever the output space has dependencies built into its admissible configurations. A decoder may use calibrated marginal scores to search the feasible set under an additive task loss. But simply applying the unconstrained coordinatewise decision rule can produce an invalid output. The following example shows this even when the conditional labels themselves always satisfy a structural constraint.

Coordinatewise guarantees also have a precise scope. If estimated scores approach the true marginals and the target is unconstrained Hamming prediction, then each coordinate whose marginal is strictly on one side of \(1/2\) eventually receives the correct threshold decision. At a marginal of exactly \(1/2\), either decision has the same coordinatewise risk. This is a guarantee about marginal probabilities and additive coordinate risk; it is not a guarantee that a model has learned the full conditional joint distribution.

The same caution applies to the word “factorized.” A product distribution constructed from scores \(r_i\) assigns probability \(\prod_i r_i^{y_i}(1-r_i)^{1-y_i}\) to a vector \(y\). That is a particular joint distribution with independent coordinates. Marginal calibration establishes that its factors match the true marginals; it does not establish that this product is the true joint distribution. If the task loss depends on co-occurrence patterns or on the probability of a whole configuration, joint information may be necessary.

Common Mistakes

• Assuming factorization asserts independence. The factorized logistic risk is a sum of expectations, so each term uses one marginal. This algebra does not assert that the labels are independent.
• Treating calibrated marginals as a calibrated joint distribution. Distinct joint distributions can share all coordinate marginals, as in the worked example. A product of marginals is an approximation unless independence is justified.
• Using a Hamming-optimal decoder for every task. Thresholding marginals minimizes unconstrained Hamming risk, not necessarily exact-match risk or another set-level loss.
• Thresholding before checking feasibility. With a constrained output set, coordinatewise decisions can be inadmissible. For additive loss, compare the risks of feasible actions instead.