Why Estimate Weights From Predictive Scores?
Averaging predictive distributions instead of parameters preserves the distinct predictive distributions produced by different NLME models. The next question is how much weight to assign each distribution. Stacking answers this by choosing nonnegative weights that make the observed outcomes score well under the weighted predictive mixture. The weights are learned from predictive performance on data not used to fit the candidate models.
The target matters. A score for a single concentration measurement answers a different predictive question from a joint score for all measurements in a new subject. In either case, the weight-fitting records must represent the target and be scored using predictive distributions generated without using the scored outcomes to fit the candidate models.
The \(p_{im}\) values are densities when the outcome is continuous and probabilities when it is discrete. A continuous density need not be less than one. Comparisons require the candidate models to score the same observed outcome, with the same units and measure, for each scoring unit. Adding constants common to every candidate's log score does not change the maximizing weights.
Weights apply to predictive distributions, not to parameter estimates. A weight of 0.7 on one model means that its predictive distribution contributes 70% to each mixture distribution. It does not mean that 70% of its parameter estimates should be combined with another model's estimates.
Optimization and Its Justification
The candidate predictions are treated as fixed while optimizing the weights. For every scoring unit, the mixture density must be positive at the observed outcome for the objective to be finite. The feasible weights lie on the simplex: they are nonnegative and sum to one. A model may receive weight zero; positivity of every weight is not required.
The objective is concave in the weights. To see this, write \(q_i(w)=\sum_m w_m p_{im}\). For any direction \(v=(v_1,\ldots,v_M)\) along which the mixture remains positive,
Thus any local maximum on the simplex is also a global maximum. Strict concavity, and therefore a unique optimum, is not guaranteed: candidate predictions can be redundant or the available outcomes may not distinguish their contributions. Numerical optimization should enforce the simplex constraints rather than optimize unconstrained weights and truncate the result afterward.
A useful first-order condition also helps interpret the solution. Define \(g_m=\partial\ell/\partial w_m=\sum_i p_{im}/q_i(w)\). At an optimum, every model with positive weight has \(g_m=n\); every zero-weight model has \(g_m\leq n\). To justify the condition, transferring a small amount of weight from an active model \(a\) to model \(b\) changes the objective to first order by \(\epsilon(g_b-g_a)\). If \(b\) is active, transfers in either direction must not improve the objective, so their derivatives agree. If \(b\) is at zero, only transfers into \(b\) are feasible, and they must not improve it. Finally, \(\sum_m w_m g_m=\sum_i q_i/q_i=n\), so the common derivative among active models is \(n\).
Worked Example: An Interior Optimum for Two Models
Suppose two hypothetical candidate models give these probabilities for the observed category on each of three held-out scoring units:
| Scoring unit | Model A | Model B |
|---|---|---|
| 1 | 0.80 | 0.20 |
| 2 | 0.20 | 0.80 |
| 3 | 0.40 | 0.40 |
Let \(w\) be the weight on A, so that \(1-w\) is the weight on B. The three mixture probabilities are \(0.20+0.60w\), \(0.80-0.60w\), and \(0.40\). The objective is
Differentiating gives \(\ell'(w)=0.60/(0.20+0.60w)-0.60/(0.80-0.60w)\). Setting this equal to zero makes the first two mixture probabilities equal, which gives \(w=0.50\). The second derivative is negative throughout the feasible range, so this stationary point is the global maximum.
At \(w=0.50\), the mixture probabilities are \(0.50,0.50,0.40\), their product is \(0.1000\), and \(\ell(0.50)=\log(0.1000)=-2.3026\). Using A alone gives product \(0.80(0.20)(0.40)=0.06400\) and score \(-2.7489\); using B alone gives the same product and score. In this hypothetical example, the fitted mixture has the larger summed log predictive density on these three scoring units.
Boundary Weights and a Signed Score Difference
An optimum need not be interior. A boundary solution assigns all weight to one model, or assigns zero weight to one or more models. A small positive weight for a candidate model is beneficial only if it increases the objective; the direction of a score difference must be stated explicitly.
Worked Example: Checking Whether a Small Weight Helps
Consider four hypothetical held-out outcomes. Model A assigns probabilities \(0.80,0.80,0.80,0.20\), while model B assigns \(0.20,0.20,0.20,0.80\). Let \(\lambda\) be the weight on B. The mixture probabilities are \(0.80-0.60\lambda\) for each of the first three outcomes and \(0.20+0.60\lambda\) for the fourth. Therefore,
The derivative is \(-1.80/(0.80-0.60\lambda)+0.60/(0.20+0.60\lambda)\). Setting it to zero gives \(0.48-0.36\lambda=0.36+1.08\lambda\), hence \(\lambda=0.08333\). The second derivative is negative, so this is the maximum. At this weight, the mixture probabilities are \(0.75,0.75,0.75,0.25\), and the summed score is \(3\log(0.75)+\log(0.25)=-2.2493\).
With A alone, the score is \(3\log(0.80)+\log(0.20)=-2.2789\). Writing the comparison in the order “A alone minus the fitted mixture” gives
The negative difference means the fitted mixture's score is higher by 0.02953 log-score units on these outcomes. Reversing the subtraction would give \(+0.02953\). Reporting the subtraction order prevents the sign from obscuring the conclusion.
Choosing the Scoring Unit and Splitting the Data
For NLME predictions, records within a subject are often dependent through shared subject-level effects. If the external prediction target is a new subject's entire concentration profile, the natural score is commonly the joint predictive density of that profile under each model. The mixture then combines those subject-level joint densities, and the sum in the objective runs over subjects. Scoring every concentration as though it were an independent new subject would instead target a different prediction problem and can overstate the amount of information available for estimating weights.
If the target is a single future concentration conditional on a defined history, score that conditional predictive density for each target record. State what information is available at prediction time and ensure that the scoring calculation does not condition on the outcome being predicted. The choice of unit follows the external prediction target, not convenience.
Weight fitting is itself a data-adaptive step. A clean design distinguishes candidate-model development, weight estimation, and final performance evaluation. Candidate models are fitted and frozen without the weight-estimation outcomes; predictive scores on a separate stacking set determine the weights; and the final evaluation set remains untouched until both models and weights are fixed. If the same observations are used both to choose the weights and to report the final performance, the reported score is an in-sample score for the weight-fitting procedure and can be optimistic as an estimate of future performance.
Worked Example: Keeping the Final Test Set Untouched
Suppose two frozen models are evaluated on a hypothetical stacking set of two categorical outcomes. Model A assigns probabilities 0.90 and 0.10 to the outcomes observed; model B assigns 0.10 and 0.90. With weight \(w\) on A, both mixture probabilities are \(0.10+0.80w\) and \(0.90-0.80w\). Their product is maximized when the two probabilities are equal, giving \(w=0.50\). The stacking-set score is \(2\log(0.50)=-1.3863\).
Now apply the fixed weight to a separate hypothetical test set on which A's observed-outcome probabilities are 0.80 and 0.70, and B's are 0.20 and 0.30. The mixture probabilities at \(w=0.50\) are 0.50 and 0.50, so the test score is \(\log(0.50)+\log(0.50)=-1.3863\). For comparison, A alone would score \(\log(0.80)+\log(0.70)=\log(0.56)=-0.5798\) on this particular test set.
The mixture's lower score on this small test set does not contradict its stacking-set optimum: the two sets contain different outcomes, and two test observations provide little precision about relative performance. The key is that the test set did not determine the weight. If the weight were changed after seeing these test scores and the same data were then used to claim final performance, that claim would no longer be an untouched evaluation.
Diagnostics, Alternatives, and Limitations
After fitting weights, check the optimization and the predictive inputs rather than treating the returned numbers as self-validating. Useful checks include:
- Confirm that all weights are nonnegative, sum to one within numerical tolerance, and yield positive mixture density for every scored outcome.
- Recalculate the objective directly from the fitted weights and compare it with plausible boundary choices, such as each single model.
- Check whether the optimum is on a boundary, whether multiple solutions have nearly identical objective values, and whether small changes in scoring units produce large changes in weights.
- Verify that each row represents the intended target unit and that predictions were generated without using that unit's held-out outcome to fit or adapt the candidate model.
Weights estimated from a modest stacking set can be highly variable, especially when candidate predictions are similar or the number of independent scoring units is small. A large weight is not proof that a model is scientifically correct, and a zero weight does not show that a model is universally useless: the result describes performance for the scored target and data distribution. Stacking also cannot repair shared misspecification if all candidates predict poorly in the same way.
The main alternatives answer different questions. Equal weights avoid estimating weights from a limited stacking set but may score less well. Selecting one candidate by its score chooses a boundary solution and discards the other predictive distributions. A pre-specified weight avoids data-driven weight fitting, while still requiring an independent evaluation to measure performance. The introductory material on averaging predictive distributions explains why these approaches combine distributions rather than parameters; stacking adds a score-based rule for choosing the combination.
These fitted weights are not posterior model probabilities. Bayesian model averaging uses model probabilities and integrates over model uncertainty in a different way. That distinction matters for interpretation: stacking weights optimize predictive score on the weight-estimation data, rather than representing the probability that each candidate model is true.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two models assign probabilities 0.70 and 0.30 to one observed category. If the weight on the first model is 0.40, what is the mixture probability?
- Why does maximizing the stacking objective have a global optimum on the simplex, even if a numerical optimizer finds only a local maximum?
- If a weight-fitting set contains several measurements per subject and the target is prediction for a new subject's full profile, what should generally be the scoring unit?
- Why is a score reported on the same data used to estimate stacking weights not an untouched estimate of external performance?
Solutions. (1) The mixture probability is \(0.40(0.70)+0.60(0.30)=0.28+0.18=0.46\). (2) Each log of a positive affine function of the weights is concave, so their sum is concave; on the convex simplex, any local maximum is a global maximum. (3) Generally score the joint predictive density for the whole subject profile, so the scoring units match the new-subject prediction target and within-subject dependence is respected. (4) The weights were chosen to perform well on those outcomes, so the same scores reflect both weight fitting and evaluation; a separate untouched set is needed for a final external performance estimate.