Tuning Conditional Distribution Models with Held-Out Scores

Why Tune With Held-Out Scores?

A conditional distribution model produces a distribution of possible responses at each covariate value, rather than only a point prediction. Its tuning parameters might include a kernel bandwidth, a series dimension, or a regularization level. Each choice changes the fitted conditional distributions, and a held-out score provides a common way to compare those choices on observations not used to fit them.

The score determines what differences between distributions matter, while the validation design determines which covariate values and outcomes contribute to the comparison. Thus, tuning is not just a computational step: it defines a finite-sample selection rule and, through the score and validation distribution, a target population risk.

Setup and Definitions

Let $(X,Y)$ have a target joint distribution, and let $P_x$ denote the conditional distribution of $Y$ given $X=x$. A candidate model $j$ supplies a predictive conditional distribution $Q_j(\cdot\mid x)$. The candidates may correspond to different bandwidths, series dimensions, regularization levels, or combinations of these. For a score $S(Q,y)$, smaller values indicate better forecasts.

Definition: The target integrated score risk of candidate $j$ is $$ R(j)=E\big[S(Q_j(\cdot\mid X),Y)\big]. $$ The expectation is over the target joint distribution of covariates and outcomes. Given a held-out sample $(X_i^{\mathrm{val}},Y_i^{\mathrm{val}})$, $i=1,\ldots,n_{\mathrm{val}}$, its empirical score is $$ \widehat R_{\mathrm{val}}(j)=\frac{1}{n_{\mathrm{val}}}\sum_{i=1}^{n_{\mathrm{val}}} S\big(Q_j(\cdot\mid X_i^{\mathrm{val}}),Y_i^{\mathrm{val}}\big). $$ The held-out-score selector is any index $\hat j$ minimizing $\widehat R_{\mathrm{val}}(j)$ over the candidate set.

A scoring rule is proper if, when the true distribution is $P$, its expected score under $Y\sim P$ is minimized by reporting $P$. It is strictly proper if $P$ is the unique minimizing distribution. For example, the log score is $S(Q,y)=-\log q(y)$ when the distributions have densities with respect to a common dominating measure. Its expected excess over the true density $p$ is the Kullback–Leibler divergence, when the expectations are well-defined. CRPS and energy scores are other proper scores for distributional forecasts.

For a strictly proper score, the conditional expected score at each $x$ is minimized by $P_x$. Consequently, the population risk compares candidates according to their conditional distribution quality, averaged over the target distribution of $X$. It does not, in general, weight every covariate region equally.

Worked Examples of Score-Based Selection

Worked Example: Choosing a Bandwidth by Gaussian Log Score

Suppose two bandwidths produce Gaussian predictive distributions with fixed variance $1$. At three held-out covariate values, the observed responses are $0,1,2$. Bandwidth $h_1$ predicts means $0,0,0$, while bandwidth $h_2$ predicts means $0,1,1$. For $N(m,1)$, the negative log density at $y$ is $$ -\log q(y)=\frac{1}{2}\log(2\pi)+\frac{1}{2}(y-m)^2. $$

For $h_1$, the squared residuals are $0^2,1^2,2^2$, with sum $5$. Its average held-out log score is therefore $$ \frac{1}{2}\log(2\pi)+\frac{5}{2\cdot 3} \approx 0.91894+0.83333=1.75227. $$ For $h_2$, the squared residuals are $0^2,0^2,1^2$, with sum $1$, so its average score is $$ \frac{1}{2}\log(2\pi)+\frac{1}{2\cdot 3} \approx 0.91894+0.16667=1.08561. $$ The held-out log score selects $h_2$. The common variance makes the comparison equivalent to comparing average squared residuals, but that equivalence would not hold for candidates with different variances.

Worked Example: Selecting a Series Dimension

Suppose three candidate series dimensions give the following negative log densities on four held-out observations. The entries are already on the same score scale.

Held-out observation	Dimension $K=2$	Dimension $K=4$	Dimension $K=8$
1	1.2	0.9	0.7
2	1.0	0.8	1.4
3	1.1	0.9	0.8
4	0.9	1.0	1.1
Mean	1.05	0.90	1.00

The calculations are $(1.2+1.0+1.1+0.9)/4=1.05$, $(0.9+0.8+0.9+1.0)/4=0.90$, and $(0.7+1.4+0.8+1.1)/4=1.00$. The selected dimension is $K=4$. The lowest dimension is not automatically preferred: the score compares predictive performance, including any benefits and costs reflected in its predictions. Nor does this one validation average establish that $K=4$ is best for every population or every score.

Worked Example: Validation Weights Can Change the Selected Model

Imagine two covariate strata, A and B. Candidate models have mean held-out score losses of $0.1$ and $3.0$ in A and B, respectively, for candidate $M_1$; candidate $M_2$ has losses $0.8$ in both strata. Suppose a validation sample has 90% of its observations in A and 10% in B. Its averages are $$ \widehat R(M_1)=0.9(0.1)+0.1(3.0)=0.39,\qquad \widehat R(M_2)=0.9(0.8)+0.1(0.8)=0.8. $$ It selects $M_1$.

If the target population instead gives equal weight to A and B, the corresponding target risks are $$ R(M_1)=0.5(0.1)+0.5(3.0)=1.55,\qquad R(M_2)=0.5(0.8)+0.5(0.8)=0.8. $$ That target selects $M_2$. The calculations show how a validation sample can favor a model that performs well in the covariate regions it represents, even when another model has lower risk for the intended target population. The example uses stratum-average score losses; it does not require that these particular values arise from a particular parametric family.

What a Finite-Candidate Validation Guarantee Says

A held-out score can reliably rank a finite collection when its variability is controlled and the validation observations are independent of model fitting. The following result makes the role of these conditions explicit. Treat the fitted candidate distributions as fixed after conditioning on the training data.

Theorem: Suppose there are $M$ fixed candidate models, the validation observations are independent and identically distributed from the target joint distribution, and every candidate's validation score lies in a common interval $[a,b]$ almost surely, with $a<b$. Let $j^\star$ minimize the target risk $R(j)$, and let $\hat j$ minimize the empirical validation score. Then, for any $0<\delta<1$, with probability at least $1-\delta$, $$ R(\hat j)-R(j^\star) \leq (b-a)\sqrt{\frac{2\log(2M/\delta)}{n_{\mathrm{val}}}}. $$

Proof. For a fixed candidate $j$, its $n_{\mathrm{val}}$ validation scores are independent, bounded in $[a,b]$, and have mean $R(j)$. Hoeffding's inequality gives, for any $\varepsilon>0$, $$ P\left(\left|\widehat R_{\mathrm{val}}(j)-R(j)\right|>\varepsilon\right) \leq 2\exp\left(-\frac{2n_{\mathrm{val}}\varepsilon^2}{(b-a)^2}\right). $$ Applying the union bound over the $M$ candidates shows $$ P\left(\max_{1\leq j\leq M}|\widehat R_{\mathrm{val}}(j)-R(j)|>\varepsilon\right) \leq 2M\exp\left(-\frac{2n_{\mathrm{val}}\varepsilon^2}{(b-a)^2}\right). $$ Choose $\varepsilon=(b-a)\sqrt{\log(2M/\delta)/(2n_{\mathrm{val}})}$; the last probability is at most $\delta$. On the complementary event, empirical minimization gives $\widehat R_{\mathrm{val}}(\hat j)\leq \widehat R_{\mathrm{val}}(j^\star)$, and hence $$ \begin{aligned} R(\hat j)-R(j^\star) &=[R(\hat j)-\widehat R_{\mathrm{val}}(\hat j)] +[\widehat R_{\mathrm{val}}(\hat j)-\widehat R_{\mathrm{val}}(j^\star)]\\ &\quad+[\widehat R_{\mathrm{val}}(j^\star)-R(j^\star)]\\ &\leq \varepsilon+0+\varepsilon=2\varepsilon. \end{aligned} $$ Substituting the chosen $\varepsilon$ proves the bound. $\square$

The result compares the selected candidate to the best candidate in the specified finite set, not to the true conditional distribution or to every possible model. It also requires bounded scores. Log scores are generally unbounded, so this theorem cannot be applied to them without additional assumptions or a different concentration argument. A finite empirical log score alone does not establish the theorem's guarantee.

Why Strict Propriety Matters

Propriety explains what a population score comparison means. Suppose $P_x$ is the true conditional law and define the conditional risk of a forecast $Q$ by $$ r_x(Q)=E[S(Q,Y)\mid X=x]. $$ Assume the score is strictly proper and these conditional expectations exist. Then $r_x(Q)\geq r_x(P_x)$ for each $x$, with equality only when $Q=P_x$ (up to the usual equivalence of distributions). Integrating over $X$ gives $$ E[S(Q(\cdot\mid X),Y)]-E[S(P_X,Y)] =E\big[r_X(Q(\cdot\mid X))-r_X(P_X)\big]\geq 0. $$ If equality holds, the nonnegative integrand must be zero almost surely, so strict propriety implies $Q(\cdot\mid X)=P_X$ almost surely. This establishes the conditional distribution target of population minimization; it does not mean that a finite validation sample will identify the best model without error.

Different proper scores can still rank imperfect candidate models differently. They penalize discrepancies in different ways: the log score strongly reflects the assigned density at the realized outcome, while CRPS evaluates cumulative distribution discrepancies and the energy score evaluates distances between outcomes and forecast draws. When no candidate equals the truth, minimizing one score's risk need not select the same candidate as minimizing another's. Score choice should therefore reflect the predictive task and the kinds of distributional errors that matter.

Validation Design and Practical Interpretation

The simple theorem assumes an independent validation sample from the target joint distribution. In applications, several design choices affect whether that assumption and target are appropriate.

Keep fitting separate from scoring. Fit every candidate using training data, then compute its score on the same held-out observations. If a candidate is modified after inspecting those scores, the holdout has also become tuning data.
Match the target covariate distribution. The ordinary average estimates risk under the validation distribution of $X$. If the intended population has a different covariate distribution, appropriate weighting or a validation sample representative of that population is needed; weights must be chosen and justified for the target.
Respect dependence in the sampling design. Repeated observations on people, sites, or time periods may not be independent. A random row-wise split can then leak information across training and validation. The split and uncertainty assessment should reflect the units and dependence structure relevant to prediction.
Do not overinterpret a narrow score difference. Two candidates with nearly equal validation averages may be difficult to distinguish at the available sample size. The finite-candidate bound illustrates that sampling uncertainty grows with the score range and the number of candidates and decreases with validation sample size.
Refitting is not rescoring. After selecting a tuning value, it is common to refit using more data. The held-out score describes the candidates as evaluated under the original fitting procedure; it is not automatically an unbiased score for the final refitted model.

Cross-validation can reuse observations across folds to make more efficient use of data, but the fold-specific fitted models and scores are dependent. The independent-validation theorem above does not apply to those fold averages without an argument suited to that design. Whatever procedure is used, state which observations train each fit, which observations score it, and what population the resulting average is intended to represent.

Common Mistake

Pitfall: “The model with the lowest validation score is the best conditional distribution model.” This overstates what the comparison establishes. The selected candidate minimizes an empirical score among the candidates tried, for the validation design and score used. Sampling variability, an unrepresentative covariate distribution, model-fitting differences, and reuse of validation data can all affect the result. A proper score targets the true distribution at the population level, but it does not remove these finite-sample and design limitations.

A careful conclusion names the score, candidate set, validation design, and observed result. For example: “Among these bandwidths, the specified held-out log score selected $h_2$ on this validation sample.” That statement is informative without claiming that $h_2$ is universally optimal.

Check Your Understanding

Answer the questions by referring to the score, candidate set, and validation target.

What population risk does an unweighted held-out score average estimate?
Why does the finite-candidate theorem require bounded scores, and why is that condition not automatic for the log score?
Under what condition does strict propriety identify the true conditional distribution as the population minimizer?
How can two proper scores select different candidates when neither candidate is the true conditional distribution?
In the two-stratum example, why does the validation sample select a different model from the equal-weight target population?
Why can repeatedly modifying candidates after inspecting a holdout invalidate its role as independent validation data?