Comparing Power Indices for Prediction

Prediction Is Not the Same Question as Inference

A Tweedie power affects the shape of the response distribution as well as how its variance changes with the mean. When several powers seem plausible, a useful question is which fitted model predicts new responses more accurately. That is a predictive question: its answer depends on performance for observations that were not used to fit the candidate model.

For semicontinuous outcomes with both exact zeros and positive amounts, a single overall score can conceal an important failure. A model may predict the positive amounts well but misstate how often zeros occur, or get the zero frequency right while predicting positive amounts poorly. Comparing powers therefore calls for both an overall held-out score and targeted calibration checks.

Write \(p\) for the Tweedie power and consider candidates \(1<p<2\), the range with a point mass at zero and a continuous distribution on positive values. For each candidate, estimate the regression and dispersion parameters using training data only. For a held-out response \(y_i\) and its covariates \(x_i\), let \(f_{p,i}(y_i)\) denote the candidate's predictive density or mass, evaluated at the response using the fitted parameters. The held-out mean negative log score is

where \(m\) is the number of held-out observations. A smaller \(S_p\) indicates better average predictive performance on that held-out set. If using cross-validation, each observation is scored using a model fitted without that observation's fold; the fold scores can then be averaged.

What the Predictive Score Measures

For \(1<p<2\), the Tweedie response distribution has a mixed form: an atom at zero and a density for positive outcomes. A common dominating measure is the sum of a point mass at zero and Lebesgue measure on the positive half-line. With this convention, \(f(0)\) means the probability \(P(Y=0)\), while \(f(y)\) for \(y>0\) is a density. Both are evaluated as contributions to a log score. The positive density has units determined by the response scale, so scores are comparable only when candidates use the same response scale and scoring convention.

The score should assess the full predictive distribution, not just a fitted mean. A log score uses the Tweedie mass at zero for a zero response and the Tweedie density for a positive response. In practice, the density may be evaluated by an exact series or an approximation; whichever method is used should be applied consistently to all candidates.

Why the Log Score Rewards Honest Predictive Distributions

The reason to use a proper score is that it does not reward a model merely for making confident predictions that happen to fit the training data. The following result states the population property of the negative log score. It applies to discrete, continuous, and mixed outcomes when their distributions have densities with respect to a common dominating measure.

This result concerns expected performance under the true distribution. A finite held-out sample is noisy, so a lower observed score is evidence for better prediction, not a guarantee. Repeated folds or an independent test set can help assess whether a difference is stable.

Separate Calibration Checks for Zeros and Positive Amounts

A log score summarizes the entire predictive distribution, but it does not by itself explain where a candidate succeeds or fails. In the compound Poisson-Gamma representation for \(1<p<2\), the number of positive jumps is Poisson with mean \(\lambda=\mu^{2-p}/\{\phi(2-p)\}\), where \(\mu=E[Y]\) and \(\phi\) is the dispersion parameter. The response is zero exactly when there are no jumps. This gives a direct zero-probability prediction.

To verify the first formula, if \(N\) is the Poisson jump count, then \(P(Y=0)=P(N=0)=e^{-\lambda}\). For the second, \(Y=0\) on \(N=0\), so \(E[Y]=E[Y\,1_{\{Y>0\}}]=P(Y>0)E[Y\mid Y>0]\). Since \(E[Y]=\mu\) and \(P(Y>0)=1-P(Y=0)\), rearrangement gives the stated conditional mean.

For a set \(G\) of observations with similar predicted zero probabilities \(q_i=P(Y_i=0\mid x_i)\), compare the observed zero fraction with the average predicted probability, \(|G|^{-1}\sum_{i\in G}q_i\). Calibration means that these quantities agree across groups, not merely in the full sample. For positive amounts, compare observed positive outcomes with their predicted conditional distributions or conditional means. When predictions vary within a group, compare observed positive totals and counts with their corresponding expectations rather than treating a single unweighted average as exact.

Predictive Power Selection Versus Likelihood Inference

A power chosen to minimize a held-out score is a tuning choice for prediction. It need not equal the power that maximizes the likelihood for a particular observed sample, and neither choice automatically quantifies uncertainty about the power. Profile likelihood, likelihood-based estimation, and uncertainty assessment answer inferential questions; held-out scoring asks which fitted predictive procedure performs better on excluded observations.

For a fair predictive comparison, the full fitting procedure must be repeated using only the training portion of each split. This includes estimating regression coefficients and dispersion, and choosing any preprocessing or tuning settings. If the power itself is selected using the held-out responses and those same responses are then used to report performance, the reported score is optimistic: the holdout has influenced the choice it is supposed to evaluate.

A common mistake is to compare training log-likelihoods across powers and call the largest one the best predictor. A training likelihood rewards fit to the observations used for estimation and does not measure out-of-sample performance. Another mistake is to compare only predicted means. Different powers can yield similar means while assigning very different probabilities to zero or different distributions to positive values. Use a held-out score for overall prediction, then inspect zero and positive calibration to understand the result.