Why Compare Models?
A household survey might record both the number of visits a family makes to a clinic and the fraction of its monthly budget spent on transportation. These outcomes need different response models: counts are nonnegative integers, while shares lie between zero and one. Even within each outcome type, plausible models can differ in their variance assumptions, link functions, and fitted predictions. Model comparison asks which candidate gives a credible description for the question at hand, not simply which produces the smallest standard error.
A useful comparison starts with the outcome’s support and the conditional mean of interest. It then examines whether the fitted mean behaves reasonably across covariate values, whether diagnostics reveal systematic shortcomings, and whether uncertainty is reported in a way appropriate to the estimation method. No single diagnostic establishes that a model is correct.
A Practical Comparison Framework
For a count outcome, Poisson regression commonly specifies a log conditional mean. A negative binomial model can use the same conditional mean while allowing a different variance. Under the NB2 variance specification, for example, the conditional variance is \(\mu_i+\alpha\mu_i^2\), where \(\mu_i=E(Y_i\mid X_i)\) and \(\alpha>0\). The distinction matters when counts vary more than the Poisson variance permits, but extra variation alone does not establish that the conditional mean is right.
For a fractional outcome \(Y_i\in[0,1]\), fractional logit specifies a logit link for the conditional mean, while fractional probit uses the standard normal cumulative distribution function. Both can accommodate observed zeros and ones as conditional-mean models. Their coefficients are on different link scales, so comparing their raw coefficient magnitudes is not informative. Compare predicted means, marginal effects, diagnostics, and out-of-sample performance instead.
A disciplined comparison checks whether candidate models use the same observations and covariates, whether predictions respect the outcome’s range, and whether residual or calibration summaries suggest systematic error. For likelihood-based candidates fitted to the same data, AIC can summarize fit relative to model complexity; lower AIC is preferred among those candidates, but a difference is not proof of truth. Robust standard errors do not change the likelihood or AIC. For conditional-mean estimators, robust inference is particularly useful when the working variance distribution may be imperfect, provided the mean specification and sampling assumptions needed for consistency are credible.
Name the outcome, population, and conditional quantity to explain or predict. Check that each candidate uses comparable observations.
Choose links and variance assumptions appropriate to the outcome, and state the assumptions before interpreting estimates.
Look for calibration, residual patterns, boundary behavior, and validation performance; use information criteria only when their comparison is valid.
State the effect in outcome units or a meaningful transformed scale, give robust uncertainty, and explain any model-selection limitations.
Comparing Models for Household Visit Counts
Worked Example: Poisson or Negative Binomial?
Suppose a hypothetical survey follows 240 households for one month and records each household’s number of clinic visits. The exposure period is the same for every household, so the models use the same observation window. Let \(x_i=1\) indicate receipt of a travel subsidy and \(x_i=0\) otherwise. The target is the conditional mean number of visits, not a causal effect: subsidy receipt was not randomly assigned.
A Poisson log-mean model and an NB2 log-mean model are fitted with the same covariate. Both use \(\log(\mu_i)=\beta_0+\beta_1x_i\). The fitted subsidy coefficients are \(-0.223\) for Poisson and \(-0.198\) for NB2. Their sandwich-robust standard errors for the NB2 coefficient are 0.104. Suppose the adjusted Pearson dispersion is 2.60 for Poisson and 1.08 for NB2; their AIC values are 624.8 and 581.2, respectively. These are hypothetical fitted summaries.
The Poisson dispersion above one indicates that the observed residual variation is large relative to the Poisson variance model, after adjustment for fitted means and model degrees of freedom. The NB2 dispersion is nearer one, and its AIC is lower by \(624.8-581.2=43.6\). Because the models use the same outcome and observations and are likelihood-based, these diagnostics favor NB2 for this comparison. They do not prove that NB2’s mean specification is correct, nor do they establish a causal subsidy effect.
Under the log link, exponentiating the coefficient gives the ratio of fitted conditional means for subsidy and no subsidy, holding other covariates fixed. For the NB2 estimate,
Thus the fitted mean for subsidy recipients is about 0.820 times that for otherwise comparable nonrecipients, or about 18.0% lower. The robust Wald interval for the coefficient is
Exponentiating both endpoints gives an interval for the mean ratio:
The interval includes 1, so the two-sided robust Wald test does not reject a zero coefficient at the 5% level. This does not establish that the effect is zero; the interval is compatible with a meaningfully lower mean and with a ratio close to one. For illustration, if the fitted mean without subsidy is 1.80 visits per household-month, the corresponding fitted mean with subsidy is \(1.80(0.8204)=1.477\) visits per household-month. These predictions describe the fitted association, not necessarily the effect of assigning a subsidy.
Comparing Models for Budget Shares
For a fractional response, the fitted mean must remain in \([0,1]\). Fractional logit uses \(\mu_i=\Lambda(X_i\beta)\), where \(\Lambda(t)=1/(1+\exp(-t))\); fractional probit uses \(\mu_i=\Phi(X_i\beta)\), where \(\Phi\) is the standard normal cumulative distribution function. These are conditional-mean specifications. Their use does not require the observed fraction to have a Bernoulli distribution. Robust standard errors are commonly used for inference under the mean specification even when the variance does not follow a Bernoulli formula.
Worked Example: Comparing Fractional Logit and Probit
A hypothetical survey records each household’s transportation budget share, including observed zeros. Let \(x_i\) indicate whether the household received a transit discount. Fractional logit estimates a coefficient of 0.350 for the discount, with robust standard error 0.140. Its fitted mean is 0.200 when \(x_i=0\) and 0.262 when \(x_i=1\). A fractional probit fit gives corresponding predicted means of 0.200 and 0.294. A held-out sample has RMSE 0.184 for logit and 0.186 for probit, and neither model shows a substantial calibration problem in the examined groups.
For fractional logit, the coefficient is a change in the log odds of the conditional mean, not a change of 0.350 in the budget share. The robust 95% Wald interval is
On the mean-odds scale, the estimated ratio is \(\exp(0.350)=1.419\), with interval \([\exp(0.0756),\exp(0.6244)]=[1.079,\ 1.867]\). This describes the odds of the conditional mean fraction, not the odds that a household experiences a binary event.
For a more direct interpretation, the fitted logit predictions imply a discrete change of \(0.262-0.200=0.062\), or about 6.2 percentage points of budget share. Suppose the model’s delta-method calculation using the full robust covariance matrix reports a robust standard error of 0.025 for this predicted contrast. The resulting interval is
This interval concerns the fitted change in conditional mean share, with units of share points. It is not obtained by treating the coefficient’s standard error as the standard error of the predicted contrast.
The held-out errors are very similar, while logit and probit predict different changes for the discount group. Their link-scale coefficients should not be compared directly. The researcher might prefer logit for a modest predictive advantage and an interpretable mean-odds ratio, but should describe the model choice as a practical judgment: a 0.002 RMSE difference alone is weak evidence that one link is generally superior.
Boundary Outcomes and the Target Population
Worked Example: What Does a Beta Model Compare?
Suppose a hypothetical sample contains 100 households’ monthly budget shares for a particular category. Twenty households report a share of exactly zero, and the mean among the other 80 households is 0.300. The overall sample mean is
A standard beta regression models outcomes strictly between zero and one. If it is fitted only to the 80 positive observations, its target is the share distribution among households with positive shares, whose sample mean here is 0.300. That is not the same target as the overall mean share of 0.240. Excluding the zeros and then describing the estimate as the average share for all 100 households would change the target population without acknowledging it.
A fractional logit conditional-mean model can retain the boundary observations and target the conditional mean for the full sample. The appropriate choice depends on the research question and the model’s assumptions; boundary support alone does not make every fractional model adequate. Here, if the target is the mean share among all households, retaining the zero-share households is essential. If the target is explicitly the share among households spending a positive amount, the positive-only analysis addresses a different but potentially valid question.
Short Justification: Why Exponentiate the Count Coefficient?
Suppose a count model specifies \(\log(\mu_i)=\beta_0+\beta_1x_i\), with \(x_i\) binary. At \(x_i=0\), the mean is \(\exp(\beta_0)\); at \(x_i=1\), it is \(\exp(\beta_0+\beta_1)\). Their ratio is therefore
This algebra justifies interpreting the exponentiated coefficient as a ratio of conditional means. If a Wald interval is first calculated for \(\beta_1\), exponentiating its two endpoints preserves their order because the exponential function is increasing. The resulting interval is on the ratio scale. This transformation does not turn an observational association into a causal effect; causal interpretation requires an appropriate research design and assumptions beyond the fitted regression form.
Common Pitfalls and Reporting
A comparison is useful only when it keeps the estimand and sample clear. Report what each model predicts, which diagnostics influenced the choice, and what uncertainty method was used. Robust standard errors address uncertainty around estimates under the stated conditions; they do not make predictions automatically calibrated, remove omitted-variable bias, or validate the chosen link.
- Choosing only by the smallest standard error. A model can produce precise estimates while fitting the conditional mean poorly. Check predictions and diagnostics as well as uncertainty.
- Comparing link-scale coefficients across logit and probit. The scales differ. Compare predicted means or marginal effects on the outcome scale.
- Treating a lower AIC as proof. AIC is a relative criterion for comparable likelihood fits, not a test that identifies the true model. Pair it with substantive checks and validation.
- Calling a robust standard error a remedy for misspecification. Sandwich uncertainty does not correct a wrong conditional mean or justify a causal interpretation.
- Ignoring the sample changed by boundary exclusions. A model fitted only to interior fractions may describe a selected subset rather than the full target population.
Check Your Understanding
Use the model assumptions and comparison principles in this tutorial to answer the following.
- A Poisson model has an adjusted Pearson dispersion of 2.4, while a negative binomial model has a value of 1.1. What does this suggest, and what does it not prove?
- A log-link count model estimates a coefficient of \(-0.15\) for a binary covariate. What is the fitted ratio of conditional means, and how should it be described?
- Why is it inappropriate to decide between fractional logit and fractional probit by comparing the absolute sizes of their coefficients?
- A 95% robust interval for a mean ratio includes 1. What conclusion is appropriate, and what conclusion would overstate the evidence?
- A beta regression excludes households with exact zero shares. How does that affect the population described by its fitted mean?