Comparing Logit and Probit Probability Predictions

Why Compare Probability Predictions?

Logit and probit models both describe the probability of a binary outcome as a function of covariates. Their link functions differ, so fitted probabilities need not be identical even when the models use the same predictors. The practical question is often not whether their coefficients match, but whether they give meaningfully different probabilities for people, firms, or other units of interest.

A fair comparison holds the covariate profile fixed: evaluate both fitted models for the same values of every predictor. Comparing predictions for different profiles confounds model differences with differences in the units being compared. It is also important to distinguish a numerical difference from a substantive one. A difference of one percentage point may be immaterial for one decision and consequential for another.

Definition: For a binary outcome \(Y\) and covariate profile \(x\), the fitted logit and probit probabilities are \(\hat p_L(x)=\Lambda(x'\hat\beta_L)\) and \(\hat p_P(x)=\Phi(x'\hat\beta_P)\). Here \(\Lambda(t)=1/(1+\exp(-t))\) is the standard logistic cumulative distribution function, \(\Phi(t)\) is the standard normal cumulative distribution function, and the coefficient estimates are obtained separately for the two models. The pointwise prediction difference is \(\hat\Delta(x)=\hat p_L(x)-\hat p_P(x)\).

The sign of \(\hat\Delta(x)\) indicates which model predicts the higher probability at that profile; its magnitude is measured in probability units. For example, \(\hat\Delta(x)=0.03\) means the logit prediction is 0.03 higher, or 3 percentage points higher, than the probit prediction. It does not mean the logit model’s coefficient is 3% larger.

Make the Comparison on Common Profiles

The models should be fit to the same outcome and, for a direct link comparison, use the same sample, covariates, coding, and estimation conventions. Then evaluate each model at an identical set of profiles. Profiles might be observed cases, a substantively chosen reference case, or a grid of covariate values within the data’s support. Using profiles beyond the observed range can produce extrapolations that are difficult to interpret.

Logit coefficients are on a conditional log-odds scale, whereas probit coefficients are on a latent-index scale. Their numerical sizes therefore do not have a direct one-to-one interpretation. A common approximate relationship between coefficient scales is not a rule that makes the models’ predictions equal: the fitted coefficients are estimated from the data, and the relationship between the two link functions is not exact. Compare predicted probabilities, not raw coefficients, when the question concerns probability predictions.

Worked Example: Comparing Predictions Across the Index

Suppose two hypothetical fitted models are evaluated at three common profiles. At those profiles, the logit linear predictors are \(-2\), \(0\), and \(2\); the probit linear predictors are \(-1.1\), \(0\), and \(1.1\). Calculate and compare the predicted probabilities.

For logit, use \(\Lambda(t)=1/(1+\exp(-t))\). For probit, use \(\Phi(t)\). At the first profile,

\hat p_L=\Lambda(-2)=\frac{1}{1+\exp(2)}\approx 0.1192,\qquad \hat p_P=\Phi(-1.1)\approx 0.1357.

Thus \(\hat\Delta\approx 0.1192-0.1357=-0.0165\): the logit prediction is about 1.65 percentage points lower. At the middle profile, both indices are zero, so \(\Lambda(0)=\Phi(0)=0.5000\), and the difference is zero.

At the third profile,

\hat p_L=\Lambda(2)=\frac{1}{1+\exp(-2)}\approx 0.8808,\qquad \hat p_P=\Phi(1.1)\approx 0.8643,

so \(\hat\Delta\approx 0.8808-0.8643=0.0165\). The difference changes sign across the profiles. This illustration shows why a single statement such as “logit always predicts higher probabilities” is not justified. The direction and size of the difference depend on the fitted indices and the profile.

Summarize Differences Across Profiles

A comparison at one profile answers a local question. If the substantive question concerns a population or a range of covariate values, calculate differences across a common set of profiles. Useful summaries include the mean signed difference, the mean absolute difference, and the largest absolute difference. For profiles \(x_1,\ldots,x_n\), with equal weights, these are

\overline{\Delta}=\frac{1}{n}\sum_{i=1}^n\hat\Delta(x_i),\qquad \operatorname{MAD}=\frac{1}{n}\sum_{i=1}^n|\hat\Delta(x_i)|,\qquad D_{\max}=\max_i|\hat\Delta(x_i)|.

The mean signed difference can conceal offsetting differences: positive and negative values may average near zero even when individual predictions differ substantially. The mean absolute difference describes typical discrepancy without cancellation, while the maximum identifies the largest discrepancy among the chosen profiles. If profiles represent a target population unequally, use weights that reflect that target rather than treating every profile as equally common.

Worked Example: Do the Models Change a Treatment Comparison?

In a hypothetical analysis, both models are evaluated for otherwise identical profiles with and without a training program. The fitted logit probabilities are 0.4013 without training and 0.5498 with training. The fitted probit probabilities are 0.4013 without training and 0.5398 with training. Compare the predicted probability changes.

For logit, the predicted change is

$$ 0.5498-0.4013=0.1485, $$

or 14.85 percentage points. For probit, it is

$$ 0.5398-0.4013=0.1385, $$

or 13.85 percentage points. The difference between the models’ predicted changes is \(0.1485-0.1385=0.0100\), or 1.00 percentage point. Both models predict a higher outcome probability for the profile with training, and the estimated changes are similar in this example. The comparison supports the conclusion that the direction and approximate size of the predicted change are not very sensitive to the link choice here. It does not establish that training causes the change; that interpretation requires appropriate causal assumptions and design.

Judge Differences in the Substantive Context

There is no universal cutoff that makes two probability predictions “close enough.” Judge the discrepancy against the decision, policy, or scientific question. Ask whether the models agree on the direction of a relevant comparison, whether they differ by a practically important number of percentage points, and whether a difference changes a conclusion that depends on a particular probability value.

For example, predictions of 0.49 and 0.51 differ by only two percentage points, but they fall on opposite sides of a 0.50 decision boundary. Conversely, predictions of 0.20 and 0.22 may lead to the same immediate decision while still mattering if many individuals are affected or the outcome is costly. A boundary is relevant only when the application provides a reason to use it; it is not a general test of model quality.

Differences may be small near some parts of the probability range and more noticeable elsewhere. The comparison should therefore include profiles where the outcome is relatively unlikely, profiles with intermediate predicted probabilities, and profiles where it is relatively likely, when those profiles are relevant and supported by the data. Do not infer that the models agree throughout the covariate space from agreement at one reference case.

Worked Example: Quantifying Uncertainty in a Prediction Difference

At a particular hypothetical profile, suppose the estimated probabilities are \(\hat p_L=0.70\) and \(\hat p_P=0.66\), so \(\hat\Delta=0.04\), or 4 percentage points. Suppose a paired resampling procedure that refits both models to the same resampled data gives estimated variances \(0.000900\) and \(0.000784\) for the two predicted probabilities, and estimated covariance \(0.000420\). Find an approximate standard error and 95% interval for the difference.

For two random quantities, the variance of their difference is the sum of their variances minus twice their covariance. Therefore,

\widehat{\operatorname{Var}}(\hat\Delta) =0.000900+0.000784-2(0.000420) =0.000844, \qquad SE(\hat\Delta)=\sqrt{0.000844}\approx 0.0291.

Using a normal approximation, the interval is

0.0400\pm 1.96(0.0291) =0.0400\pm 0.0570 \approx [-0.0170,\ 0.0970].

The interval includes zero, so these hypothetical results do not clearly distinguish the two models’ probabilities at this profile at the 5% two-sided level. The interval is for the difference in predicted probabilities, not for either probability alone. The covariance matters because both models were fit to the same data; treating the predictions as independent would discard their dependence and could give an incorrect standard error. The interval also does not say that there is a 95% probability the fixed difference lies inside these particular endpoints.

Practical Comparison and Common Pitfalls

A disciplined comparison proceeds from the target question to the profiles and then to the differences. First specify which probability or probability contrast matters. Next ensure that both models use comparable data and covariate definitions. Then compute fitted probabilities at common profiles, summarize their differences over the relevant profile set, and assess uncertainty when inference about the differences is needed.

A model comparison can be hard to interpret if the models use different samples, predictor sets, or variable coding. In that case, an observed prediction difference may reflect more than the link function. Similarly, reporting only an average signed difference may hide profiles at which the predictions disagree in opposite directions. State how profiles were selected and what population, if any, their summaries represent.

Pitfall: comparing coefficients as though they were on the same scale. A larger numerical probit coefficient does not by itself imply a larger probability effect. Compute probabilities or probability contrasts on common profiles.
Pitfall: treating agreement at one profile as general agreement. The fitted curves can differ elsewhere. Examine a relevant range of profiles, not just a single convenient case.
Pitfall: interpreting a small average signed difference as universal agreement. Positive and negative differences can cancel. Also inspect absolute differences and their range.
Pitfall: calling a probability difference causal. Logit–probit comparison concerns fitted predictions under the specified models; it does not by itself identify a causal effect.

Key takeaway: Compare fitted logit and probit probabilities at identical, substantively relevant covariate profiles. Report differences in probability units, examine their variation across profiles, and decide whether they affect the conclusion of interest. Similar predictions at selected profiles are evidence of robustness to link choice for those profiles—not proof that the models are equivalent everywhere.

Check Your Understanding

Answer each question using the definitions and comparison principles in this tutorial.

At a profile, the logit prediction is 0.72 and the probit prediction is 0.68. What is \(\hat\Delta(x)\), in probability units and percentage points?
Why is it generally inappropriate to compare the numerical sizes of a logit coefficient and a probit coefficient directly?
What can happen to a mean signed prediction difference when some profiles have positive differences and others have negative differences?
Why should a resampling procedure for comparing predictions refit both models on the same resampled data?
Give one reason that a two-percentage-point difference could matter in a particular application, and one reason it might not.