Calibration Checks For Logistic Association Models

What Calibration Checks Ask

A logistic association model produces a fitted probability of case status for each participant, conditional on the SNP and included covariates. Calibration checks ask whether those probabilities agree with the observed case frequencies in relevant parts of the data. For example, among participants assigned probabilities near 0.10, do case outcomes occur at about a 10% rate? A systematic mismatch may indicate that the fitted model does not describe the conditional outcome probabilities adequately.

This is a check of the individual association model, not a check that genome-wide association test statistics have the expected distribution across many variants. The latter concerns scan-wide test calibration and is a different question. A model can have a plausible fit at one SNP while a scan-wide analysis has other sources of distortion, or vice versa.

Model and Assumptions

For participant \(i\), let \(Y_i=1\) denote a case and \(Y_i=0\) a control; let \(X_i\) be the coded SNP genotype, and let \(C_i\) be a vector of covariates. A covariate-adjusted logistic model specifies

Y_i\mid X_i,C_i\sim \operatorname{Bernoulli}(p_i),\qquad \operatorname{logit}(p_i)=\log\!\left(\frac{p_i}{1-p_i}\right) =\beta_0+\beta_G X_i+C_i^{T}\gamma .

Here \(p_i=P(Y_i=1\mid X_i,C_i)\), \(\beta_G\) is the conditional log-odds change per one-unit increase in the chosen genotype coding, and \(\gamma\) contains the covariate coefficients. For an additive dosage coded 0, 1, 2, one genotype unit is one additional counted allele. The inverse-logit transformation is \(p_i=\exp(\eta_i)/(1+\exp(\eta_i))\), where \(\eta_i=\beta_0+\beta_GX_i+C_i^T\gamma\).

The checks below presume that the observations are independent conditional on the included predictors, the binary outcome and predictors are recorded appropriately, and the conditional mean is adequately represented by the model. In particular, the log odds must have the specified form in genotype and covariates; omitted nonlinear patterns or interactions can undermine fit. Standard model-based inference also presumes an estimable finite fit. Related participants or other dependence can violate conditional independence, and a usual logistic fit does not by itself handle that dependence.

Definition: A model is calibrated in a group of participants if their average fitted probability is close to their observed case proportion. If the group contains \(m\) participants, with \(O\) observed cases and fitted probabilities \(\hat p_i\), the observed proportion is \(O/m\), the expected case count is \(E=\sum_i\hat p_i\), and the average fitted probability is \(E/m\). Calibration is about agreement in probability, not merely about whether a SNP coefficient is statistically significant.

Individual Residuals and Their Interpretation

For a fitted model, the basic response residual is \(r_i=Y_i-\hat p_i\). A case has residual \(1-\hat p_i\), while a control has residual \(-\hat p_i\). The Bernoulli variance conditional on the predictors is \(p_i(1-p_i)\), since \(Y_i\) takes values 0 and 1 with probabilities \(1-p_i\) and \(p_i\). This gives the Pearson residual

r_{Pi}=\frac{Y_i-\hat p_i}{\sqrt{\hat p_i(1-\hat p_i)}}.

The denominator standardizes the response residual using its model-based standard deviation. Unlike residuals in a normal linear model, binary-response residuals are not expected to form a bell-shaped cloud with constant spread: their possible values and variance depend on \(\hat p_i\). A large Pearson residual can flag an observation the model predicts poorly, but it is not by itself proof of a coding error or an invalid model.

A deviance residual is another useful summary. For a binary observation it is

r_{Di}=\operatorname{sign}(Y_i-\hat p_i) \sqrt{2\left[ Y_i\log\!\left(\frac{Y_i}{\hat p_i}\right) +(1-Y_i)\log\!\left(\frac{1-Y_i}{1-\hat p_i}\right) \right]} .

Terms of the form \(0\log(0)\) are taken as zero. Both residual types help locate observations or predictor regions with mismatches. Looking for patterns across fitted probability, genotype, or covariates is usually more informative than treating each residual as a stand-alone test.

Worked Example: Computing a Fitted Probability and Residual

Suppose a hypothetical fitted model is \(\operatorname{logit}(\hat p_i)=-2.00+0.40X_i+0.03A_i\), where \(X_i\) is genotype dosage and \(A_i\) is age in years. For a 50-year-old participant with dosage 1 who is a case, calculate the fitted probability and Pearson residual.

The linear predictor is \(\hat\eta_i=-2.00+(0.40)(1)+(0.03)(50)=-0.10\). Therefore,

\hat p_i=\frac{\exp(-0.10)}{1+\exp(-0.10)} =\frac{0.9048}{1.9048}\approx 0.4750.

Since \(Y_i=1\), the response residual is \(1-0.4750=0.5250\). The fitted Bernoulli standard deviation is \(\sqrt{(0.4750)(0.5250)}\approx 0.4994\), giving

r_{Pi}=\frac{0.5250}{0.4994}\approx 1.051.

This participant was a case despite a fitted probability below one half, so the residual is positive. Its size alone does not establish poor overall calibration; the residual should be considered alongside other observations and model diagnostics.

Grouped Calibration and Residual Patterns

A calibration table or plot groups participants by fitted probability and compares the average fitted probability with the observed case proportion. Groups may be formed using ranges chosen in advance or quantiles of fitted probabilities. The plot can expose a pattern hidden by an overall average: for instance, overprediction in low-risk groups and underprediction in high-risk groups. Grouping is a summary, however, and the result can depend on how boundaries are chosen.

Residuals plotted against fitted probabilities or covariates can also reveal structure. A curved trend across a continuous covariate may suggest that a linear term on the log-odds scale is inadequate. A systematic difference by genotype category may point to a mismatch between the chosen genotype coding and outcome pattern. Residual patterns associated with a covariate can indicate that its functional form or an omitted interaction deserves investigation. Such patterns diagnose possibilities; they do not uniquely identify the source.

Worked Example: Comparing Observed and Expected Cases

In a hypothetical dataset, two groups each contain 100 participants. In the first, the model assigns every participant fitted probability 0.10 and 14 are cases. In the second, each fitted probability is 0.30 and 25 are cases. Compare the grouped calibration summaries.

For group 1, the expected case count is \(E_1=100(0.10)=10\), compared with \(O_1=14\). The observed proportion is \(14/100=0.1400\), against an average fitted probability of 0.1000. For group 2, \(E_2=100(0.30)=30\), compared with \(O_2=25\). Its observed proportion is 0.2500, against an average fitted probability of 0.3000.

Across both groups, the total observed count is \(14+25=39\), and the total expected count is \(10+30=40\). The overall proportions, 0.1950 observed and 0.2000 expected, are close, but that closeness masks the opposing group discrepancies. This is why a single whole-sample comparison is not enough to assess calibration.

For a descriptive comparison, a grouped Pearson discrepancy can be calculated from case and control counts. In group 1 it is \((14-10)^2/10+(86-90)^2/90=1.600+0.1778=1.778\). In group 2 it is \((25-30)^2/30+(75-70)^2/70=0.8333+0.3571=1.190\). The sum is \(1.778+1.190=2.968\). This number summarizes the grouped differences; it should not be assigned a universal cutoff or p-value without a justified reference distribution that accounts for grouping and model fitting.

Convergence Output Is Part of the Diagnostic

A fitted probability is only useful if the optimization produced a trustworthy estimate. Review the convergence flag, iteration count, warnings, coefficient estimates, standard errors, and any indication that the information matrix could not be inverted or was nearly singular. Convergence means that the numerical algorithm met its stopping criterion; it does not prove that the model is well specified. Conversely, a failure to converge means that ordinary coefficient, standard error, and probability output should not be treated as a completed fit.

A frequent cause of difficulty is separation: some predictor pattern perfectly distinguishes cases from controls. In that situation, the likelihood may increase as a coefficient grows in magnitude, without reaching a maximum at any finite coefficient. Sparse genotype-by-case patterns can also produce unstable estimates. These issues are especially relevant when reviewing SNP models, and should prompt inspection of genotype coding, covariate patterns, cell counts, and the optimizer messages rather than uncritical interpretation of a very large odds ratio.

Worked Example: Recognizing a Nonfinite Logistic Fit

Consider hypothetical data with 30 participants at genotype dosage 0, all controls, and 30 at dosage 1, all cases. Fit an intercept and a dosage coefficient. Let the intercept be \(-t\) and the dosage coefficient be \(2t\), for \(t>0\). The fitted case probabilities for dosage 0 and 1 are then \(\operatorname{expit}(-t)\) and \(\operatorname{expit}(t)\), respectively.

The log likelihood for these observations is

\ell(t)=30\log\!\left(1-\operatorname{expit}(-t)\right) +30\log\!\left(\operatorname{expit}(t)\right).

As \(t\) increases, \(\operatorname{expit}(-t)\) approaches 0 and \(\operatorname{expit}(t)\) approaches 1. Thus both log terms approach \(\log(1)=0\), so \(\ell(t)\) approaches 0. No finite \(t\) achieves those limiting probabilities exactly, and each log probability is at most zero. The likelihood therefore has no finite maximizing coefficient along this path. An optimizer may report nonconvergence, a large coefficient, or an exceptionally large standard error, depending on its settings. Those outputs are warnings of a nonfinite or unstable estimate, not evidence for a precisely quantified large effect.

Choosing and Interpreting Checks

Calibration plots and grouped summaries are direct checks of fitted probabilities. Residual plots help identify which predictor regions contribute to discrepancies. Convergence output addresses whether the numerical fit can be trusted at all. These checks complement one another: a model can converge but show poor calibration, or it can appear to fit the observed groups well while having unstable coefficient estimates.

A formal grouped goodness-of-fit test is an alternative to visual or descriptive summaries, but its result depends on grouping and on approximations that may be unreliable with small expected counts. A smooth calibration curve can avoid some arbitrary group boundaries, but depends on smoothing choices and can be unstable in regions with few observations. No single summary replaces inspection of the data support and model form. In particular, do not use a whole-sample observed-versus-expected total as a substitute for examining probability ranges or covariate patterns.

In case-control studies, the case fraction is often set by sampling rather than reflecting population prevalence. A model intercept and its fitted probabilities can therefore describe the sampled design without being calibrated to absolute population risk. The odds-ratio association can still be of interest, but interpreting fitted probabilities as population risks requires attention to the sampling design. Also, calibration in the data used to fit the model is not evidence that predictions will calibrate in a new population.

Common Mistakes and Practical Review

Expecting normal residuals. The outcome is binary, and the residual distribution depends on fitted probability. Look for systematic patterns and unusually large residuals, not a normal-looking histogram.
Equating convergence with good fit. Convergence is a numerical condition, not a guarantee that the conditional mean is correct. Check residual and grouped calibration patterns as well.
Equating a significant SNP test with calibration. A test of \(\beta_G=0\) asks about association under the fitted model; it does not establish that fitted probabilities agree with observed frequencies.
Confusing model calibration with scan-wide test calibration. The first concerns probabilities from a particular logistic model; the second concerns the distribution of association statistics across a genome-wide scan.
Interpreting a case-control fitted probability as population risk automatically. Case-control sampling can alter the sampled case fraction, so absolute-risk interpretation needs design information.

Key takeaway: Assess a logistic SNP model by combining fitted-probability comparisons, residual patterns, and optimizer diagnostics. These checks address the fit and numerical reliability of that model; they do not replace genome-wide test-statistic calibration checks.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A group of 80 participants has average fitted probability 0.20 and contains 20 cases. What are the expected case count and observed case proportion, and what does the comparison suggest?
Why is it inappropriate to require Pearson residuals from a logistic model to have constant variance across fitted probabilities?
A SNP model converges, but observed case proportions are consistently higher than fitted probabilities in the highest-risk groups. Does convergence resolve this discrepancy?
What does a very large coefficient and standard error together with nonconvergence suggest, and what should be inspected?
In one sentence, distinguish a logistic-model calibration check from genome-wide test-statistic calibration.

Solutions. (1) The expected count is \(80(0.20)=16\); the observed proportion is \(20/80=0.2500\). The group has more cases than expected, suggesting possible underprediction in that region, though one group alone does not establish overall poor fit. (2) The conditional variance is \(p_i(1-p_i)\), which changes with the fitted probability, so constant variance is not a logistic-model assumption. (3) No. Convergence only indicates that the optimizer met its criterion; the systematic mismatch still warrants investigation of model form, covariates, and data. (4) It may indicate separation or sparse, unstable predictor patterns; inspect coding, genotype-by-outcome counts, covariate patterns, and optimizer messages before interpreting estimates. (5) Model calibration compares fitted probabilities with observed outcomes for a particular model, whereas genome-wide test-statistic calibration concerns the distribution of association statistics across variants.