Model Selection: AIC vs. BIC in Clinical Research

Introduction

Clinical research frequently involves choosing among several plausible statistical models.

For example, a statistician may need to decide whether a regression model should contain:

• Only treatment and baseline covariates
• Additional prognostic factors
• Nonlinear terms
• Interaction terms
• Random effects
• Additional distributional parameters

Several candidate models may all be statistically reasonable. The challenge is determining which model provides an appropriate balance between goodness of fit and model complexity.

Two of the most widely used information criteria are:

• AIC — Akaike Information Criterion
• BIC — Bayesian Information Criterion

Both criteria combine a measure of model fit with a penalty for the number of estimated parameters.

Why Model Selection Is Necessary

Suppose an investigator is modeling a continuous clinical outcome such as change from baseline in a biomarker.

A simple model might contain treatment alone:

A more elaborate model might additionally include age, sex, baseline value, disease severity, treatment-by-baseline interaction, and treatment-by-sex interaction.

The larger model will usually fit the observed data at least as well as the smaller model because it has more flexibility.

But better in-sample fit does not automatically mean a better statistical model.

What Is AIC?

AIC was developed by Hirotugu Akaike as a criterion for comparing statistical models based on information loss.

The standard AIC is:

where:

• \(L\) is the maximized likelihood of the model.
• \(\log L\) is the maximized log-likelihood.
• \(k\) is the number of estimated parameters.

The first component:

measures lack of fit.

The second component:

penalizes the model for having more parameters.

Therefore:

What Is BIC?

The Bayesian Information Criterion, commonly called BIC, is:

where:

• \(L\) is the maximized likelihood.
• \(k\) is the number of estimated parameters.
• \(n\) is the sample size.

The key difference from AIC is the penalty:

versus:

Because \(\log n\) increases with sample size, BIC generally imposes a stronger complexity penalty than AIC when the sample size is sufficiently large.

AIC vs. BIC at a Glance

The Most Important Similarity

Although AIC and BIC are often presented as competing methods, they share a fundamental structure.

Both reward better likelihood fit and penalize unnecessary parameters.

The main difference is how aggressively complexity is penalized.

Why the Penalties Differ

AIC penalizes each additional parameter by:

BIC penalizes each additional parameter by:

The two penalties are equal when:

which gives:

Therefore, for practical clinical sample sizes greater than about 8, BIC imposes a larger per-parameter penalty than AIC.

This explains why disagreements between AIC and BIC often become more pronounced as sample size increases.

A Simple Numerical Example

Suppose two candidate models are fitted to the same clinical dataset.

Model B fits the data better because its \(-2\log L\) value is smaller.

Suppose \(n=100\).

AIC for Model A

AIC for Model B

AIC therefore considers the two models tied in this example.

BIC for Model A

BIC for Model B

BIC therefore strongly favors Model A.

How to Interpret a Difference in AIC

The absolute value of AIC is usually not meaningful by itself.

Instead, compare the AIC values among candidate models fitted to the same data.

Define:

where \(\operatorname{AIC}_{\min}\) is the smallest AIC among the candidate models.

A common descriptive interpretation is:

These are useful rules of thumb rather than universal decision thresholds.

How to Interpret a Difference in BIC

BIC can similarly be compared using:

Smaller values are preferred.

Because BIC has a different theoretical basis and penalty structure, AIC and BIC differences should not be interpreted as though they were identical quantities.

AIC Does Not Mean "Absolute Model Quality"

Suppose three models have:

Model 3 has the smallest AIC.

This means only that Model 3 is preferred according to AIC among the candidate models considered.

It does not prove that Model 3 is scientifically correct.

If all three models omit an important confounder or use an inappropriate distribution, AIC cannot rescue the analysis.

Model Selection Is Not a Substitute for Clinical Knowledge

Clinical statistical modeling should begin with the scientific question.

For example, if age is a clinically important prognostic factor, it may be appropriate to include age because of the study objective even if adding age does not produce the smallest AIC.

Similarly, treatment should generally remain in a model when estimating a treatment effect even if an automated model-selection procedure suggests dropping it.

AIC in Linear Regression

AIC can be used to compare linear regression models fitted to the same outcome and dataset.

Consider:

versus:

The second model has more parameters.

If its improvement in likelihood is sufficiently large, AIC or BIC may favor it.

The criterion therefore provides a quantitative way to evaluate whether the improved fit compensates for the added complexity.

AIC in Logistic Regression

Suppose the endpoint is binary:

A logistic regression model may be written as:

AIC can compare models containing different predictor sets.

model1 <- glm(
  response ~ treatment + age,
  family = binomial,
  data = dat
)

model2 <- glm(
  response ~ treatment + age + baseline + severity,
  family = binomial,
  data = dat
)

AIC(model1, model2)
BIC(model1, model2)

The model with the smaller AIC is preferred according to AIC. The model with the smaller BIC is preferred according to BIC.

AIC in Poisson Regression

Count endpoints are also commonly modeled using likelihood-based methods.

For a Poisson regression:

with:

AIC and BIC can compare competing Poisson regression models.

For example:

m1 <- glm(
  event_count ~ treatment + age,
  family = poisson,
  data = dat
)

m2 <- glm(
  event_count ~ treatment + age + baseline + severity,
  family = poisson,
  data = dat
)

AIC(m1, m2)
BIC(m1, m2)

However, if the count outcome is substantially overdispersed, the model itself may be inappropriate.

AIC and Negative Binomial Models

Clinical count data often exhibit overdispersion.

A negative binomial model may therefore be more appropriate than a Poisson model.

AIC can help compare competing negative binomial specifications, such as different covariate structures.

But comparing models with different likelihood families requires additional care.

The models should represent genuinely comparable statistical questions and their likelihoods should be defined consistently.

AIC in Survival Analysis

AIC is frequently used when comparing candidate survival models.

For example, consider two Cox proportional hazards models:

One model might contain treatment and baseline disease status, while another also contains age and laboratory measurements.

library(survival)

cox1 <- coxph(
  Surv(time, status) ~ treatment + stage,
  data = dat
)

cox2 <- coxph(
  Surv(time, status) ~ treatment + stage + age + biomarker,
  data = dat
)

AIC(cox1, cox2)

A smaller AIC indicates better relative support under the AIC criterion.

However, proportional hazards assumptions, clinical interpretability, missing data, functional-form assumptions, and the study estimand remain important.

AIC and Parametric Survival Models

AIC and BIC are particularly useful when comparing fully likelihood-based parametric survival models.

For example, investigators may compare:

• Exponential models
• Weibull models
• Log-normal models
• Log-logistic models
• Generalized gamma models

These models can differ substantially in their assumptions about the hazard function.

AIC and Mixed-Effects Models

Repeated-measures clinical trials frequently require mixed-effects models.

For example:

The random effect \(b_i\) accounts for within-patient correlation.

Researchers may need to compare:

• Random-intercept models
• Random-intercept plus random-slope models
• Different covariance structures
• Different fixed-effect specifications

AIC is commonly used in this setting.

However, the details of likelihood estimation matter.

REML vs. ML in Mixed Models

This is one of the most important practical issues when using AIC for mixed models.

Restricted maximum likelihood, or REML, estimates variance components while accounting for the fixed effects.

When comparing models with different fixed effects, models should generally be compared using maximum likelihood rather than REML, because REML likelihoods depend on the fixed-effect specification.

For example, in R:

library(lme4)

m1 <- lmer(
  outcome ~ treatment + time + (1 | subject),
  data = dat,
  REML = FALSE
)

m2 <- lmer(
  outcome ~ treatment + time + baseline + (1 | subject),
  data = dat,
  REML = FALSE
)

AIC(m1, m2)
BIC(m1, m2)

Comparing Covariance Structures

Suppose the fixed effects are held constant while investigators compare different covariance structures.

Examples include:

• Compound symmetry
• Autoregressive AR(1)
• Unstructured covariance
• Toeplitz covariance

AIC and BIC can help determine whether the additional covariance parameters are supported by the data.

For example, an unstructured covariance matrix may fit better but introduce many additional parameters.

BIC's stronger penalty may therefore favor a simpler covariance structure while AIC may favor the more flexible structure.

AICc: A Correction for Small Samples

When sample size is small relative to the number of estimated parameters, ordinary AIC can have appreciable small-sample bias.

A corrected version called AICc is often used.

For a standard likelihood setting, AICc can be written as:

The additional correction becomes increasingly important when \(n\) is small relative to \(k\).

AIC vs. AICc vs. BIC

AIC vs. BIC: Different Goals

One reason AIC and BIC can disagree is that they are motivated by somewhat different theoretical objectives.

AIC is associated with minimizing expected information loss for predictive purposes.

BIC is associated with an asymptotic approximation to Bayesian model selection under particular assumptions and priors.

Consequently, the criteria should not be interpreted as two versions of the same hypothesis test.

A Worked Clinical Example

Suppose a randomized clinical trial evaluates whether treatment affects a continuous change-from-baseline endpoint.

The statistician considers three candidate models.

Suppose the analysis includes \(n=120\) patients.

Calculate AIC

For Model 1:

For Model 2:

For Model 3:

AIC therefore selects Model 2.

Calculate BIC

For \(n=120\):

Model 1:

Model 2:

Model 3:

BIC also selects Model 2.

When AIC and BIC Disagree

The most interesting situation occurs when AIC and BIC select different models.

Suppose:

AIC favors Model B.

BIC favors Model A.

This is not a mathematical error.

It means the improvement in likelihood from Model A to Model B is sufficient to overcome AIC's complexity penalty but not BIC's stronger penalty.

What Should You Do When They Disagree?

Do not simply choose one criterion because it gives the model you prefer.

Instead ask:

• What is the scientific purpose of the model?
• Is prediction or parsimony more important?
• Was the model structure prespecified?
• Are the additional variables clinically justified?
• Is the sample size small relative to model complexity?
• Are the candidate models scientifically plausible?
• Does the more complex model improve important clinical predictions?
• Are model assumptions satisfied?

If the analysis is primarily predictive, AIC may be particularly informative.

If the goal emphasizes a parsimonious explanatory model, BIC may receive greater consideration.

AIC and Hypothesis Tests Are Not the Same

AIC should not be interpreted as a replacement for likelihood-ratio tests, Wald tests, score tests, or prespecified hypothesis tests.

For example, suppose Model 1 is nested inside Model 2.

A likelihood-ratio test evaluates:

while AIC evaluates the relative trade-off between fit and complexity.

These can lead to different conclusions.

Nested Models and Likelihood-Ratio Tests

Suppose Model 1 is nested within Model 2.

The likelihood-ratio statistic is:

For regular models, this can be compared with an appropriate chi-square reference distribution.

AIC instead compares:

Thus, adding parameters must produce a sufficiently large likelihood improvement to compensate for the complexity penalty.

A Useful AIC Rule for One Additional Parameter

Suppose Model 2 adds exactly one parameter to Model 1.

Then:

Adding the parameter improves AIC when the improvement in \(-2\log L\) is greater than 2.

Equivalently, the likelihood-ratio statistic must exceed approximately 2 for the more complex model to have a smaller AIC.

This is less stringent than the conventional 5% chi-square threshold of approximately 3.84 for one degree of freedom.

A Similar BIC Rule

For one additional parameter, BIC favors the larger model only if the likelihood improvement exceeds approximately:

For \(n=100\):

Therefore, BIC requires a larger improvement in fit than AIC.

Variable Selection Using AIC

AIC is sometimes used in automated procedures such as forward selection, backward elimination, or stepwise selection.

For example:

full_model <- lm(
  outcome ~ treatment + age + sex + baseline +
    severity + biomarker + region,
  data = dat
)

step(
  full_model,
  direction = "both",
  trace = 0
)

This searches for a model with a relatively low AIC.

However, automated stepwise selection can produce unstable models and biased coefficient estimates, particularly when many candidate variables are considered.

Prespecified Covariates in Clinical Trials

In randomized clinical trials, model selection requires particular care.

Suppose the statistical analysis plan prespecifies adjustment for:

• Treatment
• Baseline disease severity
• Region

An analyst should not automatically remove region simply because another model has a slightly smaller AIC.

The analysis should remain consistent with the prespecified estimand and statistical analysis plan unless a justified change is documented.

Prognostic Covariates vs. Treatment Variables

AIC and BIC can be particularly useful for selecting among prognostic covariates.

Treatment, however, often has a special role.

If the scientific objective is to estimate a treatment effect, treatment should generally remain in the model regardless of whether its inclusion produces the smallest information criterion.

AIC and Interactions

Suppose investigators are considering a treatment-by-sex interaction:

The interaction adds a parameter.

AIC can help determine whether the improvement in likelihood justifies this additional complexity.

But a statistically or clinically important interaction should not necessarily be discarded merely because another model has a slightly smaller AIC.

Interaction terms should be considered in the context of:

• The prespecified subgroup question
• Clinical plausibility
• Statistical power
• Multiplicity
• Interpretability

AIC and Nonlinear Terms

Continuous clinical variables often have nonlinear relationships with the outcome.

For example, a linear age effect:

might be compared with a quadratic model:

AIC can help assess whether the additional flexibility is supported.

More flexible alternatives such as restricted cubic splines can also be compared using appropriate likelihood-based criteria.

AIC and Overfitting

Overfitting occurs when a model captures random features of the development dataset that do not generalize to new observations.

AIC reduces the incentive to add unnecessary parameters by imposing a complexity penalty.

BIC imposes an even stronger penalty as sample size increases.

Neither criterion, however, completely eliminates the risk of overfitting.

When prediction is the objective, additional approaches may be appropriate, including:

• Cross-validation
• Bootstrap validation
• External validation
• Calibration assessment
• Discrimination assessment

AIC vs. Cross-Validation

AIC and cross-validation both relate to predictive performance, but they approach the problem differently.

AIC uses the fitted likelihood and an analytical complexity correction.

Cross-validation estimates predictive performance by repeatedly evaluating the model on data not used for fitting.

What If the Goal Is Prediction?

If the primary objective is accurate prediction for future patients, AIC may often be more aligned with the objective than BIC.

But cross-validation or an external validation dataset can provide more direct evidence of predictive performance.

For clinical prediction models, investigators should also evaluate:

• Calibration
• Discrimination
• Clinical utility
• Performance across relevant subgroups
• Temporal or geographic transportability

What If the Goal Is Explanation?

When the goal is explanatory modeling, clinical knowledge may be more important than simply minimizing AIC or BIC.

A variable may be retained because it represents:

• A clinically important confounder
• A prespecified prognostic factor
• A design variable
• A component of the estimand
• A clinically important effect modifier

Information criteria can still provide useful secondary evidence about model complexity.

Do Not Compare AIC Across Different Datasets

AIC values are generally meaningful for comparing models fitted to the same observations and outcome.

Suppose Model A is fitted to 500 patients and Model B to 450 patients because of different missing-data handling.

Comparing their raw AIC values can be misleading.

Do Not Compare Incompatible Likelihoods Casually

AIC is based on the maximized likelihood.

Therefore, the underlying likelihoods need to be comparable for the model comparison to be meaningful.

This becomes particularly important with:

• Different response transformations
• Different observation sets
• Different likelihood definitions
• Quasi-likelihood models
• Some penalized models
• Mixed-model estimation methods

The software may return an AIC value even when a naive comparison would not answer the intended scientific question.

Quasi-Likelihood Models Require Caution

Some models use quasi-likelihood rather than a fully specified probability likelihood.

Because AIC is fundamentally likelihood-based, standard AIC calculations may not have their usual interpretation in such settings.

For example, analysts should be cautious when applying ordinary AIC logic to quasi-Poisson models.

AIC in R: Basic Workflow

For likelihood-based models in R, AIC is often straightforward.

AIC(model1, model2, model3)

The resulting table typically reports:

• Degrees of freedom
• AIC

For BIC:

BIC(model1, model2, model3)

The model with the smallest criterion is preferred according to that criterion.

Creating a Comparison Table

comparison <- data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  AIC = c(AIC(m1), AIC(m2), AIC(m3)),
  BIC = c(BIC(m1), BIC(m2), BIC(m3))
)

comparison

It is often useful to calculate differences from the best model.

comparison$Delta_AIC <-
  comparison$AIC - min(comparison$AIC)

comparison$Delta_BIC <-
  comparison$BIC - min(comparison$BIC)

comparison

AIC Weights

AIC differences can also be used to calculate approximate Akaike weights.

For model \(i\):

where \(R\) is the number of candidate models.

The weights can be interpreted as a measure of relative support among the candidate models under the AIC framework.

delta <- comparison$Delta_AIC

comparison$AkaikeWeight <-
  exp(-delta / 2) /
  sum(exp(-delta / 2))

comparison

Model Averaging

When several models have substantial AIC support, model averaging can sometimes be preferable to pretending that one model is unquestionably correct.

For example, several plausible models may have similar AIC values.

Rather than selecting one arbitrarily, model-averaging approaches can account for model-selection uncertainty.

This can be especially relevant in predictive modeling and some observational research settings.

AIC and Missing Data

Missing data can complicate model comparisons.

Suppose Model 1 uses treatment and baseline severity, while Model 2 adds a biomarker that is missing for 20% of patients.

If ordinary complete-case analysis is used, the two models may be fitted to different patient populations.

The AIC comparison can then become difficult to interpret.

AIC and Multiple Imputation

Multiple imputation introduces additional complexity because a model is fitted across multiple imputed datasets.

Standard AIC values from individual imputations should not simply be treated as though there were one ordinary complete dataset.

Model selection after multiple imputation should follow a method appropriate to the imputation and modeling framework.

The key point is that missing-data methodology and model-selection methodology must be considered together.

AIC and External Validation

A model with the lowest AIC in the development dataset can still perform poorly in a new population.

This is particularly important in:

• Clinical prediction models
• Machine-learning applications
• High-dimensional biomarker studies
• Rare disease research
• Small clinical datasets

External validation remains one of the strongest ways to assess whether a clinical prediction model generalizes.

Common Mistakes

1. Choosing the model with the highest AIC. Lower AIC is preferred.
2. Choosing the model with the highest BIC. Lower BIC is preferred.
3. Interpreting AIC as a p-value. AIC is not a hypothesis test.
4. Assuming the lowest AIC model is "true." AIC ranks candidate models; it does not establish truth.
5. Ignoring clinical rationale. Clinically essential variables should not automatically be removed because of a modest AIC difference.
6. Comparing models fitted to different patients. Differences in sample composition can invalidate simple comparisons.
7. Ignoring model assumptions. A lower AIC does not rescue a fundamentally inappropriate model.
8. Using REML to compare models with different fixed effects. For likelihood comparisons of different fixed-effects structures in mixed models, ML is generally appropriate.
9. Assuming AIC and BIC must agree. They can legitimately favor different models.
10. Using ordinary AIC automatically in small samples. AICc may be more appropriate when the sample size is small relative to the number of parameters.
11. Using automated stepwise selection without considering bias and instability. Model-selection uncertainty should be acknowledged.
12. Reporting only the selected model. The candidate models and selection rationale can be important for transparency.

A Practical Model-Selection Workflow

How AIC and BIC Fit Into Clinical Trial Practice

In confirmatory clinical trials, the primary analysis model is often prespecified.

This is fundamentally different from exploratory observational modeling, where model selection may play a much larger role.

For a confirmatory trial, AIC or BIC may be useful for:

• Exploring covariance structures
• Assessing sensitivity analyses
• Comparing plausible secondary models
• Evaluating functional forms
• Supporting exploratory analyses

They should not normally be used as a post hoc mechanism for changing the primary analysis simply because another model happens to produce a more favorable treatment estimate.

Model Selection in Observational Clinical Research

In observational studies, the role of model selection can be different because investigators may have many potential confounders.

However, automated selection based solely on AIC or BIC can be problematic.

A variable can be an important confounder even if it contributes relatively little to prediction.

Confounding control should therefore be guided by:

• Subject-matter knowledge
• Causal structure
• Study design
• Predefined adjustment sets
• Directed acyclic graph reasoning when appropriate

AIC vs. BIC in a Clinical Decision Framework

What Does a Difference of 1 Point Mean?

Suppose:

The difference is:

This is a very small difference.

It would be inappropriate to describe Model A as overwhelmingly superior to Model B.

When differences are small, other considerations such as interpretability, clinical plausibility, parsimony, and validation may reasonably determine the final choice.

What Does a Difference of 20 Points Mean?

Now suppose:

The difference is:

This represents a much stronger relative preference for Model A under AIC.

The magnitude of the difference matters more than the absolute AIC values.

Why More Parameters Always Improve or Preserve Likelihood Fit

Suppose Model 2 contains all parameters from Model 1 plus additional parameters.

Because Model 1 is a special case of Model 2, Model 2 can reproduce the fit of Model 1 by setting the additional parameters appropriately.

Therefore:

or equivalently:

This is precisely why a complexity penalty is necessary.

Without a penalty, increasingly complex models would always appear preferable.

The Logic of Information Criteria

The general logic can be summarized as:

Reporting AIC and BIC in a Statistical Report

A transparent model-selection section should describe:

• The candidate models considered
• The clinical rationale for each model
• The likelihood used for fitting
• The analysis population
• The information criterion used
• The criterion values
• The differences from the best model
• Any disagreement between AIC and BIC
• Model diagnostics
• The final rationale for model selection

For example:

When BIC May Be Attractive

BIC may be particularly attractive when:

• A parsimonious model is desired.
• The sample size is relatively large.
• The goal emphasizes model identification rather than maximum predictive flexibility.
• There is concern about including weakly supported predictors.
• Several increasingly complex nested models are being compared.

When AIC May Be Attractive

AIC may be particularly attractive when:

• Predictive performance is important.
• There is a legitimate reason to retain model flexibility.
• The candidate models differ modestly in complexity.
• The analysis is exploratory.
• Model averaging is being considered.

When Neither Should Decide the Model Alone

There are many situations in which neither criterion should be the sole basis for the final model.

Examples include:

• Confirmatory primary analyses
• Causal inference
• Prespecified treatment-effect models
• Models containing mandatory confounders
• Models with major assumption violations
• Very small datasets
• Complex missing-data mechanisms
• Highly adaptive exploratory analyses

Quick Comparison

The Most Important Concept

The most important concept is simple: AIC and BIC reward model fit but penalize unnecessary complexity.

The formulas are:

Both favor smaller values.

Because BIC's penalty increases with sample size, BIC generally favors more parsimonious models than AIC.

But the choice between models should never be reduced to a single number.

The statistician should consider:

• The scientific question
• The clinical meaning of the variables
• The estimand
• The prespecified analysis plan
• Model assumptions
• Goodness of fit
• Predictive performance
• Model complexity
• Sample size
• Missing data
• External validity

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.
Burnham, K.P. & Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer.
Burnham, K.P. & Anderson, D.R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods & Research, 33(2), 261–304.
Hurvich, C.M. & Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76(2), 297–307.
Vrieze, S.I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243.