Why Sample Size Changes the BIC Penalty
When a model gains parameters, it can usually describe the observed data more closely. BIC accounts for that added complexity with a penalty that depends on both the number of estimated parameters and the sample size. A larger sample size changes the penalty attached to each parameter, even when the model dimension is unchanged.
The key calculation is logarithmic: for a sample of size \(n\), the penalty per parameter is \(\ln n\), where \(\ln\) is the natural logarithm. This tutorial focuses on how that quantity changes with \(n\), and how to calculate the change for a fixed increase in model dimension. The penalty is only one component of BIC; a change in the penalty alone does not determine whether the full BIC score rises or falls.
For example, if two candidate regression models use the same observations and share the same intercept and error-variance parameters, adding two predictors increases \(k\) by two. The additional complexity penalty is therefore \(2\ln n\). Parameters common to both models remain part of the BIC calculation, but they cancel when calculating the difference in penalty.
The Sample-Size Effect
For \(n>1\), \(\ln n\) is positive and increases as \(n\) increases. However, it increases slowly: multiplying the sample size by a fixed factor adds a fixed amount to the per-parameter penalty, regardless of the starting sample size. If the sample size changes from \(n_1\) to \(n_2\), then
This identity gives a convenient way to compare penalties. The change per parameter depends on the ratio \(n_2/n_1\), not on the absolute difference \(n_2-n_1\). Thus doubling a sample adds \(\ln 2\) to the penalty per parameter whether the sample grows from 50 to 100 or from 500 to 1,000.
The second formula follows by subtracting the two penalty terms at the same sample size: \((k+\Delta k)\ln n-k\ln n=\Delta k\ln n\). Combining the two changes is also possible. If sample size changes from \(n_1\) to \(n_2\) and model dimension changes from \(k_1\) to \(k_2\), subtract the original penalty from the new one:
This combined expression should not be confused with a comparison of full BIC scores. The maximized log-likelihood may also change, and it is not included in the penalty calculation.
Worked Penalty Calculations
Worked Example: Same Model Dimension, Different Sample Sizes
A hypothetical insurer fits the same model structure to two samples, one with \(n_1=100\) policy records and one with \(n_2=400\). Each fitted model has \(k=5\) estimated parameters under the stated counting convention. Calculate the complexity penalty for each fit and the change in the penalty. Avoid treating a fourfold increase in sample size as a fourfold increase in penalty.
Method. Calculate \(k\ln n\) at each sample size, then subtract the first penalty from the second.
For \(n_1=100\), \(\ln 100=4.605170\). Therefore,
For \(n_2=400\), \(\ln 400=5.991465\). Therefore,
Subtracting gives \(29.957325-23.025850=6.931475\). The ratio of sample sizes is \(400/100=4\), so the formula for a fixed \(k\) gives the same result: \(5\ln 4=5(1.386294)=6.931470\), with the tiny difference due to rounding the logarithms above.
Check. The larger sample has the larger penalty, but the penalty is not four times as large. Rounded to four decimal places, the penalties are 23.0259 and 29.9573, and the increase is 6.9315. The common mistake avoided is multiplying the original penalty by the sample-size ratio instead of using a logarithm.
Worked Example: Adding Parameters at Two Sample Sizes
A hypothetical health-risk model is expanded by three parameters. Compare the additional BIC penalty when the sample size is \(256\) with the additional penalty when it is \(1{,}024\). The model is evaluated using the same parameter-counting convention in both cases.
Method. For a fixed sample size, an increase of \(\Delta k=3\) parameters adds \(3\ln n\) to the penalty.
At \(n=256\), \(\ln 256=5.545177\), so the added penalty is
At \(n=1{,}024\), \(\ln 1{,}024=6.931472\), so the added penalty is
The difference between these added penalties is \(20.794416-16.635531=4.158885\). Since \(1{,}024/256=4\), the difference can also be checked directly:
Check. Rounded to four decimal places, the extra penalties are 16.6355 and 20.7944, a difference of 4.1589. The common mistake avoided is comparing \(\ln n\) alone when the question asks for the penalty for three parameters; the per-parameter change must be multiplied by three.
Worked Example: Inferring the Sample-Size Ratio From a Penalty Change
A hypothetical claim-frequency model keeps the same six estimated parameters while its sample size increases. The reported increase in its complexity penalty is 8.3178. If the original sample had 80 observations, estimate the new sample size. Treat the reported increase as rounded to four decimal places.
Method. For a fixed number \(k\) of parameters, the increase in penalty is \(k\ln(n_2/n_1)\). Divide by \(k\) to find the change in the log sample size.
Here \(k=6\), so
Exponentiating gives \(n_2/80\approx e^{1.3863}\approx4\). Thus \(n_2\approx80(4)=320\).
Check. Using the unrounded ratio \(4\), the penalty increase is \(6\ln 4=8.317766\), which rounds to 8.3178 as reported. The inferred sample size is 320. The common mistake avoided is dividing the penalty increase by the starting sample size; the increase is first divided by the parameter count, then exponentiated to recover a sample-size ratio.
How to Interpret the Change
The logarithmic form has several practical consequences. First, a fixed multiplicative expansion of the sample produces a fixed per-parameter penalty increase. Second, a fixed increase in dimension has a larger added penalty at a larger sample size, because \(\ln n\) is larger. Third, the penalty change scales linearly with the number of parameters being added or removed.
| Change being considered | Penalty calculation | What determines the change? |
|---|---|---|
| Same model dimension, sample changes from \(n_1\) to \(n_2\) | \(k\ln(n_2/n_1)\) | The sample-size ratio and the fixed parameter count |
| Dimension increases by \(\Delta k\) at sample size \(n\) | \(\Delta k\ln n\) | The number of added parameters and the sample size |
| Sample and dimension both change | \(k_2\ln n_2-k_1\ln n_1\) | Both penalty terms, calculated with their own \(k\) and \(n\) |
These calculations describe penalty components, not necessarily changes in model preference. A full BIC comparison also includes the maximized log-likelihood for each fit. Moreover, comparisons based on different samples require care: the likelihood terms and the information available to the fitted models may differ as well. Do not conclude that one model has a lower full BIC merely because its penalty is smaller.
Common Mistakes and Checks
- Using the sample-size ratio as the penalty multiplier. The penalty contains \(\ln n\), not \(n\). For a fixed model dimension, moving from \(n_1\) to \(n_2\) changes the penalty by \(k\ln(n_2/n_1)\).
- Forgetting the number of parameters being added. The increase for \(\Delta k\) parameters is \(\Delta k\ln n\), not merely \(\ln n\).
- Taking a logarithm of a difference in sample sizes. The identity uses the ratio \(n_2/n_1\). In general, \(\ln n_2-\ln n_1\) is not \(\ln(n_2-n_1)\).
- Mixing parameter-counting conventions. Count parameters consistently across candidate fits. Shared parameters may cancel in a penalty difference, but do not silently include a parameter in one model and omit it in the other.
- Calling a penalty change a full BIC change. The full criterion also contains the fitted log-likelihood term. Calculate that term as well before making a claim about the direction of the full BIC difference.
Check Your Understanding
For each problem, calculate only the requested complexity penalty or penalty change unless the question says otherwise.
- A hypothetical rating model has \(k=4\) estimated parameters and uses \(n=25\) observations. What is its complexity penalty? (Answer: \(4\ln 25=12.8755\), to four decimal places.)
- A model gains two parameters as its sample grows from \(n=200\) to \(n=800\). By how much does the added-parameter penalty change between these sample sizes? (Answer: \(2\ln(800/200)=2\ln 4=2.7726\), to four decimal places.)
- A fixed model with eight parameters is evaluated at \(n=125\) and \(n=500\). What is the increase in its complexity penalty? (Answer: \(8\ln(500/125)=8\ln 4=11.0904\), to four decimal places.)
- A model with three fixed parameters has a penalty increase of 4.1589 when the sample size changes. Approximately what multiplicative change occurred in sample size? (Answer: \(e^{4.1589/3}\approx4\), so the sample size increased by a factor of 4.)
- A sample grows from \(n=1{,}000\) to \(n=10{,}000\). What is the increase in penalty for a fixed model, and what is it for an increase of three parameters? (Answer: Per parameter, \(\ln 10=2.3026\); for three parameters, \(3\ln 10=6.9078\), to four decimal places.)