Setting Inclusion and Elimination Thresholds

Why the Thresholds Need a Clear Meaning

A stepwise covariate search uses statistical tests to decide whether a covariate should enter or remain in an NLME model. A p-value cutoff turns each test into an operational decision rule. That makes the cutoff consequential: a small change can alter which covariates are considered, the model ultimately retained, and the interpretation of the search.

The forward-inclusion and backward-elimination cutoffs need not be the same. They govern different decisions, and their directions must be stated explicitly. In this tutorial, a candidate is added when its likelihood-ratio p-value is at or below the inclusion cutoff. In a backward step, a term is removed when its p-value is above the elimination cutoff. Thus, a smaller elimination cutoff means that fewer terms meet the standard for retention and can lead to more removals. Calling a cutoff “stricter” without specifying which decision it governs is a common source of confusion.

For a likelihood-ratio comparison, the statistic is the difference in fit between a model containing a term and the corresponding nested model without it, expressed on the deviance scale. With maximized log-likelihoods \(\ell_{\mathrm{larger}}\) and \(\ell_{\mathrm{smaller}}\), the statistic is \(D=2(\ell_{\mathrm{larger}}-\ell_{\mathrm{smaller}})\). Its p-value is computed from the appropriate null reference distribution for the tested restrictions. The comparison must use the same observations and compatible model specifications; the larger model must contain the smaller one as a special case. Convergence and a suitable reference distribution are also required for the p-value to be meaningful.

A cutoff does not repair a poor comparison. For example, changing the residual-error model between the two fits, or comparing fits based on different records, can make the likelihood difference unsuitable for the intended test. Likewise, a numerical convergence warning or an implausible estimate should be investigated before using the reported p-value as a selection decision. The test addresses the specified covariate term under the fitted model; it does not establish that the covariate is causal or that the selected model is correct.

Choosing Distinct Cutoffs

A useful threshold plan starts by specifying the consequences of the two kinds of error. A false inclusion retains a covariate whose apparent contribution is largely noise, potentially increasing model complexity and exaggerating the evidence for an effect. A false exclusion drops a useful covariate, potentially leaving systematic structure unexplained. The relative costs depend on the modeling objective, the candidate set, data support, and intended use of the model.

One possible plan is \(\alpha_{\mathrm{in}}=0.05\) and \(\alpha_{\mathrm{elim}}=0.01\). Under this plan, the forward search can admit a candidate with moderate evidence, while the backward stage applies a more demanding retention rule. The numbers are examples, not universal standards. A team should choose and document cutoffs before reviewing a long sequence of results, justify them in relation to the objective, and report them alongside the final selection procedure.

Notice the direction of the backward rule. With \(\alpha_{\mathrm{elim}}=0.01\), a term with \(p=0.03\) is removed because \(0.03>0.01\). With \(\alpha_{\mathrm{elim}}=0.10\), that same term is retained because \(0.03\leq0.10\). The smaller elimination cutoff imposes a stronger standard for retention and, all else equal, tends to produce a smaller model. It is not a rule that removes only terms with especially large p-values; it removes terms above the stated cutoff.

How Cutoffs Affect False Inclusions and Model Size

For a single valid test when the null hypothesis is true, a rule that rejects at level \(\alpha\) has a Type I error probability no greater than \(\alpha\) under the test assumptions. For forward inclusion, lowering \(\alpha_{\mathrm{in}}\) therefore makes it harder for a null candidate to enter on that particular test. It also makes it harder for a genuinely useful candidate to enter, so fewer candidates will generally be added and the selected model will tend to be smaller.

Backward elimination has the complementary operational direction. A term is removed if \(p>\alpha_{\mathrm{elim}}\); lowering \(\alpha_{\mathrm{elim}}\) makes that inequality true for more p-values. This tends to remove more terms and reduce model size, while increasing the chance that a useful but imprecisely estimated term is removed. Raising \(\alpha_{\mathrm{elim}}\) tends to retain more terms, including some that may be noise. These are tendencies, not guarantees: correlated covariates, changes in conditioning on other terms, and the order of tests can alter the path and final model.

Nominal error rates for individual tests do not equal the error rate for an entire search. If \(m\) independent null tests are each performed at level \(\alpha\), the probability of at least one false positive is \(1-(1-\alpha)^m\). This formula illustrates why repeated opportunities to add terms matter; actual covariate tests are often dependent, and a stepwise search is adaptive, so this calculation is not an exact familywise error rate for a real search.

Justifying and Checking a Threshold Plan

A defensible plan connects each cutoff to the purpose of model building rather than selecting the values because they produce a preferred result. Before the search, state the candidate set, the inclusion and elimination rules, and how deleted candidates will be handled if the search allows reconsideration. If candidates can re-enter, distinct cutoffs can cause a term to pass forward inclusion but fail backward retention. A search needs an explicit stopping and re-entry rule to avoid repeated cycling between decisions.

After selection, check whether conclusions are fragile to reasonable alternative cutoffs or modest changes in the candidate set. Large changes in the selected model signal that evidence is borderline or that candidates compete for overlapping information. Review convergence, likelihood comparisons, covariate support, and whether the retained effects make sense in the intended model. A p-value cutoff cannot compensate for poor support, collinearity, confounding, or an ill-specified model.

A common mistake is to say, “The elimination cutoff is stricter because it is smaller, so it removes fewer covariates.” That reverses the rule. Backward elimination removes a term when \(p>\alpha_{\mathrm{elim}}\); for the same p-value, lowering the cutoff makes removal more likely. The correction is to name the decision explicitly: a smaller \(\alpha_{\mathrm{elim}}\) is a stricter retention standard and tends to yield a smaller model.

Threshold-based stepwise selection is not the only approach. Information criteria compare fit and complexity through a score rather than applying the same p-value decision rule; a prespecified model emphasizes scientific commitments made before examining the results. These approaches answer related but not identical questions, and none makes the result immune to data limitations or model assumptions. Whichever approach is used, report the selection rule and treat a selected model as a modeling decision supported by evidence, not as a definitive list of true effects.