Why the Candidate Set Matters
A regression comparison can only answer a useful question if the models being compared are genuinely alternatives for the same task. Suppose one formula predicts a patient’s follow-up blood pressure using baseline measurements, while another predicts a different outcome in a different subset of patients. A numerical score for each formula does not, by itself, make them comparable. They may differ in what is being predicted, who is included, or what the analysis is trying to accomplish.
A candidate model set is a specified collection of regression models considered as alternatives for a common purpose. Defining the set requires more than listing formulas. It also requires stating the outcome, the observational units and analysis sample, and the goal of the comparison. If these are not aligned, a difference in scores may reflect different data or tasks rather than a meaningful difference between the candidate regressions.
The models in a set do not have to contain the same predictors. In fact, they are alternatives because they differ in some specified way: one may include a predictor that another omits, or they may represent different scientifically motivated accounts. Earlier work on purposeful specification helps decide which alternatives deserve consideration. The purpose here is to establish conditions under which their subsequent comparison is interpretable.
Three Things to Fix Before Comparing
First, specify the outcome. Name the measured quantity, its units, the time point or forecast horizon, and any transformation or coding. “Blood pressure” is not sufficiently precise if one candidate uses systolic pressure at one month and another uses a binary indicator of whether pressure exceeds a threshold. Those models do not predict the same recorded response.
Second, specify the analysis sample: the observational units and outcome records on which each candidate is fitted or evaluated. For a direct comparison of regression fit, a common approach is to define the eligible units first and use the same rows for every candidate. This often means restricting to units with observed values for the outcome and all predictors that appear in any candidate. That restriction should be reported because it can reduce the sample and can change which population the results describe.
Third, state the comparison goal. An explanatory comparison concerns how candidate representations describe a specified relationship; a predictive comparison concerns predictions for the stated use and prediction horizon. The goal affects which predictors are available, what counts as a relevant evaluation sample, and what constitutes a useful comparison. A candidate containing information that would not be available at prediction time is not a fair forecasting alternative.
For regression candidates, the statistical basis of the scores also matters. Scores that depend on a likelihood, residual error, or other measure must be calculated consistently for the same observed responses. In the examples below, candidates use a Gaussian linear regression for the same quantitative outcome on the same rows. More generally, changing response scales, likelihood conventions, weighting, or treatment of repeated observations can make a direct numerical comparison invalid unless the scoring method explicitly accounts for those differences.
Why Different Samples Can Mislead
A simple least-squares quantity shows why the analysis rows must be aligned. For a model fitted to \(n\) responses \(y_i\), the residual sum of squares is
Each term measures a squared prediction error for one observed response. If two models use different response records, their residual sums add errors over different collections of units. One sum can be smaller simply because it includes fewer terms, or because its sample contains easier-to-predict cases. It then does not isolate the difference between the candidate formulas.
This is not a claim that every analysis must use one particular missing-data procedure. It is a requirement to make the comparison target explicit. If candidates are fitted on different rows, their results may still answer separate questions, but a difference between their raw scores should not be presented as a direct model comparison. Likewise, if a prediction study uses a held-out set, all candidates being compared should be evaluated on the same held-out units and the same outcome definition.
Worked Example: Finding a Common Analysis Sample
Hypothetical data. A clinic proposes two Gaussian linear regressions for systolic blood pressure at a six-month visit, measured in mmHg. Candidate A uses baseline age; Candidate B uses baseline age and a baseline activity score. The records below show whether the six-month outcome and each predictor are observed.
| Patient | Six-month outcome | Age | Activity score |
|---|---|---|---|
| A | Observed | Observed | Observed |
| B | Observed | Observed | Missing |
| C | Observed | Observed | Observed |
| D | Missing | Observed | Observed |
| E | Observed | Observed | Observed |
For a direct fit comparison on complete records for every candidate, define the common sample as patients with an observed outcome, age, and activity score. Patients A, C, and E meet all three conditions, so
Both candidates are therefore fitted to the six-month outcomes for the same three patients: A, C, and E. Patient B cannot be included in either candidate’s common-sample fit because the activity score is missing, even though it is available for Candidate A. Patient D is excluded because the response is missing. Fitting Candidate A to four patients and Candidate B to three would make their raw fit quantities depend on different observations.
This choice does not establish that the three complete-record patients represent every clinic patient. It defines the data used for this particular comparison; the consequences of missingness and the resulting target population must be considered when interpreting the analysis.
Make the Goal and Data Basis Explicit
The comparison goal places practical limits on which candidates belong in the set. For an explanatory analysis, predictors should be chosen to represent the stated scientific question, and the outcome and units should remain fixed across candidates. For a forecasting analysis, each predictor must also be available at the time the prediction is made. Evaluation should use the same forecast horizon, outcome definition, and test observations for every candidate. Otherwise, one model may appear better because it has access to later information or is evaluated on a different task.
Comparability also concerns the way the response is represented. Suppose one candidate models blood pressure in mmHg and another models its logarithm. Their residual errors are on different scales. A direct comparison of the two raw residual sums does not have the same interpretation: one is in squared mmHg and the other in squared log-mmHg. There are statistical methods that can support particular comparisons across response transformations, but the required adjustments must be part of the scoring procedure. Simply placing the numbers side by side is not enough.
In a standard common-scale Gaussian regression comparison, the response definition, units, rows, and likelihood calculation are held consistent. Candidates may then differ in their predictor terms while still addressing the same data and task. If some other feature must differ—for example, the response distribution or the sample—record that fact and use a comparison method valid for that difference rather than treating the scores as automatically interchangeable.
Worked Example: Detecting an Incomparable Outcome Scale
Hypothetical data. A researcher considers two regressions for the same patients and the same visits. Model A predicts systolic blood pressure in mmHg. Model B predicts the natural logarithm of systolic blood pressure. For illustration, suppose the sum of squared residuals is 900 for Model A and 0.18 for Model B.
These are calculated on different response scales:
The numerical inequality \(0.18<900\) does not show that Model B fits better. The first quantity sums squared errors in mmHg; the second sums squared errors in log-mmHg. The units and scale differ. To make a defensible comparison, the researcher must either express predictions and errors on a common, justified basis or use a scoring framework that correctly handles the transformation. The two raw sums alone do not rank the candidates.
The patients and visit are shared, but the outcome representation is not. This example illustrates why “same sample” is necessary for many comparisons but does not by itself guarantee comparability.
Build and Document a Defensible Set
A candidate set should be broad enough to represent the stated alternatives, but it is not an invitation to include every formula that can be generated. Each candidate needs a rationale, and the rules defining the set should be settled before inspecting which candidate obtains a more favorable score. This helps keep the comparison focused on the question rather than on opportunistic changes made after seeing results.
A practical specification records the following items:
- Outcome: exact variable, measurement units, timing, and any coding or transformation.
- Observational units and sample: eligibility rules, common analysis rows or common evaluation rows, and the handling of missing values.
- Goal: explanatory question or prediction task, including the relevant horizon and intended use.
- Candidate formulas: the terms in each model and the rationale for including or excluding them.
- Scoring basis: response scale, error or likelihood assumptions, estimation method, and any weights or other features needed to understand how the scores were obtained.
Stating these choices makes the comparison reproducible and exposes limits that might otherwise be hidden. For instance, if the common sample is much smaller than the original dataset, readers can see that the comparison concerns the retained units. If a forecasting candidate uses a predictor unavailable at the stated forecast time, that candidate can be rejected before its score is considered.
Worked Example: Checking a Forecasting Candidate Set
Hypothetical setting. A transit agency wants to predict next-day passenger counts for each route. It proposes two Gaussian regression candidates with passenger count in passengers per day as the outcome. Candidate A uses the previous day’s count and a calendar indicator. Candidate B uses those same predictors plus the next day’s observed weather.
For the stated goal—making predictions at the end of the current day—Candidate B is not eligible as written: the next day’s observed weather is not yet available. Its predicted value therefore cannot be a fair operational competitor at that forecast horizon. The candidates also need the same route-days in the training or evaluation sample, the same next-day outcome records, and the same rule for missing values.
Suppose, for a transparent toy check, that there are five eligible route-days and the next-day counts are observed on four of them. If Candidate A’s predictors are available on all four, and Candidate B’s weather predictor is available on only three, a direct comparison on each model’s individually available rows would use \(n_A=4\) and \(n_B=3\). The mismatch is
Instead, the comparison must either use the three route-days shared by both candidates—so \(n_{\text{common}}=4-1=3\)—or use a principled, pre-stated approach that gives both candidates comparable evaluation data and respects information availability. In this example, the unavailable future weather predictor must also be removed or replaced by a forecast available at prediction time. The comparison goal, timing, and sample together determine which candidates are legitimate.
Common Pitfalls
- Comparing scores from different sample sizes as if they were direct rankings. The scores may reflect different numbers or kinds of observations. Use common rows for a direct comparison or choose a method designed for the differing data.
- Assuming the same response name guarantees the same outcome. Measurement time, coding, and transformation matter. Specify all of them and ensure the scoring basis is appropriate.
- Allowing information from after the prediction time. A forecast candidate must use only information available when its prediction is meant to be made.
- Changing the set after seeing the results without disclosure. This makes the comparison less informative about the originally stated alternatives. Give candidates a rationale and document changes.
- Calling the top-scoring candidate “true.” A score compares specified alternatives under a defined goal and data basis. It does not prove a scientific explanation or establish a causal effect.
Check Your Understanding
Answer each question by identifying the comparison issue and stating a suitable correction.
- Two regressions predict the same outcome, but one is fitted to 180 observations and the other to 145. Why might their raw fit scores not support a fair direct comparison?
- For a forecast made at the end of today, one candidate includes tomorrow’s measured temperature. What is wrong with this candidate as specified?
- One model predicts an outcome in its original units and another predicts its logarithm. Why is comparing their raw residual sums potentially misleading?
- List four details that should be recorded when defining a candidate model set.
- A model receives the most favorable score among the candidates. What conclusion is supported, and what conclusion would go beyond the comparison?