When the Outcome Starts Choosing the Model
A model-building plan can look systematic while allowing the observed outcome to make every important decision. For example, an analyst may try many possible predictors, keep those with small p-values, remove terms with weak associations, and repeat until the remaining results look convincing. The problem is not simply that the plan uses data. The problem is that the same outcome data both determine the specification and provide the evidence later presented as if that specification had been settled independently.
This tutorial focuses on recognizing that pattern and replacing it with a question-led candidate specification. A specification states the outcome and the terms that may appear in its model. The outcome should be held fixed across the candidate models being considered; the terms should be chosen because they represent the question, a plausible account of the outcome, or the intended use of the model—not because their observed associations happen to look favorable.
What Makes a Specification Question-Led?
Start by stating the purpose of the analysis and defining the outcome. Then identify candidate terms by asking what information or scientific explanation the model is meant to represent. For an explanatory analysis, this might mean identifying characteristics relevant to the question and specifying in advance which terms represent them. For a forecasting analysis, it includes asking what information will actually be available at the time a forecast is made. These are different questions, and they can lead to different candidate specifications even when they use the same outcome.
A term can be included because it is needed for the question even if its individual association with the outcome is weak in the observed sample. Conversely, a strong observed association alone does not establish that a term belongs in a model. It could reflect chance, an indirect relationship, a feature of this particular sample, or information that is unavailable or inappropriate for the intended use. The specification is not a prize awarded to the terms with the most attractive results.
For a numerical outcome \(Y_i\), a simple candidate linear model can be written as
Here \(i\) indexes observations, the \(x\) terms are specified predictors, and \(\epsilon_i\) represents variation not described by the model. This equation describes a candidate model, not a rule for choosing its terms. In a conventional linear regression analysis, the error assumptions and sampling design also matter for estimation and inference. Even if those assumptions are appropriate for a fixed specification, however, they do not make ordinary inferential results account for a specification selected by repeatedly inspecting the same outcome data.
A useful distinction is between a candidate set and a selection rule. A candidate set states which models will be considered and why. A rule such as “keep every term with \(p<0.05\), refit, and repeat” lets the observed outcome decide the set. Writing down the candidate set before looking at outcome associations helps prevent that rule from quietly replacing the research question.
Why Repeated Outcome Searches Change the Evidence
Suppose a plan tests many unrelated candidate terms and highlights any that show a small p-value. Under a null hypothesis, a p-value is calculated for a specified test under its assumptions; it is not a guarantee that the entire search procedure has a small false-positive rate. When the same data are searched repeatedly, there are more opportunities for at least one apparently notable result to occur by chance.
The following calculation illustrates the issue under a simplifying assumption: suppose eight tests are independent and each null hypothesis is true. If each test uses significance level \(\alpha=0.05\), the probability that no test produces a false positive is \(0.95^8\). Therefore, the probability of at least one false positive is
Thus, under these simplifying assumptions, the chance of at least one false positive across the eight tests is about 33.7%, not 5%. In actual model searches, tests may not be independent, and the number of choices may be less obvious than eight separate tests. The calculation is not a correction to apply automatically; it shows why a single-test interpretation does not describe a repeated search.
Selection also affects how to interpret estimates and intervals. If a term is kept specifically because its estimated association looked unusually large, the selected estimate tends to be unusually favorable compared with estimates from a procedure that did not select on the outcome. A conventional p-value or confidence interval calculated after that search generally does not account for the selection step. The data may still be useful for exploration, but the model and results should not be presented as though the term choices were fixed independently.
Worked Example: Screening Eight Candidate Predictors
A hypothetical analyst has a continuous outcome, weekly study hours, and considers eight possible predictors. The plan is to test each predictor separately, retain those with \(p<0.05\), and describe the retained terms as supported by the data. Suppose, for illustration, that all eight predictor associations are actually null and that the tests are independent.
The plan makes eight opportunities to obtain a false positive. Using the calculation above, the probability of at least one such result is \(1-(1-0.05)^8=1-0.95^8\approx 0.3366\). A rule that calls any one retained term “significant at the 5% level” therefore does not make the whole search a 5% procedure under these assumptions.
Diagnosis. The outcome is deciding which predictors survive. If the same data are then used to report ordinary tests for the retained terms as confirmatory evidence, the analysis ignores the search that produced those terms.
Question-led rewrite. Suppose the stated question is whether weekly study hours vary with course load and paid work commitments. Before examining outcome associations, specify course load and paid work as the core terms. If a further scientific question concerns whether the association with paid work differs by course load, identify that interaction as a separate candidate term and state why it is relevant. The specification now follows the question; it does not change merely because one observed p-value is small or large.
Diagnose the Plan Before Looking at Its Results
A useful diagnosis focuses on the decision process, not on whether the final model happens to look plausible. Ask what would have happened if the observed associations had been different. If a term would have been included only when it had a small p-value, or excluded only when its coefficient looked inconvenient, the outcome has controlled the specification.
Watch for plans that make several decisions in sequence: add a term if its association looks strong, remove it if its coefficient becomes weak after refitting, try a different form if the sign is unexpected, and stop when the results tell a coherent story. Each step can be described as reasonable in isolation. Together, they allow the outcome to guide the search and then appear as evidence for the result of that search.
Worked Example: Repeatedly Pruning an Explanatory Wage Model
In a hypothetical analysis of wages, the outcome is hourly wage in dollars. The analyst begins with a question about how education and work history relate to hourly wages, but plans to keep only terms with \(p<0.05\). A preliminary model includes education, years of experience, years with the current employer, union membership, and occupation group. Suppose its reported p-values are 0.004, 0.03, 0.21, 0.04, and less than 0.001, respectively.
Following the rule, the analyst removes years with the current employer because \(0.21>0.05\). After refitting, suppose the p-value for union membership is 0.08; the analyst then removes union membership too. This makes two outcome-dependent changes: the first follows the original model’s associations, and the second follows associations after a data-dependent refit.
Diagnosis. The plan has not merely identified a fixed answer to the question. It has defined inclusion by the observed p-values. The weak individual result for years with the current employer does not establish that the term is irrelevant to the stated question, and the later p-value for union membership does not erase the search that produced the reduced model.
Question-led rewrite. Keep hourly wage as the outcome and define a core candidate specification that includes education and experience because they directly represent the stated question about education and work history. Decide whether years with the current employer, union membership, and occupation group belong in an extended candidate specification by asking whether each represents a relevant feature of the question and is measured appropriately. State that rationale before using the observed associations to evaluate the model. Do not remove a required term solely because its individual p-value exceeds 0.05.
This rewrite does not require including every available variable. It requires making the inclusion decision for a reason tied to the analysis, rather than using the outcome’s apparent preferences as the reason.
Rewrite an Outcome-Driven Plan as a Candidate Specification
A practical rewrite makes the research question operational before any outcome-based screening begins. The steps below separate decisions that should be made from the purpose of the analysis from findings that may emerge after the data are examined.
State what is being explained or predicted, how it is measured, and what the model is intended to do. Keep the outcome definition constant across the candidate specifications being compared.
For every proposed predictor or model feature, state which part of the question it represents. Remove terms that have no defensible role rather than keeping them merely because they are available.
Identify terms required by the primary question, then define a limited set of extensions that represent clearly stated additional questions. Do not let observed outcome associations decide which candidates exist.
Distinguish planned analyses from exploratory searches. If outcome inspection motivates a new term or form, label that decision as exploratory and do not portray an ordinary test from the same search as independent confirmation.
Worked Example: Rewriting a Forecasting Search
A hypothetical clinic wants to forecast a patient’s next-month appointment count. An analyst proposes trying age, distance to the clinic, prior appointment count, and a same-month billing total. The analyst plans to add or remove each variable according to its association with the observed next-month count, then keep the model with the most attractive results.
Diagnosis. The plan lets observed outcome associations determine the terms. It also needs to check when each proposed predictor is available: a billing total measured after the forecast date may not be available when the forecast must be made. A strong observed association cannot make unavailable future information suitable for a forecast.
Question-led rewrite. Fix the outcome as next-month appointment count and state the forecast time. List only predictors available at that time. For example, prior appointment count may be included because the question asks whether the patient’s recorded appointment history helps forecast future visits. Age and distance may be considered if they are available and have a stated role in the forecasting question. Exclude a billing total that is not available by the forecast time. Define the resulting candidate set before inspecting associations with next-month counts.
The revised plan does not promise that any predictor will be strongly associated with the outcome. It specifies which information the intended forecast is allowed to use and why.
Exploration Is Not the Same as Confirmation
Exploratory work can be valuable. Looking at data may suggest an omitted question, a possible measurement issue, or a relationship worth studying. The mistake is not noticing something in the outcome data; it is hiding that discovery process and interpreting the resulting model as if every choice had been made before the data were seen.
When outcome inspection motivates a new specification, describe the change and its reason transparently. Treat results from the same data as exploratory evidence. Stronger confirmation can come from a new data set collected for that purpose, or from a genuinely prespecified analysis whose decisions are not made by searching the same outcomes. This distinction concerns how the evidence was generated; it does not guarantee that a future result will be replicated.
Do not confuse a fixed candidate set with a claim that its terms are certainly correct. A question-led specification is a disciplined starting point. It makes the model’s purpose and choices inspectable, while leaving room to report limitations and to develop new questions from exploratory findings.
Common Mistakes
- “The p-value is large, so the term does not matter.” A weak individual test is not by itself a reason to remove a term required by the question. State the role of the term and follow the prespecified plan.
- “I only tried a few alternatives.” Even a small search can make reported evidence depend on the search. Describe which alternatives were tried and whether the choices were made after inspecting the outcome.
- “The final model is reasonable, so the process does not matter.” A plausible final list of terms does not undo outcome-dependent selection. Explain how the terms were chosen.
- “Exploratory results are useless.” They can generate useful hypotheses. The correction is to label them as exploratory and avoid presenting same-data results as independent confirmation.
- “A question-led plan must include every relevant variable.” Candidate sets still need limits. Include terms for stated purposes, and consider data availability and intended use; do not add terms solely because they can be measured.
Check Your Understanding
For each question, distinguish the stated purpose of a model from decisions made after inspecting outcome associations.
- An analyst tests 12 candidate terms and keeps only those with small p-values. What makes this plan outcome-driven?
- Why does a 5% significance level for one test not necessarily describe the false-positive chance across a repeated search?
- A term required by the research question has a weak individual association in the sample. Does that alone justify removing it? Explain.
- What information should be checked before including a predictor in a forecasting specification?
- If a new term is proposed after examining outcome plots, how should results from the same data be described?