Why Data Must Match the Intended Use
A dataset can be large, carefully collected, and still be poorly suited to a particular modeling question. For example, data from adults with stable disease may offer little direct support for a prediction about children starting treatment during an acute episode. The central question is not whether data are generally “good,” but whether their source, measurements, timing, treatment experience, and completeness align with the context of use.
The intended use specifies what the model is meant to do, for whom, and how its output will inform a decision. Matching data to that use is an evidence assessment: it identifies which parts of the dataset are directly informative, where interpretation relies on assumptions, and which intended applications the dataset cannot support on its own. It does not, by itself, establish that a model is valid or accurate.
A useful assessment starts with the question and decision, then works backward to the data required. The same dataset might describe a response in the patients observed but be inadequate for predicting a response in a different population or at an unobserved treatment level. This is why description, prediction, and extrapolation must be distinguished when judging what the evidence can support.
Five Dimensions to Examine
The following dimensions provide a practical first assessment. They are related, but one cannot generally compensate for a major mismatch in another: many observations do not repair an incompatible endpoint, and detailed measurements do not create evidence for an unobserved treatment range.
| Dimension | Questions to ask | Example limitation |
|---|---|---|
| Population | Who entered the dataset? How do their characteristics and care setting compare with the intended population? | Only patients with mild disease are represented, but use concerns severe disease. |
| Measurements | Are the needed outcomes and predictors recorded? Are definitions, units, and methods suitable? | The intended endpoint is symptom improvement, but only a laboratory marker was collected. |
| Timing | Do observation times capture the processes and decision time points relevant to the use? | Measurements are available only at baseline and late follow-up, while early response matters. |
| Treatment or exposure range | Do the data include relevant doses, concentrations, regimens, and combinations? | All observations are below the exposure range for the proposed regimen. |
| Missingness | Which measurements are absent, for whom, and at what times? Could absence relate to patient status or outcomes? | Patients who worsen are more likely to stop attending follow-up. |
Population comparison should be specific to the intended use, not based only on labels such as “adult” or “clinical.” Relevant characteristics may include disease severity, age, organ function, prior treatment, concomitant therapy, and care setting. A population mismatch does not automatically make a dataset useless; it does mean that claims beyond the represented patients need an adequate rationale.
For measurements, check more than whether a column has the desired name. Confirm the endpoint definition, units, assay or instrument, collection procedure, and timing. A variable recorded in the wrong units or under a materially different definition may not answer the intended question. Also ask whether the dataset contains the inputs the proposed model requires, rather than assuming that a model can recover information that was never collected.
A Simple Coverage Summary—and Its Limits
A coverage summary can make population representation easier to inspect. Suppose the intended population is divided into clinically meaningful groups, and the target proportion in each group is known or specified. Define \(w_j\) as the target proportion in group \(j\), and let \(I_j=1\) when the dataset contains observations that provide relevant support for that group and \(I_j=0\) otherwise. A basic representation summary is:
If the target groups partition the population, the weights sum to one. The summary \(C\) is the target proportion belonging to groups with at least some relevant representation. It is not a measure of sample size, precision, or model accuracy. The designation “relevant support” needs a clear definition; one observation in a group may not be enough for a particular analysis.
This summary follows directly from adding the target-population weights of represented groups. If three groups have target weights \(0.50\), \(0.30\), and \(0.20\), and only the first two have relevant observations, then \(C=0.50+0.30=0.80\). The remaining \(0.20\) is not directly represented under this grouping. No arithmetic on \(C\), however, establishes that predictions are reliable for the represented groups or that missing groups can safely be ignored.
The choice of groups also matters. A broad “adult” category can conceal a serious gap if the intended use concerns older adults with impaired organ function. Use clinically meaningful distinctions and examine the underlying counts and measurements, not just a summary percentage. A coverage calculation is a screening aid, not a universal acceptance threshold.
Worked Example: Population Coverage for a Proposed Use
Hypothetical scenario. A model is intended to inform dosing in adults with moderate-to-severe disease. For a first-pass review, the target population is divided into moderate and severe disease, with target proportions of 60% and 40%. A dataset contains 180 patients with moderate disease and 20 with severe disease. Assume both groups used the same relevant endpoint definition and were observed under the treatment of interest.
Both target groups have at least some representation, so \(I_1=1\) and \(I_2=1\). The basic coverage summary is:
The summary is 100%: neither target group is entirely absent. But the dataset proportions are \(180/200=0.90\), or 90%, moderate disease and \(20/200=0.10\), or 10%, severe disease. The target population is 40% severe, so the severe group is relatively sparse in the data.
Therefore, the correct conclusion is not “the dataset represents the target population perfectly.” It is that both groups are present, while the severe group is substantially less represented than its target share. The 20 severe-disease observations may be insufficient for some modeling tasks, and the analyst should report that limitation rather than letting \(C=1.00\) hide it. This assessment supports considering the dataset for the use, but does not demonstrate adequate precision or predictive performance.
Measurements, Timing, and Treatment Experience
The timing of observations must correspond to the process being modeled and the time at which the output will be used. Sparse sampling can still be informative when it captures the relevant response, but measurements that miss an early peak, a delayed effect, or a decision time point may not support the intended quantity. Check actual collection times as well as scheduled times: a nominal visit at week 4 does not guarantee that patients were measured at week 4.
Treatment range is similarly tied to the question. Data collected at one dose may inform description at that dose, but usually offer limited direct information about how outcomes change across doses. If the use concerns a new regimen, examine whether data cover the relevant doses or exposures and whether combinations, adherence, and treatment changes resemble the intended application. Do not treat extrapolation beyond observed treatment experience as if it were direct empirical coverage.
These checks are not a request to select a model structure or a level of model detail. They establish what evidence the dataset contains. A mismatch may indicate that a proposed use needs additional data, a narrower claim, or an explicit rationale for applying evidence beyond its observed setting.
Worked Example: Treatment-Range Support
Hypothetical scenario. A proposed use concerns predicting response at exposures from 2 to 8 mg/L. A dataset contains 50 patients: 30 have measured exposures from 2 to 4 mg/L, and 20 have exposures from 4 to 6 mg/L. None has an exposure above 6 mg/L. For a simple range summary, define direct coverage as the proportion of observations with exposure inside the intended range of 2 to 8 mg/L.
All 50 observations fall inside that broad intended interval, so the direct-in-range proportion is \(50/50=1.00\), or 100%. This figure alone is misleading: the data contain no observations from 6 to 8 mg/L. Divide the intended range into three intervals, 2–4, 4–6, and 6–8 mg/L. The observed counts are 30, 20, and 0, respectively.
The data directly inform the first two intervals but not the third. In particular, the fraction of observations in the highest interval is \(0/50=0\). Thus, although every observation is inside the overall target range, the evidence does not cover its upper part. The dataset may support analysis over the observed range of 2 to 6 mg/L; using it to predict from 6 to 8 mg/L requires extrapolation and additional justification. The range check also assumes exposures were measured comparably and that the endpoint is available for these observations—conditions that must be checked rather than presumed.
Missing Data: Amount, Pattern, and Consequence
A missing-data percentage is a useful starting description, not a sufficient judgment. Record which variables are missing, for which patients, at which times, and after which events. Missingness can be especially consequential when it removes observations needed for the intended use, such as follow-up after treatment starts or measurements from patients with severe disease.
Consider a dataset in which patients who experience worsening symptoms are more likely to miss a follow-up visit. A complete-case summary uses only patients with observed follow-up. If those patients tend to be doing better than the patients whose outcomes are missing, the observed mean response may overstate the response in the full group. This is a possible bias, not a certainty: the direction and size depend on how the missing outcomes differ. When the relationship between missingness and unobserved values cannot be checked from observed data alone, assumptions or additional information are needed to justify an analysis.
Also distinguish structural absence from accidental absence. A variable never collected in a study cannot be recovered by counting complete records. By contrast, a measurement missed at some visits may still be usable for some purposes, depending on its pattern, the data available, and the assumptions made. A dataset should not be called complete merely because a large proportion of its rows contain some observations.
Worked Example: How Missing Follow-Up Changes the Evidence
Hypothetical scenario. A study records a symptom improvement score at follow-up for 80 of 100 patients. The observed scores sum to 320 points, giving a mean among measured patients of \(320/80=4\) points. The other 20 patients have no follow-up score; records show that 12 of them stopped attending after symptoms worsened.
The available-data calculation is:
This is the mean for the 80 patients with recorded follow-up, not automatically the mean for all 100 patients. The fact that 12 missing measurements followed worsening symptoms raises a specific concern: observed patients may have better outcomes on average. But the missing patients’ scores are unknown, so the full-group mean and the amount of any bias cannot be computed from the stated data.
The defensible conclusion is that the dataset has 20% missing follow-up, with a pattern that may be related to outcome. Reporting “the mean improvement was 4 points in all 100 patients” would overstate what was calculated. The 4-point result describes observed cases only; its relevance to the intended population depends on how the missingness is addressed and on the assumptions used. Those assumptions should be stated, and the missing outcomes should not be silently treated as zero or as equal to observed outcomes.
Making a Bounded Judgment
A useful assessment states both what is supported and what is limited. It should connect each limitation to the intended quantity or decision rather than produce an unqualified label such as “representative.” For example: “The dataset includes the intended age groups and endpoint definition, but has few observations at the upper exposure range and substantial follow-up missingness after worsening. It therefore offers direct evidence over the lower observed range; application to high exposures or to patients with worsening outcomes requires additional support.”
When a mismatch is found, possible responses include narrowing the intended use to the population, times, or treatment range actually represented; seeking complementary data; or making the limits of the evidence explicit. These options are not interchangeable. Adding a dataset with a different endpoint may not resolve an endpoint mismatch, and adding many observations from the same narrow exposure range may not fill a gap at higher exposures.
The key comparison is not simply between “good” and “bad” data. It is between the information required by the stated use and the information the dataset actually contains. A careful match assessment helps prevent a model from being asked to answer a question its data cannot adequately inform.
Common Pitfalls
- Using sample size as a substitute for relevance. A large dataset can still omit the intended population, endpoint, or treatment range. Report size alongside the dimensions that determine relevance.
- Equating any representation with adequate representation. A group with a few observations is technically present, but may be weakly supported. Examine counts and distribution, not only whether a group appears.
- Calling the intended range “covered” because all data lie inside it. Data can lie within a broad target interval while leaving part of that interval unobserved. Inspect where observations occur across the range.
- Treating missingness as harmless because the percentage is small. Even a modest amount of missing data can matter if it is concentrated among patients important to the intended use.
- Assuming a match proves model performance. Data relevance is necessary for a credible use, but does not establish that the model predicts adequately or informs a decision correctly.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A proposed use concerns patients with severe disease, but the dataset contains only mild and moderate disease. What population limitation should be reported?
- A dataset includes measurements at baseline and month 12. The intended prediction concerns response at month 1. Why might the dataset be insufficient for that use?
- For the coverage summary, target groups have weights 0.50, 0.30, and 0.20. Relevant observations are present in the first and third groups only. Calculate \(C\), and state what it does not establish.
- A dataset has outcomes for 90 of 100 patients, and the 10 missing outcomes are mostly from patients who discontinued for lack of efficacy. Can the observed mean automatically be interpreted as the mean for all 100? Explain.
- A dataset contains 40 patients at exposures from 1 to 3 mg/L and none from 3 to 5 mg/L. The intended use includes 1 to 5 mg/L. What can be said about direct treatment-range support?
Solutions. (1) The dataset has no direct observations from the severe-disease group, so it does not directly represent that target population. Applying it there would require additional support or a narrower intended use. (2) The observations do not directly capture the month-1 response; baseline and month-12 data alone may not inform the intended early response quantity. (3) \(C=(0.50)(1)+(0.30)(0)+(0.20)(1)=0.70\), or 70%. This indicates the target share in groups with some representation, not adequate sample size, precision, or model performance. (4) No. The observed mean describes the 90 patients with recorded outcomes. Discontinuation for lack of efficacy suggests missingness may be related to outcomes, so interpreting that mean for all 100 needs explicit assumptions or additional information. (5) The dataset directly supports only the observed 1-to-3 mg/L range. It contains no direct observations from 3 to 5 mg/L, so use there would extend beyond the observed treatment experience.