Building an Empirical Study Map

From an Economic Question to a Study Map

An empirical study can calculate a precise difference and still leave its central question unanswered. The difference might describe the wrong population, measure the wrong outcome, or compare groups whose outcomes would have differed even without a policy. Before selecting an estimator, an economist needs to specify what the study is intended to learn and what features of the data would support that interpretation.

A study map is a compact account of how an economic question connects to a target population, an estimand, a data-generating process, recorded data, and the assumptions and limitations that stand between the data and the intended conclusion. It is a planning and interpretation tool, not a statistical estimator. Its purpose is to make the study’s claims and dependencies explicit.

Consider the question, “Does a publicly funded job-training offer increase the annual earnings of eligible applicants one year later?” To map it, we must decide who counts as eligible, what “offer” means, which earnings outcome and follow-up date matter, and whether the available data can support a causal comparison. Each choice narrows the claim the study can responsibly make.

Define the Population and Estimand First

The target population is the group of units to which the research question refers. It is not automatically the set of records in the analysis. A study of applicants to one city’s training program may be designed to speak about those applicants, all eligible residents of the city, or eligible residents elsewhere. Those are different targets, and data from one group do not automatically establish results for another.

The estimand is the exact quantity of interest in that population. For a causal question about an offer, let \(Y_i(1)\) denote person \(i\)’s annual earnings one year later if offered training, and \(Y_i(0)\) earnings at that horizon if not offered. For a defined target population of \(N\) eligible applicants, one possible estimand is the average effect of the offer:

This is an effect of being offered the program, not necessarily an effect of attending it. Keeping the distinction explicit prevents a study of offers from being described as if it estimated the effect of participation. A useful specification also states the unit, outcome units, and time horizon: here, the unit is an eligible applicant, the outcome is annual earnings in dollars, and the horizon is one year after the offer.

Describe the Data-Generating Process and Data Structure

The data-generating process (DGP) is the process that determines the relevant assignments or exposures, outcomes, inclusion in the data, and recorded values. A study map does not need a complete model of every social mechanism. It should, however, describe the features that matter for the proposed comparison. For the offer study, these include how offer assignment occurs, whether applicants take up training, what affects later earnings, who can be followed, and how earnings are obtained.

A simple representation distinguishes offer assignment \(Z_i\), training participation \(D_i\), the outcome \(Y_i\), and recorded outcome \(Y_i^{\mathrm{obs}}\). The offer may influence participation, while prior experience, health, or local labor demand may influence participation and earnings. The recorded outcome may differ from the economic outcome of interest because of reporting or administrative-record limitations. These distinctions help identify what the data actually compare.

The data structure describes how observations are organized over units and time. If each applicant contributes one row with baseline characteristics and one-year earnings, the outcome data are a cross-section at that follow-up, even if baseline information was collected earlier. If each applicant has earnings recorded at several dates, the data are longitudinal. The structure affects which changes and comparisons can be described; it does not, by itself, establish causality.

Identification Is Not the Same as Sampling

An estimand is identified when the assumptions and available information determine it from the distribution of observed data. Identification is a claim about whether the data and assumptions distinguish the desired quantity; it is not a claim that an estimate is precise or that the assumptions are true. Sampling concerns whether observed units represent the target population and how much sample-to-sample variation remains. A study can have a credible comparison for its participants but a narrow target, or broad coverage but a comparison that does not identify a causal effect.

For a randomized offer, assignment must be implemented as described, the offer must have a consistent meaning, and relevant outcomes must be observed. In a finite group of \(N\) applicants, define the assignment indicator \(Z_i\) as 1 for an offer and 0 otherwise. Under complete random assignment with \(n_1\) offers and \(n_0\) non-offers, the difference in group means has a useful justification: each group is, by the assignment mechanism, a random subset of the applicants. Therefore, averaging over the possible assignments, each group mean has expectation equal to the corresponding finite-population mean of potential outcomes. Subtracting those expectations gives:

This is a justification for the randomization-based difference as an unbiased estimator of the average offer effect in the randomized group, under the stated conditions. It does not say that the realized difference must equal the estimand. Nor does it justify extending the result to people outside that group without further evidence or assumptions.

Sampling Limits and the Target of Generalization

A study map should name the sampling frame and explain how the observed units relate to the target population. The earlier material on sampling frames, inclusion probabilities, nonresponse, and coverage error provides the relevant vocabulary: a frame can omit target units, participation can be selective, and groups can have different chances of appearing in the data. These issues affect how far a result can be generalized, even when the comparison within the observed sample is well designed.

The target population also determines how subgroup results should be combined. A sample’s composition need not match the target population’s composition. If subgroup effects differ, an unweighted average of sample subgroup effects may describe the sample rather than the target.

Measurement Risks Belong on the Map

The variable in the economic question is often not recorded directly. “Annual earnings” might mean taxable earnings in administrative records, self-reported earnings, or earnings from a particular employer. Each operational definition has a scope. Administrative data may omit informal work or earnings outside the covered system; survey reports may involve recall error. The earlier material on operationalization, measurement error, misclassification, missing data, linkage, and data revisions provides tools for examining these risks.

The key study-map question is whether a measurement problem changes the intended interpretation. If the outcome is measured with similar error in both randomized groups, the resulting contrast may still be affected by noise or incomplete coverage. If reporting accuracy differs by offer status, the group difference can also be distorted systematically. For example, suppose true mean earnings are $34,000 for the offered group and $32,000 for the non-offered group, but the offered group overreports by an average of $1,000 while the other group reports accurately. The recorded difference is \((\$34{,}000+\$1{,}000)-\$32{,}000=\$3{,}000\), although the true difference is $2,000. Random assignment does not repair differential measurement.

A Practical Study-Map Checklist

Before interpreting an empirical result, write down each link in the study’s argument. A useful map includes:

• Question and target: Which population, unit, policy or exposure, outcome, and time horizon does the question specify?
• Estimand: What precise population quantity answers that question, and is it an association, a descriptive contrast, or a causal effect?
• DGP: How are assignment or exposure, outcomes, inclusion, and missingness produced? Which factors could affect both exposure and outcome?
• Data structure and measurement: What does one record represent, when are variables observed, and how do recorded measures relate to the intended concepts?
• Identification: Which assumptions make the desired quantity recoverable from the observed data, and which are design features versus unverified claims?
• Sampling and generalization: Which target units can enter the data, who is missing or nonresponding, and what limits extension beyond the observed sample?

These questions keep a study’s conclusion proportional to its evidence. They also clarify what additional data or design changes would be most useful: better follow-up, broader coverage, more credible assignment, improved measurement, or a narrower target population.

Common Mistake: Treating the Estimate as the Study Question

A common mistake is to see a numerical difference and describe it immediately as “the effect.” That wording skips the definition of the estimand and the assumptions needed to connect the comparison to it. A second mistake is to treat a large or representative sample as a substitute for identification. More observations can improve the description of the observed data, but they do not automatically remove confounding, differential measurement, or a mismatch between the sample and target.