Specification Analysis Across Moments And Data Targets

Why Compare Moments and Data Targets?

A structural model is a set of restrictions on how observed outcomes are generated. Estimation chooses parameter values so that some implications of the model agree with the data. That agreement is not, by itself, a complete assessment of specification: a model may fit the moments used to estimate it while missing other important features of the data. Specification analysis therefore asks not only whether a model fits overall, but which implications fit, which do not, and what restrictions could explain the pattern.

A target can be a mean, variance, probability, quantile, transition rate, or other summary chosen to represent a feature of the data. It need not be a raw moment in the narrow mathematical sense. For each target, the structural model implies a value at a given parameter vector. Comparing the observed and implied values makes discrepancies visible in interpretable units. Comparing several targets together also requires attention to their sampling uncertainty and dependence.

Definition: Let $t_n$ be a vector of observed sample targets, and let $m(\theta)$ be the corresponding vector of targets implied by a structural model with parameter vector $\theta$. At an estimate $\hat{\theta}$, the target discrepancy is $d=t_n-m(\hat{\theta})$. Its $j$th entry is the observed target minus the model-implied target. A positive entry means the model predicts less than observed for that target; a negative entry means it predicts more.

The sign convention matters: reversing it does not change a squared fit criterion, but it reverses the substantive description. Also, discrepancies should first be read in the targets’ original units. A difference of 0.02 is two percentage points if the target is a probability, but it is a very different amount if the target is measured in thousands of dollars.

Read Discrepancies Before Combining Them

A useful first pass is a table containing each target, its observed value, the model-implied value, their signed difference, and—when available—the target’s standard error. This makes systematic patterns easier to see than one overall fit statistic does. For example, several underpredicted high-outcome targets might point to a different problem than a single large discrepancy in an unrelated target.

A raw difference does not say how surprising it is. For a target with estimated standard error $SE_j$, a simple scale-adjusted residual is $d_j/SE_j$. This places discrepancies in approximate standard-error units, but it does not account for dependence across targets. When targets are correlated, assess the vector using their joint covariance matrix rather than treating each marginal residual as independent.

Definition: If the target vector has estimated covariance matrix $\widehat{\Sigma}_n$, a joint quadratic discrepancy is $Q=d'\widehat{\Sigma}_n^{-1}d$, provided the matrix is invertible. This summarizes discrepancy relative to estimated sampling variation and dependence. It is a diagnostic summary; its value alone does not identify which structural restriction is responsible.

The covariance must correspond to the targets being compared. If targets come from the same sample, their estimation errors may move together. If the model-implied targets depend on estimated parameters, uncertainty in those estimates can also matter for formal inference. For descriptive diagnostics, report the construction of the discrepancy and its scale; do not interpret a standardized residual as an exact test unless the relevant distributional assumptions and reference procedure have been established.

Worked Example: Interpreting Signed Target Differences

Suppose, purely hypothetically, a researcher estimates a structural participation model and compares it with three observed targets from a sample of eligible households. The targets are the participation rate, the participation rate among households with children, and average monthly program spending among participants. At the fitted parameter vector, the observed and implied values are:

Target	Observed	Model-implied
Participation rate	0.300	0.280
Participation rate, households with children	0.420	0.370
Monthly spending among participants	$480	$510

Using $d=t_n-m(\hat{\theta})$, the discrepancies are:

d = \begin{pmatrix} 0.300-0.280 \\ 0.420-0.370 \\ 480-510 \end{pmatrix} = \begin{pmatrix} 0.020 \\ 0.050 \\ -30 \end{pmatrix}.

The model underpredicts overall participation by 0.020, or 2.0 percentage points, and underpredicts participation among households with children by 0.050, or 5.0 percentage points. It overpredicts spending by $30 per participating household per month. The pattern is more informative than the statement that “the model misses”: participation is low in both groups, especially for households with children, while spending is high. These signs guide further diagnosis, but do not alone establish which structural mechanism is wrong.

Comparing Targets on Their Joint Scale

When target estimates have different standard errors or are correlated, marginal comparisons can give a misleading impression of the joint discrepancy. For a vector of targets with covariance $\widehat{\Sigma}_n$, the quadratic quantity $Q$ accounts for both features. Equivalently, one can transform the discrepancy into whitened components. Such components depend on the order or factorization used to whiten the vector, so they are useful for diagnosis only when that construction is stated. The quadratic total does not depend on the order.

Worked Example: A Correlated Pair of Targets

Consider two hypothetical observed targets: the estimated rate of a service in one subgroup and the estimated rate in a second subgroup. The fitted model’s signed discrepancies are $d_1=0.04$ and $d_2=-0.03$, or 4 and -3 percentage points. Their standard errors are 0.02 and 0.03, and their estimated correlation is 0.75. Thus the marginal standardized discrepancies are $z_1=0.04/0.02=2.00$ and $z_2=-0.03/0.03=-1.00$.

For two standardized targets with correlation $\rho$, the joint quadratic discrepancy is

Q = \frac{z_1^2-2\rho z_1z_2+z_2^2}{1-\rho^2} = \frac{2^2-2(0.75)(2)(-1)+(-1)^2}{1-0.75^2} = \frac{8}{0.4375} \approx 18.29.

The positive correlation matters: if sampling errors tend to move the two target estimates together, a discrepancy that is positive for one and negative for the other is less aligned with their usual joint sampling variation. Treating the targets as independent would instead give $2^2+(-1)^2=5$. The larger covariance-adjusted value highlights the opposing pattern. This calculation describes joint scale-adjusted mismatch; it is not, by itself, a formal hypothesis-test conclusion.

A further distinction is whether a target helped estimate the model. Targets used directly in a minimum-distance or indirect-inference criterion are in-sample targets: their fit reflects the estimation procedure as well as specification. Other targets can be reserved for validation. A model that matches estimation targets but misses validation targets may be fitting only a narrow set of implications. Conversely, a good validation fit is evidence about those particular targets, not proof that every feature of the structural model is correct.

Locate the Restrictions Behind a Pattern

A discrepancy vector does not label its cause. Structural restrictions jointly determine the model-implied targets, and changing one parameter or restriction can move several targets at once. Diagnosis therefore requires linking a pattern to restrictions in the model and checking that link through controlled comparisons.

A practical procedure is to group targets by the mechanism or population they represent, inspect signed and scaled residuals within each group, and identify restrictions that connect those implications. Then compare the original specification with a clearly defined alternative that relaxes a relevant restriction. Re-estimate both specifications using a stated target set and criterion. Record how each target changes, not just whether the overall objective falls. If multiple restrictions are changed at once, the source of any fit improvement is ambiguous.

This is a diagnostic exercise, not a license to add parameters until every observed target is matched. A more flexible model will generally be able to fit at least as well in sample, but its extra flexibility may lack substantive justification or may make parameters weakly identified. Earlier material on minimum-distance estimation and overidentifying-moment tests provides the estimation and formal-testing context; here the emphasis is on reading the pattern of target fit and relating it to restrictions.

Worked Example: Diagnosing an Equality Restriction

Suppose a hypothetical structural model predicts treatment take-up in two regions. The original model imposes the restriction that the two regional effects are equal, so it predicts the same take-up rate $\theta$ in both regions. Observed rates are 0.40 and 0.48. For a transparent diagnostic, suppose both targets have standard error 0.02, are treated as uncorrelated, and the constrained fit uses equal weights.

Under the equality restriction, the fitted common rate is the average:

\hat{\theta}=\frac{0.40+0.48}{2}=0.44.

The observed-minus-implied discrepancies are $0.40-0.44=-0.04$ and $0.48-0.44=0.04$, or -4 and +4 percentage points. In standard-error units they are $-0.04/0.02=-2.00$ and $0.04/0.02=2.00$. The opposite signs are consistent with the equality restriction forcing a common prediction between the two observed rates. The sum of squared standardized discrepancies is

Q_{\mathrm{equal}}=(-2)^2+2^2=8.

Now relax only the equality restriction, allowing separate regional rates. The fitted predictions become 0.40 and 0.48, so both discrepancies are zero and $Q_{\mathrm{separate}}=0^2+0^2=0$. This comparison isolates the equality restriction as the source of this particular in-sample mismatch, given the stated targets, weights, and other fixed parts of the model. It does not establish that the separate-rate specification is substantively correct: the targets may be noisy, and the relaxed model uses an additional free parameter. Other evidence and an appropriate formal assessment are needed before adopting it.

A Short Justification for Residual Patterns

The geometry of minimum distance explains why some discrepancies remain even after estimation. Let the criterion be $J(\theta)=[t_n-m(\theta)]'W[t_n-m(\theta)]$, where $W$ is a positive-definite weight matrix. Suppose an interior minimizer $\hat{\theta}$ exists and $m$ is differentiable. Define $G(\hat{\theta})=\partial m(\hat{\theta})/\partial\theta'$, the matrix of local target changes with respect to parameters. Differentiating the criterion gives

\frac{\partial J(\theta)}{\partial\theta} = -2G(\theta)'W[t_n-m(\theta)].

At an interior minimum, the gradient is zero, so

G(\hat{\theta})'W d=0.

Thus the fitted discrepancy is orthogonal, in the $W$-weighted geometry, to local target changes available through the estimated parameters. This does not mean $d=0$. If there are more targets than freely varying parameter directions, or if a restriction prevents parameters from moving in a needed direction, some discrepancies can remain. This also explains why one cannot infer a unique cause from a residual alone: several restrictions or parameter changes may generate similar movements in target space.

List the targets and units.
State which targets are observed, how each is measured, and whether it was used for estimation or reserved for validation.

Compute signed discrepancies.
For each target, subtract the model-implied value from the observed value, and describe the sign in context.

Assess scale and dependence.
Compare discrepancies with their standard errors and use joint covariance information when targets are dependent.

Trace a pattern to restrictions.
Form a specific explanation, then compare a model variant that changes the relevant restriction while keeping other choices clear.

Common Pitfalls

Calling a small aggregate criterion “good specification.” A weighted total can conceal an important miss in one target or target group. Inspect signed, target-level discrepancies as well as any aggregate.
Comparing raw differences across incompatible units. A discrepancy of 0.03 in a probability and a discrepancy of 30 in dollars cannot be ranked by magnitude alone. Use context and an appropriate scale.
Ignoring correlated targets. Marginal standard errors do not describe the joint pattern when targets share data or move together. Use the joint covariance when evaluating that pattern.
Attributing every residual to one restriction. A residual is evidence of mismatch, not a diagnosis. Compare controlled model variants and check which implications move.
Treating improved fit after relaxing restrictions as proof. Added flexibility can reduce in-sample discrepancies by construction. The substantive interpretation and the consequences for identification still matter.

Key takeaway: Specification analysis is most useful when it preserves the structure of the evidence: define the targets, report observed-minus-implied discrepancies in context, account for scale and dependence, and connect systematic patterns to restrictions through controlled comparisons. A fit summary describes mismatch; it does not, on its own, explain or resolve it.

Check Your Understanding

Answer each question using the definitions and diagnostic logic developed above.

A model predicts a 0.25 participation rate when the observed rate is 0.28. What is the signed discrepancy under the convention used here, and how should it be described?
Why might two target discrepancies of the same raw magnitude provide different amounts of evidence of mismatch?
Two standardized target residuals have opposite signs and their estimates are positively correlated. Why can treating them as independent misrepresent the joint pattern?
A model fits the moments used in estimation closely but misses several reserved validation targets. What does this suggest, and what does it not establish?
Why does relaxing a restriction and improving in-sample fit not by itself show that the relaxed model is preferable?