Why Measuring Innovation Is Difficult
Innovation is central to long-run productivity growth, but it is not directly observed as a single, complete quantity. A firm may spend on research that produces no marketable invention; another may improve a process without patenting it. Even a successful invention can take years to affect measured output. Empirical work therefore uses indicators—observable variables intended to capture part of the innovation process—and must be careful not to confuse an indicator with the underlying concept.
A knowledge spillover occurs when knowledge created by one firm, researcher, or institution benefits another without the creator receiving full compensation for that benefit. A nearby firm might learn from employees who move between firms, published research, suppliers, or informal contact. Proximity or exposure alone does not prove that a spillover occurred: firms may be similar for other reasons, and knowledge may not transfer or be useful to the recipient.
Earlier material on the Romer model, creative destruction, technology diffusion, and absorptive capacity provides theoretical reasons to expect research, invention, and adoption to matter for growth. Measuring those mechanisms requires additional decisions: which data represent each mechanism, over what period, and against what comparison.
What Common Indicators Capture
| Indicator | What it can capture | Important limitation |
|---|---|---|
| Research and development (R&D) expenditure | Resources devoted to research and development | Spending is an input, not a count of successful innovations; costs and accounting practices can differ. |
| Patent counts | Inventions for which patent protection is sought and granted | Many inventions are not patented; patenting propensities and patent value vary. |
| Patent citations | Recorded references to earlier patents, sometimes used as a proxy for influence | Citations are incomplete, affected by rules and strategy, and depend on how long a patent has been observed. |
| Productivity measures | Output relative to measured inputs | Residual productivity may reflect unmeasured inputs, utilization, prices, or measurement error as well as technology. |
| Adoption data | Whether or how quickly a firm uses a technology or practice | Adoption is not the same as invention; firms differ in suitability, resources, and reporting. |
Research intensity is often measured as R&D expenditure divided by sales or another scale measure. This helps compare firms of different sizes, but the ratio has an interpretation tied to its denominator: a rise can occur because R&D increased, because sales fell, or both. Patent counts have a similar scale problem. Counts may be divided by employment, sales, or research spending, but the resulting ratio answers a different question from the raw total.
A patent is also not a uniform unit of inventive value. Some patents may have little commercial or technological impact; a small number may be highly influential. Citation-based measures attempt to distinguish influence, but recent patents have had less time to accumulate citations. Comparisons should therefore consider the observation window, technological field, patenting rules, and whether the count includes applications, grants, or both.
Worked Example: Comparing Patent and Research Indicators
Two hypothetical firms report the following annual data. Firm Alder spends $12 million on R&D, has sales of $120 million, and receives 6 patent grants. Firm Birch spends $10 million, has sales of $80 million, and receives 5 patent grants. Calculate R&D intensity and patents per $10 million of R&D for each firm. Explain what the comparisons do and do not establish.
Method. Calculate each ratio using the stated denominator, then compare like with like.
Alder’s R&D intensity is \(12/120=0.10\), or 10%. Its patents per $10 million of R&D are \(6/(12/10)=6/1.2=5\).
Birch’s R&D intensity is \(10/80=0.125\), or 12.5%. Its patents per $10 million of R&D are \(5/(10/10)=5/1=5\).
Birch devotes a larger share of sales to R&D, while the firms have the same patent-grant count per $10 million of R&D in this simple calculation. These ratios do not establish that the firms have equal innovation quality or research productivity. Patent values may differ, research may produce results after a lag, and costs or patenting choices may differ. The common mistake is to interpret the ratio as a direct measure of the number or value of innovations.
Check. Alder’s \(12/120\) is 0.10, and Birch’s \(10/80\) is 0.125. Both patent ratios equal 5, using the stated $10 million unit.
Productivity as a Residual Measure
Productivity measures relate output to inputs. A basic growth-accounting calculation uses a production function to separate the contribution of measured inputs from the change in output not accounted for by those inputs. That remaining component is often called total factor productivity (TFP) growth. It is useful as an indicator of technological or organizational change, but it is not a direct observation of innovation.
Assume, for this calculation, that output follows a Cobb–Douglas production function \(Y=A K^\alpha L^{1-\alpha}\), with constant returns to scale, a fixed output elasticity \(\alpha\) for capital, and measured capital \(K\) and labor \(L\). Here \(A\) is the productivity term. Taking logarithms gives \(\ln Y=\ln A+\alpha\ln K+(1-\alpha)\ln L\). Subtracting the equation at the start of a period from the equation at the end gives the growth-accounting relationship:
The growth rates \(g_Y\), \(g_K\), and \(g_L\) are percentage changes over the same period. The approximation uses percentage growth rates in place of exact log differences; it is most appropriate for modest changes. The equation says that output growth left over after subtracting the weighted growth of measured inputs is attributed to \(A\) in this model. It does not show that every part of the residual was caused by research or innovation.
Worked Example: Calculating a Productivity Residual
A hypothetical manufacturer’s real output grows by 5% over a year. Its measured capital stock grows by 4%, and its labor input grows by 2%. Assume the Cobb–Douglas model above with \(\alpha=0.30\), constant returns to scale, and the same annual measurement period for all growth rates. Estimate TFP growth using the growth-accounting approximation.
Method. Subtract the capital and labor contributions from output growth. Capital’s weighted contribution is \(0.30(4\%)=1.20\%\). Labor’s weighted contribution is \((1-0.30)(2\%)=0.70(2\%)=1.40\%\).
Together, measured input growth contributes \(1.20\%+1.40\%=2.60\%\). The residual is \(5.00\%-2.60\%=2.40\%\). Estimated TFP growth is therefore 2.40% for the year.
This is a model-based residual, not a direct count of innovations. If capacity utilization rose, labor quality improved but was not measured, or output and inputs were measured with error, those changes could also affect the residual. The common mistake is to call all residual growth “the effect of R&D”; the calculation includes anything that changes measured output relative to the specified measured inputs.
Check. The input contributions sum to \(2.60\%\), and \(5.00\%-2.60\%=2.40\%\), as required.
From Association to Spillover Evidence
A positive relationship between one firm’s research and another firm’s productivity is not, by itself, evidence of a knowledge spillover. For example, firms in the same industry may both benefit from an unobserved demand boom. Productive firms may also be more able to finance research, creating reverse causality between productivity and R&D. A credible empirical design must explain why the observed variation in exposure to other firms’ knowledge is not simply tracking these other influences.
A difference-in-differences comparison is one possible design. It compares the before-to-after change in an outcome for an exposed group with the corresponding change for a comparison group. Under the parallel-trends assumption—that absent the exposure, the groups would have had the same average change—the difference can estimate an exposure effect. For a spillover interpretation, researchers also need a plausible exposure measure and reasons to rule out other changes that affect the exposed group differently.
Worked Example: Estimating a Possible Local Spillover
A hypothetical research center opens near a group of manufacturing firms. Researchers compare productivity-index values for nearby firms (the treated group) and firms farther away in the same broad sector (the comparison group). The treated group’s average index is 100 before and 108 after; the comparison group’s is 102 before and 105 after. Assume the index is measured consistently and higher values indicate greater productivity. Calculate the difference-in-differences estimate and state the condition needed for a causal interpretation.
Method. First calculate each group’s change. Nearby firms change by \(108-100=8\) index points. Farther firms change by \(105-102=3\) index points. Subtract the comparison change from the treated change: \(8-3=5\) index points.
The estimated difference-in-differences effect is 5 index points. Under parallel trends, and provided no other event-specific change differentially affects nearby firms, this is an estimate of the effect of exposure to the center on the measured outcome. It is only evidence of a knowledge spillover if the design supports the claim that exposure conveys knowledge rather than, for example, changing local labor supply or demand.
The calculation alone does not verify parallel trends. Researchers would typically inspect outcomes before the center opened, examine whether the groups were already on different trajectories, and test whether the result depends on reasonable alternative comparison groups or exposure definitions. The common mistake is to report the treated group’s 8-point increase as the effect; the comparison group also rose by 3 points, so the estimated differential change is 5 points.
Check. The difference in changes is \((108-100)-(105-102)=8-3=5\) index points.
Challenges in Estimating Knowledge Spillovers
Empirical spillover estimates are sensitive to how knowledge exposure is defined and how firms are selected into exposure. Common challenges include:
- Measurement error. R&D records may omit informal research; patent records omit unpatented innovation; citations capture only recorded references; and survey responses may be incomplete. No single proxy captures all knowledge creation or transfer.
- Reverse causality. More productive firms may attract researchers, spend more on R&D, or patent more. A positive correlation between exposure and productivity may therefore run partly from productivity to measured innovation activity.
- Omitted variables. Firms near research centers or in research-intensive industries may differ in skills, infrastructure, demand, or regulation. If those factors also affect productivity, an estimated relationship can be confounded.
- Selection into adoption. Firms with stronger management, greater resources, or better-matched workers may adopt technology sooner and also perform better for other reasons. Comparing adopters with non-adopters without addressing selection may overstate the adoption effect.
- Timing and lag choice. Research may take years to produce patents, adoption, or measured output. An interval that is too short can miss effects; a long interval can include unrelated changes.
- Spillover boundaries. Geographic distance, industry membership, supplier relationships, and patent citations are different ways to define exposure. Each can miss relevant knowledge flows or include firms that do not actually exchange useful knowledge.
A useful empirical report distinguishes its descriptive result from its causal claim. “Firms with more nearby R&D have higher productivity” describes an association. “Nearby R&D raises these firms’ productivity through a knowledge spillover” makes a causal and mechanism-specific claim. The latter requires evidence that alternative explanations are adequately addressed, not merely a positive coefficient or a plausible story.
Specify whose knowledge may benefit which firms, and through what plausible channel.
State whether exposure is measured by research spending, patents, location, citations, adoption, or another indicator, and identify what it misses.
Explain which firms serve as controls and why their outcomes provide a credible counterfactual for exposed firms.
Separate the estimated association or design-based effect from claims about the channel, and report important assumptions and limitations.
Why These Distinctions Matter
The same numerical result can answer different questions depending on its construction. R&D intensity describes research spending relative to a chosen scale measure. A patent count describes recorded patenting, not the full stock of ideas. A productivity residual describes output growth unexplained by specified measured inputs under a model. A difference-in-differences estimate describes a differential change under identifying assumptions. None should be relabeled as a direct measure of “innovation” without qualification.
For actuarial and economic analysis, this discipline matters when interpreting productivity trends, policy evaluations, or hypothetical investments in research and technology adoption. A careful conclusion identifies the indicator, its denominator and time period, the comparison being made, and the assumptions required for a causal interpretation. This tutorial is aligned with the SOA VEE Economics (Validation by Educational Experience) syllabus.
Check Your Understanding
For each problem, identify the relevant measure and interpret it cautiously.
- A hypothetical firm’s R&D spending rises from $8 million to $10 million while sales rise from $100 million to $125 million. Calculate R&D intensity in both years and explain whether the intensity changed. (Answer: 8/100 = 8% initially and 10/125 = 8% afterward; the intensity is unchanged.)
- Under the Cobb–Douglas growth-accounting approximation, output grows 4%, capital grows 2%, labor grows 1%, and \(\alpha=0.40\). Calculate estimated TFP growth. (Answer: \(4\%-0.40(2\%)-0.60(1\%)=2.60\%\).)
- A patent cohort has 24 recorded citations after two years and 36 after four years. Why is the four-year count not, by itself, evidence that the later cohort’s patents are more influential? (Answer: It has had twice as long to accumulate citations; compare cohorts over a common exposure window or adjust for citation opportunity, while recognizing that citations remain an imperfect proxy.)
- After a new supplier technology becomes available, adopting firms’ average productivity rises from 50 to 58, while non-adopters’ rises from 48 to 53. Calculate the difference-in-differences estimate and name one condition needed for a causal interpretation. (Answer: \((58-50)-(53-48)=8-5=3\) index points; parallel counterfactual trends is required, along with no differential concurrent shock.)
- Researchers find that firms with more neighboring R&D have higher productivity. Give two distinct reasons this association may not measure a causal knowledge spillover. (Answer: For example, more productive firms may attract neighboring researchers or finance more R&D, creating reverse causality; alternatively, unobserved local skills, infrastructure, or demand may raise both R&D exposure and productivity.)