Tutorials › Propensity Score Matching and Instrumental Variables › Worked Example of a Transparent Comparative-Study Report

Home  /  Tutorials  /  Reporting Observational Comparative Studies

Reporting Observational Comparative Studies · Tutorial 672 of 699

Worked Example of a Transparent Comparative-Study Report

Learn to assemble a transparent report that connects the study question, matched analysis population, diagnostics, estimated effect, sensitivity analysis, and limitations.

Intermediate 10 min read

What You'll Learn

  • State the question and target population for a matched comparison.
  • Reconcile the study cohort, exclusions, overlap restriction, and matched sample.
  • Summarize balance diagnostics without treating thresholds as proof of comparability.
  • Report a matched effect estimate with its scale, units, uncertainty, and analysis population.
  • Describe a sensitivity result without implying that it rules out unmeasured confounding.
  • Write a cautious conclusion that distinguishes an observed association from a causal claim.

From Analysis Output to a Transparent Report

A comparative-study report should let a reader follow the reasoning from the question to the conclusion. It should identify who was compared, how the analytic sample was formed, whether measured covariates were balanced after matching, what effect was estimated and on what scale, and how sensitive the result was to a stated concern. A collection of estimates without that chain is difficult to interpret, even when every calculation is correct.

The report below uses entirely hypothetical data. The question is whether, among eligible adults who began a new treatment, beginning that treatment is associated with a different mean number of hospital days during the following 90 days than beginning a comparator treatment. The comparison uses propensity-score matching on measured baseline characteristics. The report's target is the treated patients retained in the matched sample, not every eligible patient and not necessarily all patients who received the treatment.

Definition: A transparent comparative-study report makes the question, analysis population, design and diagnostics, estimate and uncertainty, sensitivity analysis, and limitations explicit enough for a reader to understand what was compared and what conclusions the evidence supports. In a matched observational study, matching does not by itself establish that treatment groups are exchangeable on unmeasured factors.

Before presenting an estimate, the report should name the target population and the outcome. It should also state the interpretation conditions relevant to a causal reading: treatment and outcome are well-defined; the measured baseline covariates are sufficient to control confounding; patients have a nonzero probability of receiving either treatment in the relevant covariate patterns; and interference between patients does not materially affect outcomes. These are substantive assumptions, not consequences of a good balance table. If they are not credible, the estimate remains a comparison of observed groups rather than a well-identified causal effect.

Worked Example: Make the Cohort Flow Add Up

Worked Example: Make the Cohort Flow Add Up

Suppose a hypothetical registry identifies 1,400 adults: 580 who began the new treatment and 820 who began the comparator. The prespecified eligibility and data-availability rules exclude 20 treated and 40 comparator patients for ineligibility, 20 treated and 30 comparator patients for missing baseline covariates, and 20 treated and 30 comparator patients for unavailable 90-day outcome data. The report should show each step and reconcile the totals.

After these exclusions, the treated group contains \(580-20-20-20=520\) patients. The comparator group contains \(820-40-30-30=720\) patients. Thus the analysis cohort has \(520+720=1{,}240\) patients. These counts should be reported separately by treatment group; a single total would hide whether exclusions were concentrated in one group.

Suppose the prespecified overlap restriction then removes 45 treated patients and 125 comparator patients. The groups entering matching contain \(520-45=475\) treated patients and \(720-125=595\) comparator patients. One-to-one matching without replacement produces 350 pairs, or 350 patients in each group. Therefore, 125 treated patients and 245 comparator patients who entered matching are not retained in the matched analysis.

The report should explain what each exclusion represents and when it occurred. In particular, exclusion for unavailable outcomes can select patients differently across treatment groups; reporting the numbers makes that possibility visible but does not eliminate the concern. The matched estimate will describe the 350 retained treated patients and their matched comparators, subject to the design and causal assumptions—not all 1,400 initially identified patients.

Report Diagnostics as Evidence, Not as Certification

A concise report names the baseline variables used in the propensity-score model, states the matching approach, and presents diagnostics after matching. The diagnostics should include measured covariate balance and the overlap or support relevant to the actual analysis. A standardized mean difference (SMD) expresses a difference in means relative to a standard deviation scale, making it less dependent on sample size than a significance test. It is a diagnostic summary, not a test that certifies absence of confounding.

Worked Example: Show What Changed in Balance

Suppose the mean age before matching is 61.0 years (standard deviation 12.0) among treated patients and 56.0 years (standard deviation 11.8) among comparator patients. Using the average of the two variances as the standardizing variance, the absolute SMD is:

$$ \left|\mathrm{SMD}\right| = \frac{|61.0-56.0|} {\sqrt{(12.0^2+11.8^2)/2}} = \frac{5.0}{\sqrt{(144+139.24)/2}} = \frac{5.0}{11.90} = 0.420. $$

After matching, suppose the respective means are 58.2 and 57.6 years, with standard deviations 12.0 and 11.8. The corresponding SMD is:

$$ \left|\mathrm{SMD}\right| = \frac{|58.2-57.6|} {\sqrt{(12.0^2+11.8^2)/2}} = \frac{0.6}{11.90} = 0.050. $$

This supports the statement that the measured age difference was smaller after matching. A report should give the same kind of diagnostic across the full set of prespecified baseline covariates, not select only favorable variables. If all reported absolute SMDs are below 0.10, say so while still presenting the values or a clear balance display. That conventional benchmark is a useful summary, not a universal guarantee that residual imbalance is harmless or that unmeasured confounding is absent.

Diagnostics should also make limited overlap and exclusions legible. For example, the hypothetical report can state that the propensity-score distributions overlapped in the region used for matching and that 45 treated and 125 comparator patients outside the prespecified support were excluded. It should not imply that overlap among retained patients repairs the lack of comparability for excluded patients. The matched population is narrower because of the design choices.

Worked Example: Report the Estimate and Its Uncertainty

Worked Example: Report the Estimate and Its Uncertainty

Suppose the outcome is hospital days during 90 days, and the matched-pair analysis computes, for each of 350 pairs, treated patient's days minus comparator patient's days. The mean of these paired differences is \(\bar{d}=-1.80\) days, with standard deviation \(s_d=10.47\) days. Negative values indicate fewer hospital days in the treated group.

The standard error of the mean paired difference is the standard deviation of the pair differences divided by the square root of the number of pairs:

$$ SE(\bar{d})=\frac{s_d}{\sqrt{n}} =\frac{10.47}{\sqrt{350}} =\frac{10.47}{18.71} =0.560\text{ days}. $$

Using a two-sided 95% \(t\) interval with \(df=350-1=349\), and \(t^*\approx1.967\), the margin of error is \(1.967(0.560)=1.10\) days. The interval is:

$$ \bar{d}\pm t^*SE = -1.80\pm1.967(0.560) = (-2.90,-0.70)\text{ days}. $$

The test statistic for \(H_0:\mu_d=0\), against a two-sided alternative, is \(t=-1.80/0.560=-3.21\), with 349 degrees of freedom; the two-sided \(p\)-value is approximately 0.0014. The report can say that the matched sample showed a mean difference of 1.80 fewer hospital days for treated patients, with a 95% confidence interval from 2.90 fewer to 0.70 fewer days. The interval describes uncertainty under the stated sampling model and analysis; it is not a 95% probability statement about the fixed parameter.

This calculation is appropriate only if the reported standard error reflects the paired analysis and the pair differences are independent across pairs. The \(t\)-interval relies on the sampling distribution of the mean pair difference being adequately approximated by a \(t\) distribution; with 350 pairs, this approximation is often reasonable, but severe skewness or influential differences still merit examination. If matching or reuse of controls induces dependence across pairs, the uncertainty calculation must account for that design instead of treating the pairs as independent.

Worked Example: Connect the Sensitivity Result to the Main Finding

Worked Example: Connect the Sensitivity Result to the Main Finding

Suppose the prespecified sensitivity analysis for hidden bias reports a two-sided upper-bound \(p\)-value of 0.0014 under \(\Gamma=1\), 0.041 under \(\Gamma=1.3\), and 0.063 under \(\Gamma=1.5\). Here \(\Gamma\) indexes the allowed degree of difference in treatment assignment odds within a matched pair due to an unmeasured factor; \(\Gamma=1\) represents no such hidden treatment-assignment difference under the sensitivity model.

A clear report does not call this a correction for hidden bias or claim that it proves the estimate robust. It states what the analysis says: under the specified sensitivity model, the upper-bound \(p\)-value crosses 0.05 between \(\Gamma=1.3\) and \(\Gamma=1.5\). Thus the conventional statistical-significance conclusion is sensitive to a degree of hidden bias between those values. This does not quantify the actual amount of unmeasured confounding, and it does not determine whether such confounding is plausible in this clinical setting.

The result should appear alongside the main estimate, not as a replacement for it. In this example, the report gives the estimated difference and confidence interval, then describes how the sensitivity result qualifies the strength of the evidence. It should also identify the sensitivity method and its assumptions sufficiently for a reader to understand what \(\Gamma\) represents; a bare statement that “sensitivity analyses were performed” is not informative.

Assemble the Results in a Reader-Friendly Order

A compact report can be organized so that each section answers a different question. The following sequence works well for this hypothetical study:

1
State the question and target.
Define the outcome and say that the estimate pertains to treated patients retained in the matched sample and their matched comparators.
2
Account for the cohort.
Give counts by treatment group at screening, exclusions, overlap restriction, and matching, with reasons for exclusions.
3
Describe the design and diagnostics.
Name the matching approach and baseline covariates, then summarize overlap and post-match balance without treating a threshold as proof.
4
Present the estimate and qualification.
Give the effect scale, estimate, uncertainty, and sensitivity result, then state limitations that affect interpretation.

A possible results paragraph is: “Among 1,240 eligible patients with available baseline covariates and 90-day outcome data, 520 began the new treatment and 720 began the comparator. The prespecified overlap restriction retained 475 treated and 595 comparator patients; one-to-one matching without replacement yielded 350 pairs. Measured baseline balance improved after matching, with the age absolute SMD decreasing from 0.420 to 0.050; the full covariate diagnostics are reported for all prespecified variables. Mean 90-day hospital use was 1.80 days lower in treated than comparator patients in the matched sample (95% CI, 2.90 to 0.70 fewer days; \(p=0.0014\)). Under the stated hidden-bias sensitivity model, the upper-bound \(p\)-value was 0.041 at \(\Gamma=1.3\) and 0.063 at \(\Gamma=1.5\). The result is an adjusted observational comparison and may still reflect unmeasured confounding or selection into the matched sample.”

This paragraph is concise, but it should be accompanied by enough detail to reproduce and assess the analysis: matching specifications, covariate definitions, missing-data and exclusion rules, diagnostics, uncertainty method, and sensitivity-analysis specification. A result paragraph cannot substitute for reproducible materials or a clear table and cohort-flow display.

Limitations, Alternatives, and Common Reporting Errors

The main alternative approaches are not interchangeable versions of the same analysis. A regression-adjusted comparison models the outcome conditional on covariates rather than constructing matched pairs; weighting targets a population defined by its weights; and an unadjusted comparison describes the observed groups without the matching adjustment. Each has different assumptions, diagnostics, and target population implications. A report should name the method actually used and avoid presenting results from multiple approaches as if they estimated exactly the same quantity.

Matching can discard patients and change the population to which an estimate applies. In this example, the matched treated group consists of 350 of the 520 eligible treated patients. The effect estimate therefore does not automatically generalize to treated patients excluded for missing data, limited overlap, or failure to find a match. Further limitations include measurement error in baseline covariates, residual confounding by factors not measured or not adequately modeled, and potential selection bias associated with outcome availability. Statistical precision does not resolve these design limitations.

  • Do not report only the matched sample size. Give the starting numbers and explain exclusions and unmatched patients by treatment group.
  • Do not call balance “proof of no confounding.” Balance diagnostics apply to measured covariates; they cannot establish balance on unmeasured factors.
  • Do not omit the target population. State whether the estimate concerns retained treated patients, an overlap population, or another defined group.
  • Do not use causal wording as a shortcut. “Treatment reduced hospital days” asserts more than this observational estimate alone supports. Prefer “the matched comparison showed fewer hospital days” unless the causal assumptions are defended.
  • Do not report a sensitivity analysis as a pass/fail test. Explain the sensitivity parameter, the result it qualifies, and what the analysis cannot establish.
Key takeaway: A transparent report ties every numerical result to the patients it describes and the assumptions it requires. Matching can improve measured comparability, but interpretation still depends on the analysis population, design diagnostics, uncertainty calculation, and the plausibility of assumptions about unmeasured confounding.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

  1. In the hypothetical example, how many treated and comparator patients entered matching, and how many pairs were retained?
  2. Why does an absolute SMD of 0.050 for age not establish that the matched groups are free of confounding?
  3. What does the 95% interval from 2.90 fewer to 0.70 fewer hospital days describe in context?
  4. How should the sensitivity results at \(\Gamma=1.3\) and \(\Gamma=1.5\) qualify the main finding?
  5. Why should the report avoid generalizing the estimate to all 520 eligible treated patients without qualification?

Solutions. (1) There were 475 treated and 595 comparator patients entering matching; 350 pairs, containing 350 patients in each group, were retained. (2) The SMD summarizes measured age balance only and cannot diagnose balance on unmeasured factors or establish the causal assumptions. (3) Under the stated paired-analysis model, the data are compatible with a mean difference in the matched population ranging from 2.90 fewer to 0.70 fewer hospital days for treated patients; it is not a probability that the fixed effect lies in that interval. (4) The upper-bound \(p\)-value is below 0.05 at \(\Gamma=1.3\) and above 0.05 at \(\Gamma=1.5\), so the conventional significance conclusion is sensitive within that range under the specified model; this does not show how much hidden bias actually exists. (5) Only 350 of the 520 eligible treated patients were retained, and exclusions or lack of matches may affect who is represented and how applicable the estimate is.

Previous

Next

Back to Tutorials
Clinical Biostats