Publication Bias and Funnel Plots

Introduction

A meta-analysis combines results from multiple studies to estimate an overall effect. This approach can substantially improve precision, but the validity of the pooled estimate depends partly on which studies are available to be included.

Ideally, all relevant studies would have an equal opportunity to become available regardless of whether their findings were statistically significant, clinically exciting, or consistent with expectations.

In practice, studies with statistically significant or otherwise favorable results may be more likely to be published, published rapidly, or published in prominent journals.

Studies with null, unfavorable, or inconclusive findings may remain unpublished or become more difficult to identify.

This phenomenon can create a distorted evidence base.

What Is Publication Bias?

Publication bias occurs when the probability that a study becomes available for review depends on characteristics of its findings.

For example, imagine ten clinical trials investigating the same treatment. Suppose five show little or no treatment effect and five show a favorable effect.

If the favorable studies are much more likely to be published, a systematic review based only on published studies may contain a disproportionate number of positive findings.

The resulting meta-analysis could therefore overestimate the treatment effect.

A Simple Conceptual Example

If the systematic review identifies only the first three studies, the pooled result will not represent the complete set of conducted studies.

This is one reason systematic reviewers attempt to identify unpublished and grey literature, clinical trial registrations, conference abstracts, and other sources of evidence when appropriate.

Publication Bias Is Not the Same as Selective Reporting

Several forms of reporting bias can affect a systematic review.

A funnel plot primarily concerns the relationship between estimated effect size and study precision. It does not directly diagnose every form of reporting bias.

What Is a Funnel Plot?

A funnel plot is a scatter plot in which each point generally represents one study in a meta-analysis.

The horizontal axis usually displays the estimated treatment effect.

The vertical axis displays a measure of study precision, commonly the standard error, with more precise studies appearing toward the top.

Because larger studies tend to have smaller standard errors, the top of the plot often corresponds approximately to larger studies and the bottom to smaller studies.

Under a simple idealized situation, study estimates should scatter around the overall effect, with greater dispersion among less precise studies.

This produces a roughly funnel-shaped distribution.

The Basic Funnel-Plot Geometry

Suppose the pooled effect estimate is:

Studies with small standard errors should tend to cluster relatively close to the pooled effect.

Studies with larger standard errors can show more sampling variation and may therefore be scattered farther to either side.

The expected shape can be represented conceptually by:

where \(SE\) is the standard error of the study-specific effect estimate.

An Actual Funnel Plot

The following figure uses simulated studies to illustrate an approximately symmetric funnel.

What Does Each Point Represent?

Each point generally corresponds to one independent study included in the meta-analysis.

For example, if a meta-analysis contains 18 eligible studies, the funnel plot will generally contain approximately 18 study-level points.

The point's horizontal position represents the study's effect estimate.

Its vertical position represents its precision.

Why Are Small Studies More Variable?

Sampling variability is generally greater in smaller studies.

Consequently, a small study may produce an estimate that is substantially above or below the underlying effect even when there is no bias.

For an estimated effect \(\hat{\theta}\), the standard error quantifies the uncertainty of that estimate.

A smaller standard error means greater precision.

A larger standard error means less precision.

Standard Error as the Vertical Axis

A conventional funnel plot often uses standard error on the vertical axis, with the axis inverted so that small standard errors appear toward the top.

For example:

would typically be arranged from most precise at the top to least precise at the bottom.

What Would Publication Bias Look Like?

One commonly discussed scenario is that small studies with unfavorable results are missing from the literature.

Suppose the favorable direction is toward the left.

If small studies with unfavorable results are less likely to be published, the bottom-left and bottom-right sides of the funnel may contain different numbers of studies.

The resulting distribution can appear asymmetric.

An Asymmetric Funnel Plot

Why Asymmetry Is Not Proof of Publication Bias

This is the single most important concept when interpreting funnel plots.

A funnel plot can be asymmetric for many reasons other than publication bias.

Potential explanations include:

• True heterogeneity between studies
• Differences in study populations
• Differences in treatment intensity
• Differences in methodological quality
• Small-study effects
• Chance
• Outcome measurement differences
• Selective reporting
• Publication bias

What Are Small-Study Effects?

The term small-study effects refers to a situation in which smaller studies tend to produce systematically different effect estimates than larger studies.

Publication bias is one possible explanation.

But it is not the only one.

For example, small studies may:

• Use different patient populations
• Be conducted in different settings
• Use less rigorous methods
• Study different treatment doses
• Use different outcome definitions
• Be more susceptible to random error

Consequently, funnel-plot asymmetry is often more accurately described as evidence of a possible small-study effect rather than as automatic evidence of publication bias.

Publication Bias Versus Small-Study Effects

These concepts overlap but should not be treated as interchangeable.

The Relationship Between Effect Size and Standard Error

A conventional funnel plot can be represented as:

where:

• \(\hat{\theta}_i\) is the effect estimate for study \(i\)
• \(SE_i\) is the standard error of that estimate

The plot therefore examines whether effect estimates systematically vary with precision.

Confidence Intervals and the Funnel Shape

An approximate 95% confidence interval for study \(i\) is:

As \(SE_i\) increases, the confidence interval becomes wider.

This produces the characteristic funnel geometry.

The funnel boundaries are therefore not themselves statistical confidence intervals for the pooled effect.

Egger's Regression Test

One of the most widely used statistical approaches for assessing funnel asymmetry is Egger's regression test.

The method examines whether standardized effect estimates are associated with their precision.

A commonly presented formulation is:

The intercept \(\beta_0\) is used to assess asymmetry.

Under a simplified interpretation, an intercept close to zero is consistent with symmetry, whereas a statistically unusual intercept suggests possible funnel asymmetry.

Understanding the Egger Intercept

Suppose a meta-analysis produces:

with a p-value:

This would provide statistical evidence against the null hypothesis of a zero intercept under the assumptions of the test.

However, it should not be translated directly into:

The appropriate interpretation is that there is evidence of funnel asymmetry, which may have several possible explanations.

Egger's Test: What It Tests

Why the Number of Studies Matters

When a meta-analysis contains only a small number of studies, funnel plots are difficult to interpret.

With five studies, for example, apparent asymmetry may simply reflect random variation.

Likewise, regression-based tests can have low statistical power when few studies are available.

Begg's Rank Correlation Test

Begg's test provides another approach to examining possible funnel asymmetry.

It is based on a rank correlation between effect estimates and their variances or related measures of study precision.

Conceptually, it asks whether study size or precision is systematically related to the magnitude of the observed effects.

Begg's test and Egger's test are therefore related but use different statistical approaches.

Egger Versus Begg

Contour-Enhanced Funnel Plots

A contour-enhanced funnel plot adds regions corresponding to different levels of statistical significance.

For example, contours may indicate:

• Not statistically significant
• p < 0.10
• p < 0.05
• p < 0.01

These contours can help investigators evaluate whether the apparent missing studies would predominantly fall in statistically nonsignificant regions.

Why Contours Can Be Informative

Suppose the apparently empty side of the funnel consists primarily of a region where studies would have nonsignificant results.

This may be consistent with a mechanism in which statistically significant results are more likely to become available.

In contrast, if the empty region is largely statistically significant, simple publication bias based solely on statistical significance may be a less convincing explanation.

Trim-and-Fill

The trim-and-fill method is another commonly discussed approach in the context of funnel-plot asymmetry.

The basic idea is to:

1. Estimate the degree of funnel asymmetry.
2. Temporarily remove or "trim" studies contributing to the asymmetry.
3. Estimate where potentially missing studies might lie.
4. Add or "fill" those hypothetical studies.
5. Recalculate an adjusted pooled estimate.

The method can therefore provide a sensitivity analysis showing how the pooled estimate might change under a particular model of missing studies.

Trim-and-Fill Does Not Recover Actual Missing Studies

This point is essential.

A filled point in a trim-and-fill funnel plot is not an identified unpublished clinical trial.

It is a model-based imputed study.

Example of Trim-and-Fill Interpretation

Suppose the observed meta-analysis estimates:

A trim-and-fill analysis might produce an adjusted estimate:

This suggests that the pooled estimate could be sensitive to the assumed asymmetry.

It does not demonstrate that the true effect is exactly 0.30.

Instead, it provides a model-based sensitivity analysis.

Why Heterogeneity Matters

Heterogeneity is one of the most important alternative explanations for funnel-plot asymmetry.

Suppose small studies investigate a different patient population than large studies.

Their true treatment effects could therefore differ.

A funnel plot may appear asymmetric even if every study was published regardless of its result.

This is why funnel-plot interpretation should be integrated with the broader heterogeneity assessment.

Example: Genuine Clinical Heterogeneity

Imagine a meta-analysis of an intervention across:

• Highly selected specialist populations
• General hospital populations
• Community settings
• Different disease severities

If smaller studies disproportionately enroll severe patients, their effect estimates may systematically differ from those of larger studies.

The resulting funnel asymmetry could be driven by clinical differences rather than publication bias.

Heterogeneity and I-squared

The \(I^2\) statistic is often used to describe the proportion of observed variation that is attributed to between-study heterogeneity rather than sampling error under a particular model.

A high \(I^2\) value does not prove publication bias is absent or present.

Instead, substantial heterogeneity is one reason to be cautious when interpreting funnel asymmetry.

Funnel Plots Under Random-Effects Models

Funnel plots can be used with random-effects meta-analysis, but interpretation requires care.

Under a random-effects model, observed study estimates vary because of both:

• Within-study sampling error
• Between-study heterogeneity

Therefore, a wide spread of study estimates is not automatically evidence of publication bias.

The assumed relationship between precision and effect estimates should be considered in light of the heterogeneity structure.

Fixed-Effect Versus Random-Effects Interpretation

Other Causes of Funnel Asymmetry

Funnel asymmetry can arise from several mechanisms.

True Small-Study Effects

Smaller studies may genuinely estimate different effects because they examine different populations, settings, or interventions.

Methodological Differences

Small studies may use less rigorous randomization, blinding, allocation concealment, or outcome assessment.

Selective Reporting

Studies may measure several outcomes but preferentially report the most favorable ones.

Publication Bias

Studies with unfavorable or nonsignificant results may be less likely to become available.

Chance

With a modest number of studies, apparent asymmetry can occur simply because of random sampling variation.

Small-Study Effects and Effect Direction

Suppose larger studies tend to estimate an odds ratio near:

while small studies tend to estimate:

The funnel plot may show small studies displaced toward larger effects.

This is evidence of a small-study effect.

It is not enough to determine why the difference exists.

Effect Measures Matter

Funnel plots can be constructed for different effect measures, including:

• Mean difference
• Standardized mean difference
• Risk ratio
• Odds ratio
• Hazard ratio
• Risk difference
• Correlation coefficients

The statistical behavior of the effect measure matters when selecting an appropriate asymmetry assessment.

Log Odds Ratios

For odds ratios, analyses commonly use the natural logarithm:

This is useful because the log odds ratio is more naturally suited to meta-analytic modeling.

For example:

corresponds to:

A funnel plot may therefore display log odds ratios rather than raw odds ratios.

Standard Error of a Log Odds Ratio

For a conventional \(2\times2\) table with cell counts \(a,b,c,d\), an approximate standard error is:

The exact calculation may depend on the statistical method and corrections used for sparse or zero-event data.

Funnel Plots for Mean Differences

For a continuous outcome, the study effect may be:

The corresponding standard error is used for the vertical position in the funnel plot.

For standardized mean differences, the effect is standardized by a measure of within-study variability.

What Is a Symmetric Funnel?

A visually symmetric funnel generally means that study estimates become more widely dispersed as precision decreases, approximately equally in both directions around the pooled effect.

This is compatible with a number of possibilities:

• No important small-study effect
• No detectable funnel asymmetry
• Publication bias too small to detect
• Insufficient power to detect asymmetry
• Other biases that happen not to produce visible asymmetry

Therefore:

What Is an Asymmetric Funnel?

An asymmetric funnel occurs when the distribution of study estimates differs systematically from the approximate symmetric pattern expected under the reference model.

For example, there may be:

• Too few small studies on one side
• Small studies concentrated toward one effect direction
• A gap in one region of the funnel
• A systematic relationship between effect size and precision

Interpreting the Bottom of the Funnel

The lower portion of the funnel deserves particular attention because it contains less precise studies.

If these studies show unusually large effects in one direction, several explanations are possible.

For example:

• Random variation
• Small-study effects
• Selective publication
• Clinical differences
• Methodological differences

The pattern should therefore be investigated rather than automatically labeled as publication bias.

Publication Bias Can Affect the Pooled Estimate

Suppose the available literature contains predominantly favorable studies.

The pooled estimate may be shifted toward the favorable direction.

Conceptually:

when larger values represent more favorable effects.

The exact magnitude of the distortion cannot generally be determined from a funnel plot alone.

Why a Funnel Plot Is an Exploratory Tool

A funnel plot does not tell the investigator:

• Which studies are missing
• Why they are missing
• Whether missing studies actually exist
• Whether missing studies would have null results
• How much the pooled estimate is biased

It provides evidence that can motivate further investigation.

Searching for Unpublished Evidence

When publication bias is a concern, statistical diagnostics should be combined with evidence-retrieval strategies.

These may include:

• Clinical trial registries
• Regulatory documents
• Conference abstracts
• Dissertations and theses
• Grey literature
• Contact with investigators
• Reference-list searching
• Citation searching

This is often more informative than relying exclusively on statistical tests for funnel asymmetry.

Publication Bias and the Search Strategy

The risk of publication bias begins before statistical analysis.

A narrow search strategy may preferentially identify published journal articles while missing:

• Unpublished trials
• Negative studies
• Older studies
• Non-English reports
• Conference proceedings
• Registered but unpublished studies

Consequently, preventing publication bias is partly a systematic-review design problem rather than simply a statistical testing problem.

Funnel Plot Workflow

When Should a Funnel Plot Be Used?

Funnel plots are most useful when there are enough studies for meaningful inspection of the distribution.

They are generally less informative when the meta-analysis contains very few studies.

The decision should also consider:

• The effect measure
• Study design
• Degree of heterogeneity
• Study-size distribution
• Outcome characteristics
• Potential causes of small-study effects

A Funnel Plot With Very Few Studies

Imagine a meta-analysis with only six studies.

The observed points might look asymmetric simply because one study has an unusually large estimate.

Removing or adding a single study could substantially change the visual appearance.

Funnel Plot Versus Forest Plot

Funnel Plot Versus Forest Plot: Different Questions

A forest plot primarily asks:

A funnel plot asks:

Both plots are useful, but they serve different purposes.

Funnel Plot Data Structure

A basic funnel-plot dataset requires at least:

The exact variables depend on the effect measure and meta-analysis software.

Study Precision

The standard error is one of the most common measures of precision.

Precision can be defined conceptually as:

Therefore:

• Small \(SE\) → high precision
• Large \(SE\) → low precision

Some funnel plots use \(1/SE\) instead of \(SE\), reversing the vertical-axis orientation.

Example Study-Level Dataset

Study       TE       SE
Study 01    0.22     0.08
Study 02    0.31     0.10
Study 03    0.18     0.12
Study 04    0.47     0.15
Study 05    0.55     0.18
Study 06    0.26     0.07
Study 07    0.12     0.20
Study 08    0.61     0.23
Study 09    0.33     0.09
Study 10    0.44     0.14

A funnel plot uses the effect estimate and standard error rather than the individual participant-level observations.

Creating a Funnel Plot in R

The metafor package provides flexible tools for meta-analysis and funnel plots.

library(metafor)

res <- rma(
  yi = TE,
  sei = SE,
  data = data,
  method = "REML"
)

funnel(
  res,
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

Using the meta Package

The meta package provides another commonly used workflow.

library(meta)

m <- metagen(
  TE,
  seTE,
  studlab = Study,
  data = data,
  sm = "MD",
  random = TRUE
)

funnel(
  m,
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

Egger's Test in R

With metafor, a regression-based test for funnel asymmetry can be performed using:

regtest(
  res,
  model = "rma"
)

The output should be interpreted as a test for funnel asymmetry rather than as a definitive test for publication bias.

Begg's Test in R

A rank-correlation approach can be performed using:

ranktest(
  res
)

Again, the resulting p-value should be interpreted in the context of the number of studies and other evidence regarding possible small-study effects.

Contour-Enhanced Funnel Plot in R

The meta package can also be used to produce contour-enhanced funnel plots.

funnel(
  m,
  contour = c(0.90, 0.95, 0.99),
  col.contour = c(
    "white",
    "gray90",
    "gray75"
  ),
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

The exact graphical syntax depends on the package version and effect measure. The important principle is to display statistical-significance contours and interpret them alongside the study distribution.

Trim-and-Fill in R

A trim-and-fill analysis can be performed using:

taf <- trimfill(res)

summary(taf)

funnel(
  taf,
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

The adjusted result should be presented as a sensitivity analysis rather than as a replacement for the primary meta-analysis.

Comparing the Original and Adjusted Estimates

Suppose:

and trim-and-fill produces:

The difference can be summarized as:

This indicates sensitivity of the estimated pooled effect to the assumed missing-study mechanism.

It does not quantify the actual amount of publication bias.

Do Not Automatically Use Trim-and-Fill as a Correction

A common mistake is to treat the trim-and-fill estimate as the "corrected" truth.

That interpretation is too strong.

The method depends on assumptions about the source and structure of funnel asymmetry.

If asymmetry is actually caused by clinical heterogeneity, methodological differences, or another mechanism, the imputed studies may not represent the missing evidence.

What If Egger's Test Is Significant?

Suppose:

for Egger's test.

The appropriate conclusion is that the analysis provides evidence of funnel asymmetry.

The next questions should include:

• Are small studies systematically different?
• Is there substantial heterogeneity?
• Do study designs differ?
• Do patient populations differ?
• Are outcome definitions consistent?
• Are there reasons to suspect selective publication?
• Could methodological quality explain the pattern?

What If Egger's Test Is Not Significant?

Suppose:

This means the analysis does not provide strong statistical evidence of funnel asymmetry under the test's assumptions.

It does not prove that publication bias is absent.

A nonsignificant result may occur because:

• There is no important asymmetry
• The test has low power
• There are too few studies
• The asymmetry is not detectable using that method

P-Values Should Not Be Used in Isolation

The funnel plot, asymmetry test, literature search, and clinical knowledge should be interpreted together.

For example:

Why Visual and Statistical Evidence Can Disagree

A human observer may see an asymmetric pattern that does not produce a statistically significant test.

Conversely, a formal test may identify asymmetry that is difficult to recognize visually.

This can occur because:

• The number of studies is small
• The distribution is irregular
• Study weights differ substantially
• Outliers influence the analysis
• The test detects a subtle association

Outliers and Funnel Plots

A single small study with an extreme effect can substantially influence the visual impression.

Suppose nine studies cluster near:

while one small study has:

The resulting plot may look strongly asymmetric.

The investigator should therefore examine the individual study rather than immediately concluding that publication bias exists.

Sensitivity Analysis

One useful approach is to evaluate how the pooled estimate changes when influential studies are examined.

For example:

leave1out(res)

can be used in metafor to examine the influence of individual studies.

This does not solve publication bias, but it can identify whether a particular study is driving the overall result or asymmetry.

Influence Diagnostics

Additional influence diagnostics may include:

• Leave-one-out analyses
• Cook's distance
• Studentized residuals
• Hat values
• Influence plots

These diagnostics answer different questions from funnel plots and can be useful when a study appears unusual.

Publication Bias and Meta-Regression

Meta-regression can investigate whether effect estimates vary systematically with study-level characteristics.

For example:

• Sample size
• Mean age
• Study year
• Treatment dose
• Study quality
• Follow-up duration

If effect size is strongly associated with study size, this provides evidence of a small-study effect, but again does not establish the mechanism.

rma(
  yi = TE,
  sei = SE,
  mods = ~ log(N),
  data = data,
  method = "REML"
)

Study Year and Small-Study Effects

Publication practices can change over time.

Older studies may differ from newer studies in:

• Diagnostic criteria
• Standard of care
• Study design
• Patient selection
• Outcome measurement

Consequently, a relationship between study size and effect can sometimes be confounded by calendar time or other study characteristics.

Funnel Plots and Clinical Trial Registries

Trial registries can provide information about studies that may not appear in the published literature.

For each potentially relevant study, reviewers can compare:

• Registered outcomes
• Protocol-defined endpoints
• Actual published outcomes
• Publication status
• Sample size
• Study completion status

This can provide more direct evidence about selective availability or reporting than funnel asymmetry alone.

What a Funnel Plot Cannot Detect Reliably

A funnel plot is not designed to detect every form of bias.

It is particularly limited for identifying:

• Selective outcome reporting
• Selective analysis reporting
• Confounding
• Measurement bias
• Risk-of-bias problems unrelated to study size
• Incorrect eligibility decisions
• Data extraction errors

Funnel Plots and Risk of Bias

Risk-of-bias assessment should be performed separately.

For randomized trials, reviewers may assess domains such as:

• Randomization process
• Allocation concealment
• Deviations from intended interventions
• Missing outcome data
• Outcome measurement
• Selective reporting

A study can be high risk of bias without being an obvious outlier on a funnel plot.

Conversely, a study can contribute to funnel asymmetry without necessarily being at high risk of bias.

Publication Bias in Diagnostic Meta-Analysis

Diagnostic test accuracy meta-analysis requires additional caution because effect estimates often involve sensitivity and specificity or transformed measures.

Traditional funnel-plot approaches may not be directly appropriate for every diagnostic meta-analysis.

Specialized methods may be required depending on:

• The bivariate model
• Hierarchical summary ROC modeling
• Threshold effects
• Study design
• Effect measure

Publication Bias in Network Meta-Analysis

Network meta-analysis introduces additional complexity because evidence comes from multiple treatment comparisons.

Small-study effects may differ between comparisons.

Consequently, a single conventional funnel plot may not fully describe the structure of potential bias.

Comparison-adjusted approaches or network-specific diagnostics may be more appropriate in some settings.

Common Mistakes

1. Calling funnel asymmetry proof of publication bias. Asymmetry has multiple possible explanations.
2. Assuming a symmetric funnel proves no publication bias. Absence of detectable asymmetry is not evidence that all relevant studies were identified.
3. Ignoring heterogeneity. Clinical and methodological differences can produce asymmetry.
4. Running Egger's test automatically with very few studies. The test can have limited power and unstable behavior.
5. Treating a nonsignificant p-value as proof of no bias. A nonsignificant test does not establish absence of publication bias.
6. Using trim-and-fill as a definitive correction. Imputed studies are model-based and hypothetical.
7. Confusing small-study effects with publication bias. Small studies can differ for reasons unrelated to selective publication.
8. Ignoring the effect measure. Funnel methods depend on the statistical properties of the outcome measure.
9. Ignoring unpublished evidence. Statistical diagnostics do not replace a comprehensive search.
10. Reporting only a p-value. The visual plot, effect estimates, heterogeneity, study characteristics, and clinical context should also be considered.

A Practical Interpretation Framework

Example Interpretation

Suppose a meta-analysis contains 24 studies.

The funnel plot appears asymmetric, with small studies tending to show larger beneficial effects.

Egger's test produces:

The investigators also observe substantial clinical heterogeneity.

A responsible interpretation would be:

Example of a Poor Interpretation

There are two problems with this statement.

First, Egger's test evaluates funnel asymmetry rather than directly proving the cause of that asymmetry.

Second, trim-and-fill does not identify actual missing studies or definitively correct the evidence base.

Example of Better Reporting

A more appropriate statement is:

How to Report a Funnel Plot in a Manuscript

A publication-bias section should generally describe:

• Whether a funnel plot was examined
• The number of studies included
• The method used to assess asymmetry
• The test result when appropriate
• Potential alternative explanations
• Any sensitivity analysis

Example Reporting Table

The disagreement among diagnostics is not necessarily a contradiction. Different methods have different assumptions and statistical properties.

Why Different Tests Can Give Different Results

Suppose Egger's test is significant but Begg's test is not.

This can occur because:

• The tests use different statistical approaches
• The tests have different statistical power
• The data contain influential studies
• The distribution is not well characterized by either method
• The number of studies is limited

Rather than selecting whichever test gives the preferred conclusion, the discrepancy should be reported and investigated.

Funnel Plot Quality Control

Before publishing a funnel plot, verify:

• Every plotted point corresponds to the intended analysis dataset.
• The correct effect measure is used.
• The standard errors are correct.
• The pooled estimate is correct.
• The vertical-axis direction is clearly labeled.
• The horizontal-axis transformation is correct.
• Duplicate studies are not included accidentally.
• Multiple reports of the same study have been handled correctly.
• Outliers have been investigated.
• The figure corresponds to the prespecified analysis.

Duplicate Reports Can Distort the Funnel

A particularly important issue in systematic reviews is multiple publications from the same underlying study.

If several reports are incorrectly treated as independent studies, the funnel plot can contain duplicate evidence.

This can affect:

• Study counts
• Standard errors
• Pooled estimates
• Apparent asymmetry
• Formal asymmetry tests

Multiple Outcomes and Multiple Effect Estimates

A single study may report several outcomes or several time points.

If these estimates are treated as independent without an appropriate model, the funnel plot can be distorted.

The meta-analytic unit of analysis must therefore be defined before constructing the plot.

Transformation of Effect Measures

The effect measure should usually be plotted on the scale used for the meta-analysis.

For example, odds ratios are often analyzed on the log scale:

Plotting raw odds ratios when the model operates on log odds ratios can change the visual geometry and complicate interpretation.

Funnel Plots and Zero Events

Binary-outcome meta-analyses may contain studies with zero events in one or both arms.

The handling of zero events can influence the estimated effect and its standard error.

Potential methods include:

• Continuity corrections
• Exact methods
• Generalized linear models
• Alternative likelihood approaches
• Specialized sparse-data methods

The funnel plot should reflect the same effect-estimation method used in the primary analysis.

Funnel Plots and Publication Bias in Clinical Research

Publication bias can be especially important when evidence synthesis is based on relatively small numbers of trials.

For example, an early evidence base for a new intervention may contain only a handful of studies.

If the studies with favorable findings are preferentially published, the pooled estimate can appear larger than it would after complete evidence ascertainment.

This is one reason systematic reviewers should document the search strategy, study-selection process, and attempts to identify unpublished evidence.

Publication Bias Is a Systematic-Review Problem

Statistical diagnostics occur near the end of the evidence-synthesis process.

The more fundamental strategy is to minimize the possibility of missing evidence in the first place.

Publication Bias and PRISMA Reporting

A systematic review should transparently describe how studies were identified, screened, included, and analyzed.

The funnel plot is only one component of transparent reporting.

The review should also describe:

• Search databases
• Search dates
• Eligibility criteria
• Study-selection procedures
• Risk-of-bias assessment
• Data extraction
• Statistical methods
• Missing-data handling
• Sensitivity analyses

Publication Bias and GRADE

When evidence is assessed using GRADE or a related framework, publication bias is one consideration among several factors affecting certainty of evidence.

Other considerations may include:

• Risk of bias
• Inconsistency
• Indirectness
• Imprecision
• Publication bias

A funnel plot therefore contributes to a broader evidence-certainty assessment rather than serving as the sole basis for downgrading evidence.

What If the Funnel Is Strongly Asymmetric?

Strong asymmetry warrants investigation.

A useful sequence is:

1. Verify the dataset.
2. Check for duplicate studies.
3. Investigate outliers.
4. Review study characteristics.
5. Assess heterogeneity.
6. Review risk-of-bias domains.
7. Examine publication and trial-registration history.
8. Perform appropriate sensitivity analyses.

The objective is to determine which explanation is most plausible.

What If the Funnel Is Symmetric?

A symmetric funnel is reassuring in a limited sense.

It suggests that the observed studies do not show obvious funnel asymmetry.

But it should not be interpreted as evidence that the evidence base is complete.

Publication bias may exist without producing a recognizable funnel pattern.

What If There Are Only Three Studies?

A funnel plot can technically be drawn with three studies.

But its interpretability is extremely limited.

There are simply too few observations to characterize the distribution reliably.

Formal asymmetry testing is also unlikely to provide useful evidence.

How Many Studies Are Enough?

There is no universal number that guarantees a meaningful funnel-plot assessment.

Rules of thumb are sometimes used, but appropriateness depends on the effect measure, heterogeneity, study-size distribution, and research context.

In practice, asymmetry tests are often considered more informative when the meta-analysis contains a reasonable number of studies rather than only a few.

The important point is to avoid mechanically applying a cutoff without considering the analysis.

Publication Bias and Time-to-Event Outcomes

For survival outcomes, hazard ratios are commonly meta-analyzed.

The funnel plot may use:

with its corresponding standard error.

As with other effect measures, the plot examines the relationship between effect estimates and precision rather than directly identifying unpublished studies.

Publication Bias and Continuous Outcomes

For continuous outcomes, standardized mean differences may be used when studies measure the same underlying construct using different scales.

For example:

The corresponding standard error determines the vertical position in a conventional funnel plot.

Publication Bias and Binary Outcomes

For binary outcomes, common effect measures include:

• Risk ratio
• Odds ratio
• Risk difference

The choice should be specified before analysis and used consistently when constructing the funnel plot.

A More Complete Funnel-Plot Interpretation

Consider a funnel plot in which small studies tend to show larger treatment effects.

Instead of writing:

the investigator should consider the broader hypothesis:

The purpose of the diagnostic is to identify a pattern that requires investigation.

Recommended Analysis Sequence

Clinical Interpretation Example

Imagine a meta-analysis of a new cancer treatment.

Large studies estimate a hazard ratio around:

Several small studies estimate:

The funnel plot is asymmetric.

Possible explanations include:

• Selective publication of favorable small studies
• Differences in cancer stage
• Different treatment lines
• Different follow-up duration
• Differences in trial quality
• Random error

A careful meta-analysis would investigate these explanations before concluding that publication bias is responsible.

How Publication Bias Can Change Clinical Conclusions

Suppose the published literature suggests:

which appears to indicate a 30% relative reduction in risk.

If unpublished null studies exist, the complete evidence base might produce:

The clinical interpretation could therefore change substantially.

This illustrates why publication bias is not merely a technical statistical issue.

Absolute Effects Still Matter

Even when publication bias affects a relative effect estimate, clinical interpretation should consider absolute risk.

For example, a relative risk of:

could correspond to very different absolute benefits depending on baseline risk.

Therefore, a publication-bias assessment should be considered alongside the clinical magnitude and certainty of the evidence.

Publication Bias in Safety Meta-Analysis

Publication bias can also affect adverse-event evidence.

For example, studies with unfavorable safety findings might be less likely to be published or might emphasize efficacy results over safety outcomes.

Consequently, reviewers should consider:

• Published safety outcomes
• Trial registries
• Regulatory data where available
• Study protocols
• Selective outcome reporting

A conventional funnel plot cannot fully resolve these issues.

Publication Bias in Oncology Evidence Synthesis

In oncology, publication bias can affect evidence concerning:

• Overall survival
• Progression-free survival
• Objective response
• Adverse events
• Quality of life
• Biomarker-defined treatment effects

Different endpoints may have different reporting probabilities, making endpoint-specific investigation important.

Funnel Plots and Subgroup Analyses

If a meta-analysis contains clinically distinct subgroups, combining all studies into a single funnel plot can conceal meaningful structure.

For example, studies may differ by:

• Disease stage
• Age group
• Treatment line
• Follow-up duration
• Geographic region

If justified, subgroup-specific analyses may help investigate whether asymmetry is associated with a particular study characteristic.

Funnel Plots and Meta-Analytic Model Choice

The funnel plot should be consistent with the effect estimates and standard errors used in the meta-analysis.

Changing the statistical model can change:

• The pooled estimate
• Study weights
• Reference lines
• Interpretation of asymmetry

Therefore, the figure should be generated from the same analysis specification used for the corresponding meta-analysis unless the purpose is explicitly a sensitivity analysis.

Funnel Plot Specification

Funnel Plot Quality-Control Checklist

Manual Funnel-Plot Validation

Suppose the pooled effect is:

and one study has:

The study should be plotted at:

If the vertical axis is inverted so that smaller standard errors are higher, this study should appear relatively high in the plot.

Approximate Funnel Boundaries

For a 95% reference region, the approximate boundaries around the pooled effect can be expressed as:

If:

the approximate limits are:

and:

Thus, a study with \(SE=0.20\) and an estimate between approximately −0.09 and 0.69 would fall within this simple reference region.

Why the Funnel Boundaries Narrow at the Top

As standard error approaches zero:

so the boundaries converge toward the pooled estimate.

This is why highly precise studies cluster tightly around the center of the funnel.

Why the Funnel Widens at the Bottom

As standard error increases:

also increases.

Less precise studies can therefore have substantially larger deviations from the pooled effect simply because of sampling variation.

A Second Actual Funnel Plot: Missing Small Studies

The following simulation illustrates the conceptual pattern that can occur when small studies on one side of the pooled estimate are absent.

Interpreting the Four Funnel-Plot Scenarios

What Should Go in the Figure Footnote?

A useful funnel-plot footnote might state:

Reporting Egger's Test

A concise report might state:

Reporting a Nonsignificant Test

Reporting Trim-and-Fill

Publication Bias Checklist

The Most Important Statistical Distinction

Three statements should be kept separate:

They are related, but they are not synonyms.

The logical relationship is closer to:

but the reverse implications do not necessarily hold.

Why the Distinction Matters

If every asymmetric funnel were automatically interpreted as publication bias, investigators could incorrectly attribute genuine clinical heterogeneity to selective publication.

Conversely, if every symmetric funnel were interpreted as proof of an unbiased literature, investigators could overlook publication bias that does not create a detectable funnel pattern.

The correct approach is therefore cautious triangulation.

Triangulating Evidence

A strong assessment combines:

• Search completeness
• Study-selection procedures
• Risk-of-bias assessment
• Funnel-plot inspection
• Formal asymmetry testing
• Heterogeneity assessment
• Study-level characteristics
• Trial-registration information
• Sensitivity analyses

No single diagnostic should be treated as definitive.

Practical R Workflow

library(metafor)

# Primary random-effects meta-analysis
res <- rma(
  yi = TE,
  sei = SE,
  data = data,
  method = "REML"
)

# Forest plot
forest(res)

# Funnel plot
funnel(
  res,
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

# Egger-type regression test
regtest(
  res,
  model = "rma"
)

# Begg-type rank correlation test
ranktest(
  res
)

# Trim-and-fill sensitivity analysis
taf <- trimfill(res)

summary(taf)

# Funnel plot after trim-and-fill
funnel(
  taf,
  xlab = "Effect Estimate",
  ylab = "Standard Error"
)

Interpreting the R Output

The key outputs to inspect include:

• Pooled effect estimate
• Confidence interval
• Between-study heterogeneity
• Egger intercept and p-value
• Begg test result
• Number of trim-and-fill imputed studies
• Difference between primary and sensitivity estimates

These should be interpreted collectively.

Example Analysis Summary

The overall interpretation would be that the evidence shows possible small-study effects and that the pooled estimate is sensitive to assumptions about missing studies.

It would not be appropriate to state that publication bias has been proven.

Clinical Study Report and Evidence-Synthesis Reporting

A publication-bias assessment should be reported transparently enough for a reader to understand:

• Which studies were assessed
• Which effect measure was used
• How the funnel plot was constructed
• Whether asymmetry was observed
• Whether a formal test was performed
• What the test showed
• What alternative explanations were considered
• Whether sensitivity analyses were performed

Suggested Statistical Methods Language

What Not to Do

• Do not call a significant Egger test proof of publication bias.
• Do not call a nonsignificant Egger test proof that publication bias is absent.
• Do not treat trim-and-fill imputations as actual studies.
• Do not ignore heterogeneity.
• Do not use a funnel plot mechanically when only a few studies are available.
• Do not interpret every small study as inherently biased.
• Do not remove asymmetric studies simply to make the funnel look symmetric.
• Do not choose the statistical method based on which produces the most favorable result.

A Decision Framework

Key Takeaways

A funnel plot is one of the most recognizable graphical diagnostics in meta-analysis.

Its basic structure is simple:

But its interpretation requires considerably more nuance.

An asymmetric funnel can indicate a small-study effect.

A small-study effect can have multiple causes.

Publication bias is one possible cause, but not the only one.

Egger's and Begg's tests provide statistical assessments of asymmetry, not direct tests proving publication bias.

Trim-and-fill can provide a sensitivity analysis but does not identify actual missing studies or definitively correct a biased literature.

References

Sterne, J.A.C., Sutton, A.J., Ioannidis, J.P.A., et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, d4002.
Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634.
Begg, C.B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088–1101.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463.
Page, M.J., Moher, D., Bossuyt, P.M., et al. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160.
Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (eds.). Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5. Cochrane.
Borenstein, M., Hedges, L.V., Higgins, J.P.T., & Rothstein, H.R. (2009). Introduction to Meta-Analysis. Wiley.