← Clinical Trials
Systematic Review NA Continuous Endpoint NCT01848782

Impact of Adding a Limitation Section in Abstract of Systematic Review: Complete Statistical Analysis

This randomized online survey trial examined whether adding a limitations section to a systematic review abstract changed readers’ interpretation. The primary outcome was confidence in the review’s results, measured on a 0-to-10 scale.

Trial start: 2013-06  ·  Enrollment: 300  ·  Sponsor: Assistance Publique - Hôpitaux de Paris
Scope of this record

This page provides an independent statistical analysis and educational interpretation of publicly reported results. ClinicalTrials.gov provides the official trial registry record. Numerical trial results on this page are restricted to the results posted in the ClinicalTrials.gov record; calculations added by Clinical Biostats are labelled as such.

1. Trial at a Glance

This page covers the registry results for Impact of Adding a Limitation Section in Abstract of Systematic Review (NCT01848782), a randomized na study. The primary outcome is Confidence in the Results of the Systematic Review.

300
Enrollment
NA
2
Arms
In the primary outcome
-0.20
Mean difference
Abstract Without Limitation Section
-0.74–0.34
95% CI
Two-sided
FeatureRegistry value
ClinicalTrials.gov identifierNCT01848782
PhaseNA
ConditionSystematic Review
AllocationRandomized
Design modelParallel
MaskingSingle
Enrollment300
Arms in the primary outcome2
Lead sponsorAssistance Publique - Hôpitaux de Paris
Results postedYes

2. Clinical Question

Population

Eligible participants were corresponding authors of clinical trials published between 2010 and 2012 and referenced in Medline. They were invited by email to take part in an online survey. The registry record reports a planned enrollment of 300 participants, with 150 in each randomized group. The participants were therefore readers with experience as corresponding authors of clinical trials, rather than a general sample of all readers or a sample of patients.

Intervention

Participants assigned to the intervention group read a standardized systematic review abstract with a limitations section added. The section focused on the quality or risk of bias of the included studies. The selected abstracts came from systematic reviews with meta-analysis, and the treatment name was hidden.

Comparator

Participants assigned to the comparison group read a standardized abstract without a limitations section. The registry describes the selected set as 30 abstracts with a conclusion in favour of the experimental treatment. Each participant read one abstract from the group to which they had been randomized.

Primary question

The primary question was whether including a limitations section changed readers’ confidence in the results of a systematic review. After reading the abstract, participants answered a question about confidence on a 10-point Likert scale, ranging from 0, not at all confident, to 10, very confident. The registry lists a one-month timeframe for the outcome. The study also assessed several related judgments, including confidence in the validity of the conclusions and perceived influence on clinical practice.

3. Trial Design

This was a randomized controlled trial with two parallel groups. The registry lists the allocation as randomized, the model as parallel, and masking as single. It does not identify who was masked or describe the randomization sequence, allocation concealment, or procedures used to implement masking; these details are not reported in the registry record.

The planned enrollment was 300, split evenly between two groups of 150 in the posted outcome results. Participants were invited by email to complete an online survey. After randomization, each participant read one standardized abstract from the assigned group and answered five questions about their interpretation of the results. The abstracts came from systematic reviews with meta-analysis, and the treatment name was hidden. In one group, an author-written limitations section describing risk of bias in the included studies was present; in the other, it was absent.

The eligible participants were corresponding authors of clinical trials published between 2010 and 2012 and referenced in Medline. This recruitment approach makes the study a test of responses among a defined group of research authors. It does not directly establish how other types of readers would respond.

The registry lists the trial as completed, with a start in 2013-06 and primary completion in 2013-07. The sponsor was Assistance Publique - Hôpitaux de Paris. The phase is listed as not applicable. The original protocol’s sample-size justification, recruitment flow, randomization details, and analysis-population definitions are not reported in the registry record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

StageAbstract With Limitation Section AddedAbstract Without Limitation Section
STARTED150150
COMPLETED150150
NOT COMPLETED00

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

CharacteristicUnitAbstract With Limitation Section AddedAbstract Without Limitation SectionTotal
Participants analysed150150300
Age, Continuousyears45.2 (SD 10.3)46.7 (SD 10.9)46.0 (SD 10.7)
Sex: Female, Male: FemaleParticipants414384
Sex: Female, Male: MaleParticipants109107216
Education qualification: MDParticipants7878156
Education qualification: PhDParticipants5864122
Education qualification: OtherParticipants14822
Clinical experience: <5 yearsParticipants6814
Clinical experience: 5 to 10 yearsParticipants342862
Clinical experience: >10 yearsParticipants110114224
Reading reports of systematic reviews: Rarely or sometimesParticipants6162123
Reading reports of systematic reviews: RegularlyParticipants8987176
No. of randomized trials involved in: 0 trialParticipants161127
No. of randomized trials involved in: 1-3 trialsParticipants484896
No. of randomized trials involved in: 4-9 trialsParticipants455499
No. of randomized trials involved in: >10 trialsParticipants413778
Authored at least one systematic reviewParticipants7782159
No. of systematic reviews peer-reviewed: 0 reviewParticipants7564139
No. of systematic reviews peer-reviewed: 1-3 reviewsParticipants5569124
No. of systematic reviews peer-reviewed: >3 reviewsParticipants201636
Training in clinical epidemiologyParticipants10176177
Training in methods of randomized trialsParticipants8671157

6. Primary Endpoint

The registry lists the primary outcome as Confidence in the Results of the Systematic Review, measured over 1 month.

ElementRegistry entry
Result typeMean
Measure of spreadStandard Deviation
Unitscore on a scale
Endpoint classificationContinuous Endpoint

The primary endpoint was confidence in the results of the systematic review. Participants were asked how confident they were in the study’s results, using a 10-point Likert scale from 0, not at all confident, to 10, very confident. The registry describes the outcome as a mean score with standard deviation and gives a timeframe of one month.

This endpoint captures a reader’s stated degree of confidence after exposure to an abstract. It is a continuous score: a mean summarizes the group’s average response, while a standard deviation describes how spread out individual scores are around that average. In general, a mean score provides a compact summary of responses, but it does not show the full distribution, explain why people chose particular answers, or establish whether a score difference would matter in practice.

The endpoint was used to address the trial’s central question about interpretation: whether the presence of a limitations section was associated with a change in readers’ confidence. It measures a judgment reported in a survey, not the accuracy of that judgment, the quality of the underlying systematic review, or a change in clinical decisions. The registry does not report a threshold for a meaningful difference on this scale or explain whether the one-month timeframe refers to the timing of assessment or another aspect of outcome collection.

7. Results Posted in the Registry

Mean difference: Abstract Without Limitation Section

-0.20

95% CI: -0.74 to 0.34

Recalculated from the posted summary statistics

Primary outcome by arm, with 95% confidence intervals
Primary outcome by arm with 95% confidence intervals3.94.24.44.64.95.1Abstract With Limitation Sect… (n=150)Abstract Without Limitation S… (n=150)score on a scale (95% CI)
Treatment effect versus the control arm
Forest plot of mean difference vs control (95% CI)Abstract Without Limitation S…-0.83-0.58-0.33-0.070.180.43mean difference vs control (95% CI)

8. Secondary Efficacy Endpoints

The registry also posts results for secondary outcomes. For each outcome below, the endpoint type, the test applied to the posted summary statistics, the result and its interpretation are given in turn. Secondary endpoints are usually not adjusted for multiple testing, so they support rather than confirm the primary finding.

Secondary outcomeTypeEffect measureEstimate (95% CI)Planning size, 80% power
Confidence With the Validity of the ConclusionsContinuous EndpointMean difference-0.10 (-0.64 to 0.44)9,058 per arm
Beneficial Effect of the Experimental InterventionContinuous EndpointMean difference-0.10 (-0.66 to 0.46)9,458 per arm
Influence of Results on Clinical PracticeContinuous EndpointMean difference0.00 (-0.56 to 0.56)n/a

Confidence With the Validity of the Conclusions

Treatment effect versus the control arm
Forest plot of mean difference vs control (95% CI)Abstract Without Limitation S…-0.73-0.48-0.230.030.280.53mean difference vs control (95% CI)

Beneficial Effect of the Experimental Intervention

Treatment effect versus the control arm
Forest plot of mean difference vs control (95% CI)Abstract Without Limitation S…-0.74-0.49-0.230.030.290.54mean difference vs control (95% CI)

Influence of Results on Clinical Practice

Treatment effect versus the control arm
Forest plot of mean difference vs control (95% CI)Abstract Without Limitation S…-0.64-0.39-0.130.130.390.64mean difference vs control (95% CI)

The posted secondary outcomes follow the same broad pattern as the primary result: group averages are close, and the reported mixed models comparisons do not show a clearly separated pattern between the abstract formats. Across the outcomes, the estimates do not suggest a strong and consistent shift in readers’ judgments when a limitations section is included. This overall pattern is compatible with the primary finding of little observed separation in average confidence, while remaining subject to uncertainty about the size and direction of effects.

The secondary outcomes address related but distinct judgments: confidence in the validity of conclusions, perceived beneficial effect of the experimental intervention, influence of results on clinical practice, and rigor of the systematic review. They broaden the interpretation beyond confidence in results, but they are still questionnaire responses collected after reading an abstract. They do not establish whether readers’ judgments were accurate or whether their clinical decisions changed.

Four secondary outcomes have posted results, although the registry lists five outcome measures. The status of the remaining measure is not reported in the registry record. The registry record does not report a prespecified hierarchy among the secondary outcomes or a multiplicity adjustment. In general, when several outcomes are examined, separate unadjusted tests can yield apparently notable results by chance, even when there is no underlying effect. This makes individual secondary results less persuasive when considered on their own.

Secondary outcomes are most useful as context for the primary question and as evidence about whether a pattern is coherent across related measures. They should not be treated as independent confirmation without a documented analysis plan, clear prioritization, and attention to multiplicity. The overall consistency of the posted results is informative, but it does not remove uncertainty about generalizability, measurement, or the study’s unreported analytic choices.

Secondary outcomes registered for this trial, for reference (only those above were recalculated):

9. Statistical Interpretation

Clinical Biostats interpretation

For the primary outcome, the mean confidence score was 4.6 in the group that read an abstract with a limitations section and 4.4 in the group that read an abstract without one. The registry’s mixed models analysis reports a mean difference of 0.2, with a 95% confidence interval from -0.4 to 0.7 and p = 0.5. The simple unadjusted recalculation uses the opposite contrast direction, without a limitations section minus with one: its mean difference is -0.2, with a 95% confidence interval from -0.744 to 0.344 and p = 0.471.

Both results describe a small difference in the group averages, and both confidence intervals include zero. The posted results therefore do not establish a clear difference in average confidence between the two assigned abstract formats. That is not the same as demonstrating that the formats have identical effects. The interval is compatible with values on either side of zero, so the result leaves uncertainty about the direction and size of any difference.

The trial reports 150 participants in each group for this endpoint. The standard deviations were 2.5 for the abstract with a limitations section and 2.3 for the abstract without one. These spreads describe variation in participants’ scores within each group; the means alone do not convey that individual responses varied. The result is about average confidence in this survey setting, not about whether participants correctly judged the underlying evidence.

The registry also reports a mean difference and confidence interval from a mixed models analysis. The analysis model’s full specification, including any covariates or assumptions, is not reported in the registry record. The reported analysis should be distinguished from the simple unadjusted recalculation, which uses posted group means and standard deviations and does not reproduce a model-based analysis.

Interpretation should also account for the study’s setting and participants. The abstracts were selected from a sample and described as having conclusions in favour of the experimental treatment; participants were corresponding authors of clinical trials invited to an online survey. The results may not represent responses to other kinds of abstracts or readers. No conclusion about a change in actual clinical practice follows directly from a survey score. The registry includes a separate outcome concerning perceived influence on clinical practice, but that too is a reported judgment rather than observed practice.

10. Statistical Methods Explained

The primary outcome is continuous because it is a score on a numerical scale. The registry summarizes each group using a mean and standard deviation. In general, the mean is the arithmetic average of the observed scores, while the standard deviation describes how much scores vary around that average. Neither summary tells the whole story: two groups can have similar means while individual responses differ considerably, and summaries do not show the distribution’s shape or possible clustering at particular scale values.

What does a mean difference describe?

A mean difference compares the average score in one group with the average in another. Its sign depends on which group is subtracted from which. The posted group means are 4.6 with a limitations section and 4.4 without one. The simple recalculation expresses the contrast as without a limitations section minus with one, giving -0.2. The registry’s mixed models analysis reports 0.2. The opposite signs are consistent with contrasts expressed in opposite directions; they should not be read as conflicting evidence about the size of the group separation. The reported registry analysis is the sponsor’s analysis, and it takes precedence if there is a disagreement with an unadjusted recalculation.

What does the confidence interval add?

A confidence interval gives a range of estimates compatible with the data and the method used to construct it. In general, an interval that includes zero for a difference is consistent with no average difference, as well as with differences in either direction within the interval. It does not assign a probability to the true effect being inside that particular interval, and it does not prove equivalence. The primary registry interval runs from -0.4 to 0.7 for the reported mixed models estimate; the unadjusted recalculated interval runs from -0.744 to 0.344 for the reversed contrast. Their endpoints are not identical because the registry analysis and the simple summary-statistic calculation are not the same analysis.

How should the p-values be read?

The registry reports p = 0.5 for its mixed models analysis of the primary outcome. The unadjusted recalculation gives p = 0.471. In general, a p-value describes how unusual data at least as incompatible with a specified null model would be if that model and the test assumptions held. It is not the probability that the null hypothesis is true, and it does not measure the size or importance of an effect. A relatively large p-value does not establish that two groups are equivalent. The registry record does not report the full statistical analysis plan or the details needed to assess every model assumption.

Why distinguish the model-based result from the recalculation?

The reported analysis is identified as a mixed models analysis of the final values. A mixed model can account for structures such as repeated observations or clustering when those features are present and specified in the model. That is a general description of the method; the registry record does not describe the model structure, covariates, or whether repeated measures or clustering were included in this analysis. The posted summary-statistic recalculation instead compares the two group means using the posted sample sizes and standard deviations. It is simple and transparent, but it is unadjusted and cannot reconstruct unreported model choices.

The recalculated primary comparison is based on 150 participants per group, means of 4.6 and 4.4, and standard deviations of 2.5 and 2.3. The reported standard deviations are used directly rather than approximated. The recalculation is therefore a check based on the posted summaries, not a substitute for the sponsor’s analysis or an independent analysis of participant-level data.

What is not documented?

The registry record does not report the original sample-size assumptions, a prespecified clinically or practically important score difference, the full statistical analysis plan, or whether interim analyses were planned or conducted. It also does not describe missing-data methods, the analysis population, or the handling of protocol deviations. These omissions limit assessment of how closely the posted analyses follow the intended plan. In general, complete methods documentation helps readers distinguish prespecified decisions from choices made after data collection, but no such distinction can be made from the information recorded.

Several secondary outcomes are listed, and four have posted results. In general, assessing multiple outcomes increases the chance that at least one result will appear notable by chance if each is tested separately. The registry does not report a multiplicity adjustment or a hierarchy for interpreting the outcomes. That uncertainty is one reason to read secondary findings as supporting context rather than as independent confirmation of the primary result.

11. Confidence Intervals and Precision

The primary registry analysis estimates a mean difference of 0.2, with a 95% confidence interval from -0.4 to 0.7. The estimate is close to zero, but the interval spans zero and includes values on either side. In practical statistical terms, the posted result does not pin down a single precise effect size: it remains compatible with a modest difference in one direction, little average difference, or a difference in the other direction within the reported interval.

The unadjusted recalculation expresses the comparison as the score without a limitations section minus the score with one. Its estimate is -0.2, with a 95% confidence interval from -0.744 to 0.344. Because the contrast is reversed, the sign also reverses. The recalculated interval is based on group summary statistics; the registry interval comes from the mixed models analysis. The distinction matters: intervals from different methods need not have identical endpoints, even when their overall messages are similar.

In general, a confidence interval reflects both the estimated difference and its uncertainty under the analysis method. A narrower interval indicates greater precision for that estimate, while a wider interval leaves a broader range of effects compatible with the data. Precision is not the same as importance. The registry does not report a threshold for a meaningful change in confidence on the 0-to-10 scale, so the interval cannot be classified against a stated importance threshold.

The group standard deviations were 2.5 and 2.3, and there were 150 participants in each group for the primary outcome. These summaries indicate that individual scores varied within both groups. The confidence interval concerns uncertainty about the group contrast, not the range of individual scores. It also does not show whether responses were concentrated near particular points on the Likert scale, because the score distribution is not reported.

12. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

Effect to detectParticipants per arm, 80% powerParticipants per arm, 90% power
100% of the observed effect2,2653,032
75% of the observed effect4,0265,390
50% of the observed effect9,05812,126
Sample size needed versus the size of the effect to detect
Sample size needed versus size of effect to detect2,0005,00010,00020,00040%60%80%100%80% power90% powersize of effect to detect, as % of the effect observed (log vertical scale: participants per arm)
Treat these figures cautiously. The observed difference is imprecise: its confidence interval includes no difference at all. A sample size built on a noisy estimate can be far too small or too large, so the smaller-effect rows are the more cautious guide.
Method and assumptions. Two-sided α = 0.05, 1:1 allocation, two-sample comparison of means with the pooled standard deviation. Calculated for the comparison: Abstract Without Limitation Section.

The sample-size illustrations are planning calculations based on the observed primary effect, not the trial’s original design assumptions. They give participants per arm for the continuous outcomes in this trial; they are not totals across both arms and are not total event counts. The listed assumptions are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. Each planning scenario uses an effect to detect equal to the stated fraction of the observed effect.

The 80% power column describes the number of participants per arm in a hypothetical future study that, under those assumptions, would have an 80% chance of detecting the specified effect if that effect were the true one. The 90% power column gives the corresponding planning target for a 90% chance. In general, higher target power requires more participants when other assumptions are held fixed. These are planning illustrations, not observed power for the completed trial. The original trial’s own design assumptions and sample-size justification are not reported in the registry record.

Rows for planning for 100%, 75%, or 50% of the observed effect show how the target changes if the effect selected for planning is smaller than the observed estimate. Smaller effects are harder to distinguish from random variation, so they require more participants for the same target power. This relationship is especially relevant when the observed effect is small: planning directly from an unstable estimate can produce a target that may not be appropriate for a future study.

The figures should therefore be read as conditional illustrations, not as a recommendation or as evidence that the completed trial was adequately or inadequately sized. They depend on the assumptions listed and on the observed effect used as their starting point. The registry does not report the original rationale for enrolling 300 participants, nor whether the trial used other planning inputs or adjustments.

13. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z1−α/2 = 1.960) and the desired power (z1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n = 2 (z1−α/2 + z1−β)2 σ2 / Δ2

n is the number of participants in each arm, Δ is the difference in means to detect and σ is the common standard deviation.

Worked calculation for this trial

  1. Observed difference Δ = -0.200; pooled standard deviation σ = 2.402.
  2. n = 2 × (1.96 + 0.842)2 × 2.4022 / -0.2002 = 2,264.4, rounded up to 2,265 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
delta <- -0.199999999999999                      # observed difference in means
sigma <- 2.40208242989286                     # pooled standard deviation
n_per_arm <- function(power, frac = 1) {
  ceiling(round(2 * (z_a + qnorm(power))^2 * sigma^2 / (delta * frac)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (t-based, so very slightly larger)
cat("power.t.test:", round(power.t.test(delta = delta, sd = sigma, power = 0.80)$n, 1), "\n")

# Simulation: exact two-sample t-test using simulated means and variances
set.seed(1848782)
n    <- n_per_arm(0.80)
reps <- 20000
m_diff <- rnorm(reps, delta, sigma * sqrt(2 / n))             # difference in sample means
s2     <- sigma^2 * rchisq(reps, 2 * (n - 1)) / (2 * (n - 1))   # pooled sample variance
tstat  <- m_diff / sqrt(s2 * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(tstat) > qt(0.975, 2 * (n - 1))), reps))

Output from running the code

100% of observed effect: n per arm = 2,265 (80% power), 3,032 (90% power)
 75% of observed effect: n per arm = 4,026 (80% power), 5,390 (90% power)
 50% of observed effect: n per arm = 9,058 (80% power), 12,126 (90% power)
power.t.test: 2265.4 
Simulated power at n = 2,265 per arm: 0.802 (20000 replicates)

The simulation draws many sets of group means and variances at the planned sample size and counts how often the two-sample t-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

14. Analyses Reported by the Sponsor

MethodEstimate95% CIP-value
Mixed Models AnalysisMean Difference (Final Values) 0.2-0.4 to 0.70.500

The registry’s primary mixed models analysis reports a mean difference of 0.2, a 95% confidence interval from -0.4 to 0.7, and p = 0.5. The simple unadjusted recalculation from the posted group summaries gives a mean difference of -0.2, a 95% confidence interval from -0.744 to 0.344, and p = 0.471. The recalculation uses the opposite contrast direction: without a limitations section minus with one, whereas the registry’s reported estimate is positive.

After accounting for that contrast direction, the two analyses describe a similar near-zero separation between the group averages and both intervals include zero. The intervals are not identical because the sponsor reports a mixed models analysis while the recalculation uses posted group means, standard deviations, and sample sizes without adjustment. The model’s full specification is not reported in the registry record, so the recalculation cannot reproduce it.

The recalculation is a transparent check of the summaries, not a replacement for the reported analysis. The registry analysis takes precedence as the sponsor’s own analysis if the estimates conflict. In this case, the broad interpretation is similar, while the exact estimates and interval endpoints differ. Neither result establishes that the groups are equivalent or that the effect is exactly zero.

15. Multiplicity, Missing Data and Analysis Populations

The registry lists one primary outcome and five outcome measures in total, with four secondary outcomes having posted results. It also lists five analyses. The record does not report whether the outcomes were ranked in a prespecified hierarchy, whether one secondary outcome was considered confirmatory, or whether statistical testing was adjusted for the number of outcomes. These details are not reported in the registry record.

In general, multiplicity arises when researchers test more than one outcome or make multiple comparisons. If each test is interpreted separately at a conventional threshold, the chance of obtaining at least one apparently unusual result can rise as more tests are performed. Adjustment methods or a prespecified testing hierarchy can address this issue, but they change how results should be interpreted. Because the registry does not describe such a plan, the secondary p-values should be treated as exploratory evidence rather than as a set of independent definitive tests.

The primary outcome remains the central endpoint for the trial’s stated question. The registry reports the sponsor’s mixed models analysis for that endpoint, but the full statistical analysis plan is not reported. Whether the model included covariates, how it handled any repeated or clustered structure, and whether any planned sensitivity analyses were conducted are not reported in the registry record. The record also does not report interim analyses.

Missing-data methods and analysis populations are likewise not described. The posted results list 150 participants in each group, but the registry record does not state whether these numbers represent everyone randomized, everyone who completed the survey, or everyone included in the analysis. It does not report the amount or pattern of missing outcome data, how incomplete questionnaires were handled, or whether participants were analyzed according to their randomized assignments. Those details matter because missing responses can affect group comparisons if the likelihood of missingness or the observed responses differ between participants.

Randomized allocation can support a fair comparison when followed by appropriate analysis, but randomization alone does not resolve missing outcomes, deviations from assignment, or selective outcome reporting. Without the relevant documentation, it is not possible to assess those features fully. This uncertainty does not establish that a problem occurred; it identifies information that is not reported in the registry record.

16. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

ArmParticipants with serious adverse eventsParticipants at riskPercentage
Abstract With Limitation Section Added01500.0%
Abstract Without Limitation Section01500.0%

Deaths

ArmDeaths (participants affected of at risk)
Abstract With Limitation Section Added0 of 150
Abstract Without Limitation Section0 of 150

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

ArmParticipants affected of at risk
Abstract With Limitation Section Added0 of 150
Abstract Without Limitation Section0 of 150
Do not overextend the safety comparison. Serious-adverse-event counts are safety observations, not efficacy outcomes, and should not be combined with the primary result into a single benefit-risk number.

The registry records zero serious adverse events in the group assigned to abstracts with a limitations section and zero in the group assigned to abstracts without one. Each group lists 150 participants at risk. The other adverse-event counts are also recorded as zero in both groups, and deaths are recorded as zero in both groups.

These figures describe what the registry reports for the study; they do not establish that no adverse event could occur or that every event was captured. The registry does not provide further details about how adverse events were defined, solicited, monitored, or verified. Because the intervention was presentation of an abstract in an online survey, the trial’s main focus was interpretation rather than a clinical treatment outcome. The safety entries should therefore be read as recorded event counts, not as evidence about the safety of the health care interventions discussed in the abstracts.

17. What the Result Does, and Does Not, Mean

The posted results address whether adding a limitations section to a systematic review abstract changed the average confidence score among the surveyed participants. The primary group means were close, and the sponsor’s mixed models estimate was 0.2 with a 95% confidence interval from -0.4 to 0.7. The unadjusted recalculation, expressed in the reverse direction, was -0.2 with a 95% confidence interval from -0.744 to 0.344. The reported estimates are compatible with little average separation, but the uncertainty means that the results do not demonstrate exact equality.

The outcome is self-reported confidence after reading an abstract. It does not directly measure whether participants correctly understood the evidence, whether they gave more appropriately calibrated judgments, or whether they changed their clinical decisions. The trial also does not establish whether the findings apply to other populations of readers, other types of systematic reviews, abstracts with different conclusions, or other ways of describing limitations.

The secondary outcomes provide related information about readers’ views, but they are also survey judgments. The registry does not report a multiplicity plan, detailed missing-data methods, or a full statistical analysis plan. These omissions make it harder to evaluate how the analyses were selected and how robust the estimates are to analytic choices.

In general, a result that does not establish a clear difference should not be interpreted as proof that an intervention has no effect. A claim of equivalence would require an appropriate design and a prespecified margin defining how small a difference must be to count as equivalent; no such margin is reported in the registry record. Nor does the observed estimate alone establish that adding a limitations section improves or worsens interpretation. The evidence reported is specifically about measured responses in this randomized online survey.

18. Why This Trial Matters Statistically

This trial asks a methodological question about how information is presented in a research abstract and how readers respond to it. That is statistically important because abstracts compress evidence into a short format, and readers may rely on them when deciding how much confidence to place in a review’s conclusions. Testing a presentation feature through randomized allocation offers a way to compare responses while reducing some sources of systematic difference between groups.

The study also illustrates the distinction between a measured response and a broader claim. Confidence scores are relevant to interpretation, but they are not the same as accuracy, calibration, or changes in practice. A well-defined endpoint helps keep conclusions aligned with what was actually measured. Here, the primary measure captures participants’ stated confidence after reading one abstract, in a survey context.

For biostatistical interpretation, the trial is a useful example of why estimates and intervals matter more than a simple significant-or-not label. The sponsor’s estimate is near zero and its interval includes zero, but that does not prove there is no difference. The unadjusted recalculation reaches a similar broad reading while illustrating how contrast direction changes the sign and how a model-based estimate can differ from a calculation based on summary statistics.

The multiple outcomes also make the study a reminder that secondary analyses need context. A set of related judgments can help assess whether the pattern is coherent, but unadjusted testing across outcomes can make isolated results less reliable. Finally, the registry’s missing detail about original planning, missing data, and model specification shows why complete reporting is essential for readers to assess precision and reproducibility, even when the headline comparison is straightforward.

19. Limitations

The study’s participants were a specific group: corresponding authors of clinical trials published between 2010 and 2012 and referenced in Medline. Their responses may not represent those of clinicians who are not trial corresponding authors, patients, members of the public, or other research readers. The registry does not report participant characteristics beyond the eligibility description, response rates, or how many people were invited, so the representativeness of those who completed the survey cannot be assessed from the record.

The stimulus materials were also restricted. The registry describes 30 selected abstracts with conclusions in favour of the experimental treatment, drawn from a sample of systematic reviews. The treatment name was hidden, and the abstracts were standardized. These design choices may help isolate the presentation feature being tested, but they limit what can be inferred about reactions to abstracts with other conclusions, different topic areas, or less standardized wording. The registry does not report how abstracts were selected beyond the stated description or how variation between abstracts was handled in the analysis.

Outcomes were answers to survey questions after reading one abstract. This does not show whether participants retained their judgments, evaluated the evidence accurately, or acted differently in clinical practice. Although the registry lists a one-month timeframe, it does not clarify the timing details of the survey assessment. The record does not report the exact wording or scoring details for the secondary questions beyond their titles.

Important methodological details are not reported in the registry record, including the randomization procedure, who was masked, analysis populations, missing-data methods, full model specification, and the original sample-size justification. The record also does not describe multiplicity adjustment or an outcome hierarchy. These reporting gaps limit assessment of bias, precision, and reproducibility; they do not by themselves show that the trial was conducted improperly.

20. Related Tutorials

21. Related Statistical Calculators

22. Sources

Continue through the Clinical Biostats statistical pathway

Use this trial's endpoint as a starting point for deeper study of effect measures, confidence intervals, and sample-size planning.

23. Record Summary

NCT01848782 is a completed randomized, two-arm, parallel online survey trial of adding a limitations section to systematic review abstracts. The registry lists planned enrollment of 300 and reports 150 participants in each group for the primary outcome. Eligible participants were corresponding authors of clinical trials published between 2010 and 2012 and referenced in Medline.

The primary outcome was confidence in the results of a systematic review, measured on a 10-point Likert scale. The registry reports a mixed models mean difference of 0.2, with a 95% confidence interval from -0.4 to 0.7 and p = 0.5. The simple unadjusted recalculation in the reverse contrast direction is -0.2, with a 95% confidence interval from -0.744 to 0.344 and p = 0.471. The posted estimates are close to zero, with intervals that include zero.

Four secondary outcomes have posted results. The registry does not report the full analysis plan, original sample-size justification, missing-data approach, or multiplicity procedures. The findings concern reported survey judgments and should not be taken as direct evidence about accuracy or clinical practice.

Clinical Biostats