A Chatbot Intervention for Reducing HPV Vaccine Hesitancy: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for A Chatbot Intervention for Reducing HPV Vaccine Hesitancy (NCT05959564), a randomized na study. The primary outcome is Attitudes Toward HPV Vaccination.

2. Clinical Question

The study concerns communication about HPV vaccination and vaccine hesitancy. The registry lists cervical, anal, and penile cancer among the conditions. These conditions provide context for the topic of HPV vaccination; the posted primary outcome is an attitude score, not a cancer outcome or a measure of vaccination behavior. The record describes the project as a test of a chatbot intervention intended to inform future communication interventions among African American parents.

3. Trial Design

This was a randomized, parallel, single-masked study with three behavioral conditions: personalized chatbot, non-personalized chatbot, and no-chatbot control. The registry record reports enrollment of 456 participants and lists the study as completed. Its recorded start date is 2023-07-26, and its primary completion date is 2023-08-08. The phase is listed as not applicable.

In the personalized condition, participants read a standard CDC HPV vaccine message and then interacted with a chatbot delivering messages tailored to their personalities. In the non-personalized condition, participants read the same kind of standard message and then interacted with a chatbot whose messages were not tailored to personality. Participants in the control condition read the standard message without additional chatbot messages. The registry identifies the interventions as behavioral.

The posted primary outcome has results for 152 participants in the personalized condition, 154 in the non-personalized condition, and 150 in the control condition. The record does not explain the relationship between these outcome counts and the enrolled total, including whether any participants lacked outcome data or were excluded from a particular analysis. It also does not report the identity of the masked party or parties, the randomization procedure, allocation concealment, or the original design assumptions for the study.

The primary outcome timeframe is described as “baseline posttest,” with the test taking approximately 5 minutes to complete. The registry does not clarify that timeframe further. Ratings of the chatbot are listed as a secondary outcome. The record lists three outcome measures but posts one secondary outcome and no analyses. It does not report additional follow-up measurements in the posted results.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Attitudes Toward HPV Vaccination, measured over baseline posttest (test took approximately 5 minutes to complete).

The primary endpoint is “Attitudes Toward HPV Vaccination,” a continuous score intended to represent participants’ views about HPV vaccination. It is based on seven questions, including an example statement that the HPV vaccine is beneficial for adolescents. Responses to each question use a scale from 1, strongly disagree, to 5, strongly agree. The final score is the average across all seven responses. Higher scores therefore correspond to more agreement with the positively worded example, although the registry does not describe the scoring of any other items or whether they were reverse-coded.

A continuous score preserves variation across participants rather than reducing attitudes to a yes-or-no category. Comparing group means can describe whether the average score differs between study conditions. The registry reports the measure at a timeframe called “baseline posttest”; it does not further define that label or report a separate baseline result. The posted endpoint is an attitude measure, not a direct measure of vaccine uptake, completion of vaccination, or a change in disease incidence.

Using an attitude measure is relevant to the study’s stated focus on vaccine hesitancy and communication. However, a difference in this score alone cannot establish whether participants changed their subsequent decisions or behavior. The record does not report evidence about the measure’s validity, reliability, or interpretation beyond the described questions, response scale, and averaging procedure.

7. Results Posted in the Registry

• Personalized Chatbot Condition: mean 3.86 versus 3.41 on control; difference 0.45 (95% CI 0.26 to 0.64), standardized effect d = 0.53; z-test p = <0.001.
• Non-Personalized Chatbot Condition: mean 3.96 versus 3.41 on control; difference 0.55 (95% CI 0.36 to 0.74), standardized effect d = 0.64; z-test p = <0.001.

8. Statistical Interpretation

9. Statistical Methods Explained

The primary outcome is continuous: each participant’s score is the average of responses to seven attitude questions on a 1-to-5 scale. A group mean gives the average score among participants in a condition. A standard deviation describes how spread out individual scores are around that condition’s mean. These are descriptive summaries; the standard deviation is not a measure of uncertainty in the estimated group mean.

What does a mean difference describe?

General explanation: a mean difference compares the average outcome in one group with the average in another, expressed in the outcome’s original units. Here, the posted differences are 0.45 for the personalized chatbot versus control and 0.55 for the non-personalized chatbot versus control. Because the score averages responses on a 1-to-5 scale, these estimates are expressed in score points. They are not standardized effects and are not percentages.

The registry also posts pooled standard deviations and standardized effect-size values for these comparisons. A standardized effect expresses a mean difference relative to the variability of the scores, which can help describe magnitude on a scale independent of the original score units. The registry does not report how these values were used in the trial’s original planning or analysis. The posted standardized values should not be confused with the score-point mean differences used to describe the outcome on its original scale.

What does a confidence interval add?

General explanation: a confidence interval describes uncertainty around an estimated quantity under the statistical method used to construct it. A narrower interval indicates greater precision than a wider interval, all else being equal. It does not list the range of individual participant scores, and it does not guarantee that the true value lies within the particular interval. For each posted comparison, the interval is reported with the mean difference and has positive lower and upper limits.

How should the p-values be read?

General explanation: a p-value measures how compatible the observed data are with a specified null hypothesis under the assumptions of a statistical test. It is not the probability that the null hypothesis is true, nor does it quantify the size or practical importance of an effect. The posted p-values for both chatbot-versus-control comparisons are unadjusted and are reported as <0.001. The registry record does not identify the test used to calculate them or report adjusted p-values.

The trial is described as randomized, parallel, and single-masked. Random assignment is intended to support comparisons among study conditions by reducing systematic differences at assignment, but the registry does not report the sequence-generation method, concealment procedures, or who was masked. The parallel structure means participants were assigned to one of the listed conditions rather than being described as receiving each condition in turn. The record does not report whether participants were analyzed according to their assigned condition, how missing outcome values were handled, or whether any exclusions were made.

The posted comparisons use the no-chatbot condition as the reference for each chatbot condition. They answer two separate questions about mean scores relative to that reference. They do not supply a direct test of whether personalization changes the outcome relative to a non-personalized chatbot. The registry also lists ratings of the chatbot as a secondary outcome, but does not post a numerical result for that outcome. Three outcome measures are listed, while the record reports no analyses. The posted comparisons therefore do not document a complete statistical analysis plan or all analyses that may have been contemplated.

General explanation: when several outcomes or comparisons are tested, the chance of at least one false-positive result can increase if each is assessed independently without adjustment. Methods for controlling this issue can include prespecified outcome hierarchies or adjustments to inferential thresholds. The registry record does not report whether multiplicity was addressed, so the unadjusted p-values should be interpreted as such rather than assumed to reflect a family-wise adjustment.

The registry describes the primary timeframe as “baseline posttest” and says the test took approximately 5 minutes. It does not provide a separate baseline result, specify a change-score analysis, or describe repeated measurements. Consequently, the posted estimates are best read as comparisons of the reported group means, not as documented within-participant changes from baseline. The statistical model, assumptions checks, covariate adjustment, and handling of deviations from those assumptions are not reported in the registry record.

10. Confidence Intervals and Precision

The posted mean differences are 0.45 for the personalized chatbot condition versus control and 0.55 for the non-personalized chatbot condition versus control. Each comparison has a reported confidence interval with positive lower and upper limits. The registry also reports an unadjusted p-value of <0.001 for each comparison. These are inferential summaries of the estimated group differences, not descriptions of how much individual scores vary; the group standard deviations give information about that variation.

General explanation: a confidence interval combines an estimate with its sampling uncertainty under the method used. It is useful to examine the interval’s position and breadth, not only whether it includes a null difference. An interval that does not include zero is consistent with a difference in the direction of the estimate under the stated interval procedure. Interval width also matters because estimates with wider intervals are less precise. The posted intervals for both chatbot-versus-control comparisons lie on the positive side of zero.

The registry’s results do not state the confidence level in the comparison descriptions, identify the method used to construct the intervals, or explain whether the analyses adjusted for any variables. The reported p-values are explicitly unadjusted. No interval or inferential result is posted for a direct comparison between the personalized and non-personalized conditions, and no result for chatbot ratings is posted. Those omissions limit what can be concluded about the relative performance of the two chatbot formats or the secondary outcome.

Precision should also be distinguished from relevance beyond the measured score. Even a precisely estimated difference in the attitude scale would not, by itself, establish a change in vaccination behavior or persistence over time. The registry reports a short test duration and a timeframe described as “baseline posttest,” but does not report longer follow-up. The intervals describe uncertainty in the posted score comparisons; they do not resolve uncertainty about unmeasured outcomes or broader applicability.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The planning figures are participants per arm for the two-group comparisons; they are not total event counts. The curve and listed planning figures concern hypothetical future planning based on fractions of the observed mean difference, not the number of participants enrolled in this study. The displayed planning values use a two-sided significance level of 0.05, equal allocation between two arms, and a normal approximation. The effect to detect is set to the stated fraction of the observed effect. The original trial’s own sample-size assumptions and justification are not reported in the registry record.

The 80% power column gives the participants per arm in a planning scenario designed to have an 80% chance of detecting the selected effect under the listed assumptions, if that effect and the assumptions are correct. The 90% power column expresses the corresponding planning target for a 90% chance. These are design calculations, not probabilities that this completed study found a true effect. They are not observed power.

The planning illustrations are shown for 100%, 75%, and 50% of the observed effect. For the personalized-chatbot comparison, the listed participants per arm for 80% power are 56, 99, and 222, respectively; the corresponding figures for 90% power are 75, 132, and 297. For the non-personalized-chatbot comparison, the listed participants per arm for 80% power are 38, 68, and 152, respectively; the corresponding figures for 90% power are 51, 91, and 204. The smaller-effect scenarios require more participants per arm in these illustrations. These figures are planning examples based on the observed estimates, not a reconstruction of the study’s original design.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, Δ is the difference in means to detect and σ is the common standard deviation.

Worked calculation for this trial

1. Observed difference Δ = 0.450; pooled standard deviation σ = 0.845.
2. n = 2 × (1.96 + 0.842)² × 0.845² / 0.450² = 55.3, rounded up to 56 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
delta <- 0.45                      # observed difference in means
sigma <- 0.844981459362669                     # pooled standard deviation
n_per_arm <- function(power, frac = 1) {
  ceiling(round(2 * (z_a + qnorm(power))^2 * sigma^2 / (delta * frac)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (t-based, so very slightly larger)
cat("power.t.test:", round(power.t.test(delta = delta, sd = sigma, power = 0.80)$n, 1), "\n")

# Simulation: exact two-sample t-test using simulated means and variances
set.seed(5959564)
n    <- n_per_arm(0.80)
reps <- 20000
m_diff <- rnorm(reps, delta, sigma * sqrt(2 / n))             # difference in sample means
s2     <- sigma^2 * rchisq(reps, 2 * (n - 1)) / (2 * (n - 1))   # pooled sample variance
tstat  <- m_diff / sqrt(s2 * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(tstat) > qt(0.975, 2 * (n - 1))), reps))

Output from running the code

100% of observed effect: n per arm = 56 (80% power), 75 (90% power)
 75% of observed effect: n per arm = 99 (80% power), 132 (90% power)
 50% of observed effect: n per arm = 222 (80% power), 297 (90% power)
power.t.test: 56.3 
Simulated power at n = 56 per arm: 0.803 (20000 replicates)

The simulation draws many sets of group means and variances at the planned sample size and counts how often the two-sample t-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Multiplicity, Missing Data and Analysis Populations

The registry record lists one primary outcome, Attitudes Toward HPV Vaccination, and ratings of the chatbot as a secondary outcome. It also lists three outcome measures, but posts one secondary outcome and no analyses. The primary outcome has two posted comparisons against the no-chatbot control, one for each chatbot condition. The record does not report a multiplicity strategy, such as a prespecified hierarchy or an adjustment for testing multiple outcomes or comparisons. The posted p-values are unadjusted.

General explanation: when several hypotheses are examined, interpreting each test at its nominal level without adjustment can increase the chance of at least one false-positive result across the set of tests. This does not mean a particular finding is false; it means the scope of inference depends on what was planned and how many tests were conducted. Because the registry does not provide a complete analysis plan or analysis list, the extent of any multiplicity issue cannot be determined from the record.

Missing-data methods are not reported in the registry record. The record gives outcome sample sizes for each condition, but does not explain whether any participants were missing scores, how missing values were handled, or whether the reported analysis included all randomized participants. It also does not describe an intention-to-treat or per-protocol analysis population, exclusions after randomization, or any sensitivity analyses. The enrolled total alone does not establish which analysis population was used.

The record does not specify the statistical model, covariate adjustment, test assumptions, or whether the outcome analyses were prespecified. It describes the outcome timeframe as “baseline posttest,” but does not explain whether any baseline scores were used in the analysis. No direct inferential comparison between the two chatbot conditions is posted. These missing details do not negate the posted descriptive values, but they limit assessment of the full analysis strategy and how robust the estimates are to alternative analytic choices.

14. What the Result Does, and Does Not, Mean

The posted results show that the average attitude score was 3.86 in the personalized chatbot condition and 3.96 in the non-personalized chatbot condition, compared with 3.41 in the no-chatbot control condition. The two posted mean differences versus control are 0.45 and 0.55, respectively. Each comparison has a reported confidence interval with positive limits and an unadjusted p-value of <0.001. These results describe the measured attitude scores in the study conditions.

They do not show whether one chatbot format performed better than the other. A direct statistical comparison of the personalized and non-personalized conditions is not reported. The separate comparisons against control cannot be treated as a formal test between the chatbot conditions. The registry also does not post a numerical result for the secondary outcome of chatbot ratings.

The measure captures responses to seven questions on attitudes toward HPV vaccination. It does not directly measure whether a participant obtained or recommended vaccination, whether an attitude persisted, or whether any cancer outcome changed. The record reports a test taking approximately 5 minutes and does not report longer-term outcome data. Thus, the estimates should be interpreted as differences in the reported score rather than evidence about longer-term decisions, vaccination behavior, or disease outcomes.

The results also do not establish that the same estimates would apply to people beyond the study participants. The target population in the registry summary is African American parents, and the study was described as an online experiment. The record does not report enough information about eligibility, recruitment, or participant characteristics to assess how closely the analyzed participants represent a wider population. It also does not report detailed methods for missing data, analysis populations, or adjustment for other factors.

In statistical terms, the posted comparisons are evidence of differences in average scores relative to control under the reported analyses. They are not, on their own, a complete account of uncertainty about the study’s design, analysis choices, other outcomes, or applicability. The registry record does not report a protocol-level interpretation of what size of score difference would be important to participants or researchers.

15. Why This Trial Matters Statistically

This study illustrates how a randomized communication intervention can be evaluated using a continuous attitude measure and a control condition. The three arms separate a personalized chatbot, a non-personalized chatbot, and a standard-message condition without chatbot messages. That structure makes the two chatbot-versus-control comparisons directly interpretable as differences in average scores relative to the same reference group.

The posted estimates also demonstrate why reporting both effect estimates and uncertainty is useful. A mean difference describes the score-point contrast, while a confidence interval communicates uncertainty around that contrast. The unadjusted p-values provide another statistical summary, but do not convey the size of the difference or address the full scope of multiple possible comparisons. The record’s lack of a posted direct comparison between chatbot formats is a reminder that two contrasts with a shared control do not substitute for a test of the formats against one another.

The outcome’s scope is equally important. Attitudes toward HPV vaccination are relevant to the study’s stated focus on hesitancy, but are not equivalent to vaccination uptake or a clinical outcome. Clear separation between an attitude score and downstream behavior helps prevent statistical findings from being interpreted more broadly than the measured endpoint supports.

The sample-size illustrations connect observed estimates to future planning, while making clear that planning depends on the assumed effect size and design conditions. Smaller effects require larger planned samples in the listed examples. Since the registry does not report the original design justification, those illustrations should not be mistaken for evidence about how the completed study was designed. The record’s missing information on multiplicity, missing data, analysis populations, and model specification also shows why numerical results are only one part of evaluating a trial’s statistical evidence.

16. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record leaves several trial-specific details unreported. It does not give participant eligibility criteria beyond the study summary’s focus on African American parents, recruitment methods, or detailed characteristics of participants. It identifies the study as single-masked but does not say who was masked. Randomization is reported, but the sequence-generation and allocation-concealment methods are not described.

The primary outcome timeframe is labeled “baseline posttest,” without further explanation. A separate baseline result, repeated-measures schedule, or longer-term follow-up is not reported. The posted endpoint is an attitude score based on seven questions, but evidence about the measure’s reliability, validity, or interpretation is not included. The record does not establish whether score differences correspond to behavior or persist over time.

The record does not report how missing outcome data were handled, which analysis population was used, or whether any participants were excluded after randomization. Although outcome counts are listed for all three conditions, the record does not explain their relationship to total enrollment. It also does not describe the statistical model, covariate adjustment, assumptions checks, analysis prespecification, or multiplicity handling. The comparisons against control are posted, but a direct comparison between chatbot conditions and numerical results for chatbot ratings are not.

Adverse-event entries list zero counts in each condition for serious events, other events, and deaths. These registry entries should be read as the counts recorded in the record; they do not supply detail about event-collection procedures or the scope of monitoring. More broadly, the online experiment and the reported attitude endpoint limit conclusions to the outcomes and setting described. The registry record does not report evidence on vaccination behavior, clinical outcomes, or broader generalizability.

17. Related Tutorials

18. Related Statistical Calculators

19. Sources

• ClinicalTrials.gov: A Chatbot Intervention for Reducing HPV Vaccine Hesitancy — NCT05959564.
• Registry record last updated 2024-10-21; status last verified 2024-07.
• Results first posted 2024-10-21.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

20. Record Summary

The University of Maryland, College Park sponsored this completed randomized, parallel, single-masked study, which enrolled 456 participants. It tested personalized and non-personalized chatbot messages after a standard CDC HPV vaccine message against a no-chatbot control. The study summary identifies African American parents as the target population.

The primary outcome was an average score across seven HPV vaccination attitude questions, with responses on a 1-to-5 scale. Posted means were 3.86 for the personalized chatbot, 3.96 for the non-personalized chatbot, and 3.41 for control. The mean differences versus control were 0.45 and 0.55; both comparisons have unadjusted p-values of <0.001 and reported confidence intervals with positive limits.

The registry lists chatbot ratings as a secondary outcome, but does not post a numerical result for it. It does not report the original sample-size justification, analysis population, missing-data methods, multiplicity approach, or a direct inferential comparison between chatbot conditions. The posted primary results concern attitude scores, not vaccination behavior or clinical outcomes.