Testing Multiple Behavioral Science Strategies to Increase Flu-Shot Rates: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Testing Multiple Behavioral Science Strategies to Increase Flu-Shot Rates (NCT04565353), a randomized na study. The primary outcome is Number of Participants With Flu Shot Receipt at or Before Appointment.

2. Clinical Question

The trial is designed as a multi-arm comparison rather than a single test of one message against no message. That structure allows the registry to report outcomes for a range of approaches, but it also means that interpretation should account for the number of comparisons. The primary question concerns receipt within the appointment-linked outcome window; it is not a general measure of vaccination behavior in every setting or throughout the entire season.

The registry describes the broader aim as identifying which behavioral strategies may increase flu vaccination rates and whether responses differ by patient characteristics. The posted primary outcome results concern the overall comparison of each listed condition with the holdout control. Whether individual characteristics modify responses, and the details of any such analyses, are not reported in the registry record.

3. Trial Design

This was a randomized, double-masked, parallel-group trial with a behavioral intervention delivered through text messages. The registry lists University of Pennsylvania as the sponsor, identifies influenza as the condition, and marks the study as completed. The recorded start date is 2020-09-20 and the primary completion date is 2021-03-31. The phase is listed as NA.

The randomized groups comprised a holdout control and 19 behavioral message conditions. Participants assigned to the holdout control were to receive standard provider appointment reminders. The intervention conditions varied in content and timing. Examples described in the registry include a reservation framed as opt-out or opt-in, messages referring to regional vaccination rates, symbolic or herd-immunity framing, self-oriented or prosocial benefits, basic or vivid information, humor, healthy-habit questions, and reminders close to the appointment.

The primary outcome description says that messages were sent before participants’ first primary care appointment during the study period, called the target appointment. If a participant cancelled or did not attend and then scheduled another appointment during the study period, that new appointment became the target. A participant who cancelled or did not attend and did not schedule another appointment during the study period was counted as not receiving a flu shot. Participants vaccinated before receiving the SMS intervention were excluded from the primary analysis.

The registry records enrollment of 74811. Its summary says the study expected to include at least 80,000 participants. These figures describe enrollment and the stated expectation; they do not establish how many participants were included in each analysis after the endpoint-specific exclusions. The exact masking procedures, identities of the masked parties, sequence generation, allocation concealment, and handling of any deviations from assigned messages are not reported in the registry record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: A small number of patients in each experimental condition did not receive any intervention messages due to technical malfunctions. These patients are excluded from analyses since they were, for all practical purposes, not a part of of our study

Only the first 6 of 20 groups are shown.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Only the first 6 of 21 groups are shown.

6. Primary Endpoint

The registry lists the primary outcome as Number of Participants With Flu Shot Receipt at or Before Appointment, measured over The dependent variable will be assessed during a 4-day period after randomization starting three days prior to the participant's target primary care appointment through the date of the target appointment (including the date of the appointment).

The primary endpoint is the number of participants with a recorded flu shot at or before the target primary care appointment. The registry defines the observation window as a 4-day period after randomization, beginning three days before the target appointment and ending on the appointment date, including that date. Receipt was determined from electronic health records.

This endpoint is used to assess whether the messages were followed by vaccination in the period most directly connected to the appointment and intervention. It is a binary participant-level outcome: for the reported group counts, each participant is classified according to whether a qualifying receipt was recorded. The comparison can therefore be expressed as the difference in the proportions with recorded receipt between an intervention group and the holdout control.

The appointment-linked definition also shapes what the result represents. It does not count vaccination before the intervention, because those participants were excluded, and it does not treat cancellation or nonattendance as missing by default: under the registry definition, a participant without a replacement appointment during the study period was counted as not vaccinated for this endpoint. The measure is therefore an operational outcome that combines recorded vaccination with the appointment follow-up rules in the protocol description.

This endpoint does not measure all vaccination activity over the whole season. A separate secondary outcome counts flu-shot receipt before March 31, 2021, with a timeframe described as up to 6 months after randomization. The endpoint definitions should be kept distinct when interpreting the primary and secondary results.

7. Results Posted in the Registry

• Default Reservations Opt-Out Condition: 32.8% (1210/3688) versus 29.7% (1113/3742) on control; risk difference 3.1 percentage points (95% CI 1.0 to 5.2), risk ratio 1.10 (1.03–1.18); unadjusted two-proportion z-test p = 0.004.
• Default Reservations Opt-In Condition: 31.3% (1150/3670) versus 29.7% (1113/3742) on control; risk difference 1.6 percentage points (95% CI -0.5 to 3.7), risk ratio 1.05 (0.98–1.13); unadjusted two-proportion z-test p = 0.137.
• Intergroup Competition Treatment Condition: 31.5% (1163/3693) versus 29.7% (1113/3742) on control; risk difference 1.7 percentage points (95% CI -0.3 to 3.8), risk ratio 1.06 (0.99–1.13); unadjusted two-proportion z-test p = 0.102.
• Intergroup Competition Control Condition: 31.8% (1183/3717) versus 29.7% (1113/3742) on control; risk difference 2.1 percentage points (95% CI -0.0 to 4.2), risk ratio 1.07 (1.00–1.15); unadjusted two-proportion z-test p = 0.051.

8. Secondary Efficacy Endpoints

The registry also posts results for secondary outcomes. For each outcome below, the endpoint type, the test applied to the posted summary statistics, the result and its interpretation are given in turn. Secondary endpoints are usually not adjusted for multiple testing, so they support rather than confirm the primary finding.

Number of Participants With Flu Shot Receipt Before March 31, 2021

• Endpoint type: Binary: the number of participants with the event, compared as proportions.
• Test: Unadjusted two-proportion z-test, with a Wald 95% confidence interval for the risk difference.
• Result:
  • Default Reservations Opt-Out Condition: 37.7% (1391/3688) versus 34.6% (1296/3742) on control; risk difference 3.1 percentage points (95% CI 0.9 to 5.3), risk ratio 1.09 (1.03–1.16); unadjusted two-proportion z-test p = 0.006.
  • Default Reservations Opt-In Condition: 36.2% (1330/3670) versus 34.6% (1296/3742) on control; risk difference 1.6 percentage points (95% CI -0.6 to 3.8), risk ratio 1.05 (0.98–1.11); unadjusted two-proportion z-test p = 0.148.
• Interpretation:
  • The 95% confidence interval excludes no difference, so the proportion with the event is statistically higher for Default Reservations Opt-Out Condition than for the control. This is an unadjusted, secondary comparison, so it is suggestive rather than confirmatory.
  • The 95% confidence interval includes no difference, so these data do not show a statistically significant difference between Default Reservations Opt-In Condition and the control. The interval gives the range of effects that remain compatible with the data.

The secondary endpoint counts participants with flu-shot receipt before March 31, 2021, with a timeframe described as up to 6 months after randomization. It extends the observation beyond the appointment-linked primary outcome, so it addresses a broader period of recorded receipt. Its definition and timeframe differ from the primary endpoint and should not be treated as a direct substitute for the appointment-period result.

Arm-level counts are posted for the secondary endpoint across the listed conditions, but the registry lists formal comparison summaries only for Default Reservations Opt-Out and Default Reservations Opt-In. In those two comparisons, the opt-out estimate is positive with an interval above zero, whereas the opt-in interval includes zero. The remaining arm counts should not be treated as evidence of between-group differences in the absence of corresponding comparison estimates and uncertainty intervals. Raw counts are affected by arm size, so comparing counts alone would not provide a valid comparison of event frequency.

The secondary results are broadly relevant to the primary question because both outcomes concern recorded flu-shot receipt after randomization. However, the longer secondary timeframe can capture events that occur after the target appointment, and the posted record does not show whether the intervention groups continued to differ over that broader period for all conditions. The two available secondary comparisons do not establish a pattern across every strategy.

Secondary results generally carry less weight than a prespecified primary result because they may be more numerous, may be less central to the main research question, and can increase the opportunity for chance findings. The registry does not report a multiplicity adjustment for secondary comparisons or describe a hierarchy of outcomes. The record also lists no reported analyses. Thus, the secondary endpoint can add context, but the available comparisons should be treated as exploratory unless an analysis plan elsewhere establishes otherwise. Details of such a plan are not reported in the registry record.

Secondary outcomes registered for this trial, for reference (only those above were recalculated):

• Number of Participants With Flu Shot Receipt Before March 31, 2021

9. Statistical Interpretation

10. Statistical Methods Explained

For a binary outcome such as recorded flu-shot receipt, each participant is classified as having or not having the event. A group proportion is the fraction of participants in that group with the event. The registry reports event counts and group sizes, and the comparison summaries express the intervention-control contrast as a risk difference. A risk difference is the intervention-group proportion minus the control-group proportion, so a positive estimate means the observed proportion was higher in the intervention group. It is an absolute contrast, not a ratio.

The posted summaries also contain risk ratios and odds ratios. A risk ratio compares the group proportions as a ratio, while an odds ratio compares the odds of the event. These measures describe the same binary outcome on different scales and should not be read as interchangeable. The risk-difference results are the clearest direct expression of the absolute group contrast. The registry does not report a sponsor analysis model or state which measure was designated as the primary inferential contrast in a formal analysis plan.

What does a confidence interval express?

General statistical explanation: a confidence interval describes the uncertainty around an estimated effect under the method used to calculate it. A narrow interval indicates greater precision for that estimate than a wider interval. If a risk-difference interval includes zero, the data are compatible with no difference under the interval procedure; if it lies entirely on one side of zero, the estimate is less compatible with no difference at the corresponding confidence level. An interval is not a guarantee that the true effect lies within its endpoints, and it does not account for biases or design limitations that are not included in the calculation.

For these outcomes, zero is the reference point for a risk difference because zero corresponds to equal observed proportions. For a risk ratio or odds ratio, the no-difference reference is one. The primary displayed comparisons give risk-difference intervals and also report ratios, but the record does not provide a detailed account of the variance estimator or interval construction. The interval method is therefore not reported in the registry record.

How should p-values be understood?

General statistical explanation: a p-value measures how incompatible the observed data are with a specified null model, assuming that model and the analysis assumptions hold. It is not the probability that the null hypothesis is true, and it is not a measure of clinical importance. A smaller p-value signals stronger incompatibility with that null model under the calculation, but it does not correct for multiple comparisons unless an adjustment has been applied.

The comparison summaries label their p-values as unadjusted. That distinction matters because the study has many intervention-control contrasts. A conventional threshold such as 0.05 is sometimes used to flag results, but a threshold does not remove the risk of false-positive findings across a family of tests. The registry record does not report a multiplicity procedure, a hierarchical testing strategy, or a confirmatory rule for selecting among interventions. The posted p-values should consequently be read as unadjusted evidence for individual contrasts rather than as a controlled family of tests.

What can the group counts establish?

Group counts show how many participants had a recorded event in each arm, while arm sizes show the denominators used in the posted summaries. Counts alone do not describe the magnitude of a between-group difference unless the denominators are considered. The comparison summaries pair counts with proportions and effect estimates, allowing the reader to assess the direction and uncertainty of each contrast. The listed arm sizes are not necessarily the number included in every analysis, particularly because the endpoint description excludes people vaccinated before the intervention. The record does not separately report the final analysis population or an accounting of exclusions by group.

Random assignment can support a causal interpretation when the allocation process is implemented as intended and outcome ascertainment and analysis are appropriate. However, the registry’s design fields alone do not establish those conditions. Details of allocation concealment, missing-outcome handling, protocol deviations, and the statistical model are not reported in the registry record. The summaries should therefore be interpreted as randomized-group comparisons with the qualifications described in the endpoint definition, rather than as proof that all sources of bias were eliminated.

Why can extra variation matter?

General statistical explanation: for a binary outcome, the observed event rate varies from sample to sample. The uncertainty calculation depends on the data structure and assumptions about how observations were generated. If outcomes are more variable than the assumed model allows, intervals may be too narrow and tests may appear more decisive than warranted. This is often called over-dispersion in settings where a model assumes less variability than the data contain. The registry record does not report a dispersion estimate or an over-dispersion analysis for this binary endpoint, and no such conclusion should be inferred from the event counts alone.

Overall, the statistical summaries are useful for describing the observed differences and their nominal uncertainty. They do not replace a full analysis plan, and the record does not report adjusted analyses, subgroup results, or a method for controlling the error rate across the many comparisons.

11. Confidence Intervals and Precision

Precision is reflected in the width of an uncertainty interval as well as in the point estimate. The primary comparisons have intervals that are broad enough to include zero for several intervention conditions, while other intervals lie above zero under the displayed unadjusted calculations. An interval that crosses zero signals that the data do not pin down the direction of the risk difference with the same clarity as an interval wholly above or below zero. It does not prove that the groups are identical.

The control group has 3742 participants in the posted comparison summaries, and the intervention-group sizes vary across conditions. The event counts are also different by arm. Together, the sample sizes and observed event frequencies inform the precision of the estimates, but they do not guarantee narrow intervals or eliminate uncertainty. The risk-difference intervals should be considered alongside the estimates rather than relying only on whether a p-value falls on one side of a conventional threshold.

Precision is distinct from practical importance. A precisely estimated small difference can still have limited practical relevance, while an imprecise estimate can leave meaningful effects unresolved. The registry record does not specify a threshold for a difference considered important, nor does it supply a clinical interpretation threshold for the endpoint. It is therefore not possible to classify any observed risk difference as practically important from the record alone.

The reported intervals also describe uncertainty for each comparison individually. They do not, as posted, communicate uncertainty after accounting for the full set of intervention-control comparisons. The lack of a reported multiplicity adjustment does not change the numerical width of each displayed interval, but it changes how confidently a reader should interpret a collection of intervals as evidence across the study. In addition, the record does not report a full accounting of analyzed participants or detailed missing-data procedures, which limits assessment of whether the nominal precision reflects all relevant uncertainty.

For these reasons, the estimates are best read as condition-specific, unadjusted descriptions of the observed appointment-period outcome. Their intervals help distinguish better-resolved contrasts from those that remain compatible with a wider range of values; they do not settle which messaging strategy is best.

12. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations based on the observed effect, not the trial’s original design assumptions. The original sample-size justification is not reported in the registry record. For these binary outcomes, the planning figures are participants per arm, not total participants across both arms and not total events.

The stated planning assumptions are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. Each planning row considers an effect to detect equal to the stated fraction of the observed effect. Thus, the row for 100% uses the observed effect as the planning target; the rows for 75% and 50% consider smaller target effects. The figures are illustrations under those assumptions and should not be interpreted as the study’s actual enrollment target or as the number needed for an intervention to have a particular result.

The 80% power column gives the estimated participants per arm needed for an 80% chance of detecting the specified effect under the stated assumptions if that effect is real. The 90% power column gives the corresponding per-arm planning estimate for a higher chance of detection. These are prospective planning concepts, not probabilities that the trial’s result is correct. Smaller effects are harder to distinguish from random variation, so the planning figures increase as the target effect becomes a smaller fraction of the observed effect.

These calculations do not report the trial’s observed power. They also do not establish that any particular intervention effect is real, clinically important, or reliably detectable in this multi-arm setting. The assumptions use equal allocation for a two-arm comparison, whereas the actual trial has multiple intervention groups and a shared holdout control. The registry does not report how its original design accounted for the multi-arm structure, exclusions, or multiple comparisons.

13. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.297; treatment proportion p₁ = 0.328; average p̄ = 0.313.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.6557 = 1.2851.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.6553 = 0.5515.
4. n = (1.2851 + 0.5515)² / (0.0307)² = 3,589.0, rounded up to 3,589 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.297434526990914                          # control proportion
p1 <- 0.328091106290672                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(4565353)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 3,589 (80% power), 4,805 (90% power)
 75% of observed effect: n per arm = 6,339 (80% power), 8,485 (90% power)
 50% of observed effect: n per arm = 14,164 (80% power), 18,961 (90% power)
power.prop.test: 3589 
Simulated power at n = 3,589 per arm: 0.803 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

14. Multiplicity, Missing Data and Analysis Populations

The trial’s multi-arm structure creates a multiplicity issue: each of the 19 intervention conditions is compared with the holdout control, producing a set of related questions. When many tests are examined, the chance of at least one apparently unusual result can rise even if none of the interventions truly differs from control. This is a general statistical concern, not evidence that a particular posted result is false.

The comparison summaries report unadjusted p-values. The registry record does not report a correction for multiple comparisons, a family-wise error procedure, a false-discovery procedure, or a prespecified strategy for prioritizing contrasts. It also does not report a sponsor analysis plan explaining whether the main objective was to test each intervention, rank the strategies, or identify candidate messages for later testing. No conclusion about multiplicity control should be assumed.

The record lists zero reported analyses. It does not describe the statistical model, whether covariates were used, whether site was included in the analysis, or how any analyses by patient characteristics were specified. The study summary mentions interest in identifying which strategies work best for whom, but subgroup definitions, interaction tests, and results are not reported in the registry record. Those questions cannot be answered from the posted aggregate comparisons.

Missing-data handling and analysis populations also require caution. The primary endpoint description gives rules for cancellations, nonattendance, replacement appointments, and vaccination before receipt of the SMS message. However, the registry does not provide a participant flow by arm, counts excluded before analysis, or a full account of other missing electronic health record data. It is not reported whether there were additional exclusions, how incomplete outcome ascertainment was handled, or whether all analyses followed randomized assignment. These details matter because they affect the denominator and the interpretation of a randomized comparison.

The record states double masking but does not report who was masked or how masking was maintained. It does not describe allocation concealment or protocol deviations. Without these details, the design labels alone do not answer questions about implementation. The posted results should therefore be read as unadjusted summaries with important analysis and reporting details still unavailable in the registry record.

15. What the Result Does, and Does Not, Mean

The posted results describe differences in recorded flu-shot receipt between randomized SMS conditions and a holdout group during an appointment-linked period. Several unadjusted estimates are positive with intervals above zero, while other estimates remain compatible with no difference. This pattern means that the observed results are not uniform across all intervention conditions. It does not establish a definitive ranking of the strategies.

The outcome is a record of receipt, not a direct measure of intention, message engagement, knowledge, or reasons for vaccination. The trial description does not indicate whether participants read the texts, and the registry record does not report message delivery or response rates. It also does not establish that any observed difference was caused by a specific feature of the message, such as its framing or timing. Each condition may differ from another in more than one way, and direct intervention-to-intervention contrasts are not reported.

The estimates apply to the study population and endpoint rules recorded in the trial. Participants were patients of Penn Medicine and Geisinger with a target primary care appointment during the study period. Vaccination was captured through electronic health records, and the analysis excluded participants vaccinated before the SMS intervention. The rules for cancellations and missed appointments further define the population outcome. Generalization beyond this setting, patient group, and appointment-linked intervention is not established by the registry results.

The collection of unadjusted comparisons also limits how strongly individual results can be singled out. A small p-value is not proof of a real or important effect, and a result with an interval crossing zero is not proof that there is no effect. The registry does not report adjustment for multiple comparisons or a threshold for practical importance. It also does not provide results by individual characteristics, despite describing that as an aim in the study summary.

Accordingly, the posted data support a measured description of group outcomes and their uncertainty. They do not justify medical advice, a claim that a treatment works or does not work, or a conclusion that one message should be preferred in practice. Further interpretation would depend on analysis details and evidence not reported in the registry record.

16. Why This Trial Matters Statistically

Statistically, the trial is notable for putting many behavioral message strategies into a common randomized study rather than evaluating each in an isolated comparison. A shared holdout group and a consistent appointment-related endpoint create a framework for describing the observed outcomes across the approaches. The variety of interventions also makes the study relevant to questions about how message content and timing might be investigated comparatively.

The same structure raises important analytical challenges. There are many intervention-control comparisons, and the posted p-values are unadjusted. Without a reported multiplicity strategy, it is difficult to distinguish a robust pattern from findings that may arise when many contrasts are examined. Also, the posted results do not directly compare one intervention with another, so the largest point estimate cannot automatically be interpreted as the best strategy.

The study illustrates why effect estimates and intervals matter alongside significance tests. Risk differences communicate absolute group contrasts, while confidence intervals show how much uncertainty remains. The planning illustrations further show the general relationship between a smaller target effect and larger sample requirements, under stated assumptions. Those calculations are useful for understanding planning concepts but are not a substitute for the trial’s unreported original design justification.

Finally, the outcome’s operational definition is statistically consequential. It is tied to the target appointment, relies on electronic health records, excludes people vaccinated before the intervention, and classifies certain cancellations or missed appointments as not having received a shot. Clear endpoint rules make interpretation more concrete, while also defining the limits of what the result measures. This trial therefore highlights both the potential value and the interpretive discipline required in large, multi-arm behavioral studies.

17. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

Several limitations are specific to the study setting and endpoint. The intervention was delivered by SMS to patients scheduled for primary care appointments, and the primary outcome covers receipt at or before the target appointment. The results do not directly describe message effects in people without a scheduled appointment, people reached through other communication channels, or vaccination outside the endpoint window.

Outcome ascertainment relied on electronic health records. The registry does not report how complete those records were across sites or whether vaccinations received outside the participating health systems were captured. Any unrecorded vaccination could affect classification, but the extent of such outcome misclassification is not reported. The record also does not provide delivery, receipt, or engagement data for the messages.

The endpoint rules treat a participant who cancels or misses the target appointment and does not schedule another appointment during the study period as not having received a flu shot. That is the registry’s stated outcome definition, but it means appointment attendance and rescheduling are part of how the endpoint is operationalized. Participants vaccinated before receiving the SMS intervention were excluded, and the registry does not report the number of such exclusions by group or the final analysis population.

The trial included many distinct conditions, but the registry does not report multiplicity control, formal analyses for most secondary comparisons, or results of the stated interest in individual differences in response. It also does not report details of the original sample-size justification, analysis model, masking procedures, or missing-data methods. These gaps limit assessment of robustness, precision after accounting for the full analysis plan, and applicability beyond the study setting.

The registry’s summary expected at least 80,000 participants, while the recorded enrollment is 74811. The record does not explain the difference between that expectation and enrollment or state how it affected the original design. No conclusion about the reasons should be inferred.

18. Related Tutorials

19. Related Statistical Calculators

20. Sources

• ClinicalTrials.gov: Testing Multiple Behavioral Science Strategies to Increase Flu-Shot Rates — NCT04565353.
• Registry record last updated 2022-02-01; status last verified 2022-01.
• Results first posted 2021-08-11.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

21. Record Summary

The registry record describes NCT04565353, Testing Multiple Behavioral Science Strategies to Increase Flu-Shot Rates, as a completed randomized, double-masked, parallel-group study sponsored by University of Pennsylvania. It records enrollment of 74811, with a start date of 2020-09-20 and primary completion date of 2021-03-31. The phase is listed as NA.

The study compared 19 behavioral SMS conditions with a holdout control among Penn Medicine and Geisinger patients scheduled for primary care. The primary endpoint was recorded flu-shot receipt at or before the target appointment. The posted summaries show arm counts and unadjusted comparisons with risk differences, confidence intervals, and p-values. The record lists zero reported analyses and does not report a multiplicity adjustment, the original sample-size justification, or several analysis details.

A secondary outcome counts receipt before March 31, 2021, with a timeframe of up to 6 months after randomization. Arm counts are posted across the conditions, while comparison summaries are listed for two conditions. These results provide additional context but do not resolve the limitations in the primary analysis reporting.