Patient Portal Flu Vaccine Reminders (5): Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Patient Portal Flu Vaccine Reminders (5) (NCT05525494), a randomized na study. The primary outcome is Number of Participants With Annual Influenza Vaccine Among Index Patients.

2. Clinical Question

3. Trial Design

This was a completed, randomized, single-masked trial with a factorial design. The registry describes a 2x2 nested factorial structure: patients were assigned to text-based reminders with a direct scheduling link, portal-based reminders with a direct scheduling link, or a control condition. Additional randomizations within the portal and text-message arms evaluated pre-appointment reminders. Within the portal-reminder arm, a pre-commitment question and tailored monthly messages were also compared with standard portal reminders. The posted results distinguish seven groups, including versions of the text and portal strategies with and without additional components, and a no-message group.

The registry lists enrollment of 262085. The study began on 2022-09-07, had primary completion on 2023-04-01, and is marked completed. The primary outcome timeframe is 6 months. The outcome description specifies vaccination during 9/1/22 - 4/1/23, ascertained using electronic health record vaccine data and external claims and pharmacy data.

The interventions were behavioral reminder strategies rather than medication assignments. The registry describes messages with direct appointment scheduling links and, in some groups, reminders associated with an upcoming appointment. The number of reminders could be up to 3 for several message strategies. Details beyond the descriptions posted for the groups, including the full sequence of participant contact and operational procedures, are not reported in the registry record.

Although the registry identifies the design as factorial and describes nested randomizations, the reported analyses do not include a complete account of the randomization structure or statistical model. The method used for masking, beyond the designation “single,” is not reported in the registry record. The trial's original sample-size justification and other original design assumptions are also not reported.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Only the first 6 of 7 groups are shown.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Only the first 6 of 8 groups are shown.

6. Primary Endpoint

The registry lists the primary outcome as Number of Participants With Annual Influenza Vaccine Among Index Patients, measured over 6 months.

The primary endpoint counts participants with an annual influenza vaccine among index patients. The registry describes the outcome as the percentage of patients with vaccination recorded between 9/1/22 and 4/1/23, with a timeframe of 6 months. Vaccination information was to be drawn from the electronic health record, external claims, and pharmacy data. Because the result is a count of participants with an event, it is a binary outcome at the participant level: each person is classified according to whether vaccination was recorded for the specified period.

This endpoint directly reflects the behavior the reminder strategies sought to encourage: receipt of influenza vaccination during the study season. Using multiple data sources may capture vaccinations that do not appear in the health system's own record, although the detailed process for combining or reconciling those sources is not reported in the registry record.

The primary analysis population was restricted to index patients who were active UCLA Health MyChart users, with at least one login in the stated 12-month window and excluding initial profile activation activity. People not affiliated with a primary care practice were excluded from the primary analysis. The registry does not describe the operational definition of an index patient beyond these eligibility details, nor does it report how missing or conflicting vaccination records were handled.

7. Results Posted in the Registry

• Text Fixed R/R Messages With Direct Appointment Schedule Link: 47.1% (20570/43668) versus 47.4% (20689/43682) on control; risk difference -0.3 percentage points (95% CI -0.9 to 0.4), risk ratio 0.99 (0.98–1.01); unadjusted two-proportion z-test p = 0.446.
• Portal Pre-Commitment Prompt, Tailored R/R Messages w/ Direct Scheduling + Pre-Appointment Reminders: 47.4% (10374/21881) versus 47.4% (20689/43682) on control; risk difference 0.0 percentage points (95% CI -0.8 to 0.9), risk ratio 1.00 (0.98–1.02); unadjusted two-proportion z-test p = 0.907.
• Portal Pre-Commitment Prompt With Tailored R/R Messages With Direct Appointment Schedule Link: 47.2% (10317/21876) versus 47.4% (20689/43682) on control; risk difference -0.2 percentage points (95% CI -1.0 to 0.6), risk ratio 1.00 (0.98–1.01); unadjusted two-proportion z-test p = 0.626.
• Portal Fixed R/R Messages With Direct Appointment Schedule Link + Portal Pre-Appointment Reminder: 47.4% (10369/21864) versus 47.4% (20689/43682) on control; risk difference 0.1 percentage points (95% CI -0.7 to 0.9), risk ratio 1.00 (0.98–1.02); unadjusted two-proportion z-test p = 0.880.

8. Statistical Interpretation

9. Statistical Methods Explained

The primary outcome is binary at the participant level: the record indicates whether an annual influenza vaccine was observed during the specified period. For binary outcomes, group comparisons can be expressed on absolute and relative scales. The posted unadjusted recalculations report risk differences, risk ratios, and odds ratios. Each scale has a different interpretation, so the values should be read as distinct descriptions of the same underlying comparison rather than as competing versions of one number.

What does a risk difference describe?

A risk difference subtracts one group's outcome proportion from the reference group's proportion. It is an absolute contrast, so zero represents no difference. Positive and negative values indicate opposite directions according to the order of groups in the comparison. In the registry's unadjusted comparisons, the risk-difference intervals include zero. This means the data are compatible with no absolute difference under the calculation's assumptions, as well as with some values on either side of zero.

How do risk ratios and odds ratios differ?

A risk ratio compares the outcome proportions by division; 1 represents equal proportions. An odds ratio compares the odds, rather than the proportions, and 1 represents equal odds. When outcomes are common, an odds ratio and a risk ratio can be numerically different and should not be interpreted as though they were the same measure. The registry's adjusted analyses are reported as risk ratios. The method used to adjust them is not reported in the registry record.

What do confidence intervals and p-values contribute?

A confidence interval conveys uncertainty around an estimated contrast. Its width gives a sense of precision, while its position relative to the null value helps show which effects remain compatible with the data under the calculation. For a risk difference, the null is zero; for a ratio measure, the null is 1. A p-value describes how unusual the observed data, or more extreme data, would be under a specified null model. It is not the probability that the null hypothesis is true, and it does not measure the size or practical importance of an effect. The posted recalculated p-values are unadjusted.

The registry does not report the specific model, standard-error method, covariate set, or treatment of the factorial assignment used for the sponsor's adjusted results. Consequently, the adjusted estimates cannot be independently reproduced from the posted summary counts alone. The unadjusted recalculations use summary statistics and are not substitutes for the sponsor's analysis. No discrepancy is flagged between the available results.

Why does the design matter for analysis?

In a factorial or nested design, some comparisons may reflect the effect of a message channel, while others may evaluate an added component within a particular channel. A direct comparison between two arms can therefore differ in its interpretation from a comparison that pools or groups several arms. The registry posts four adjusted analyses with different groupings but does not report the model or explain the estimands. The grouping definitions should be considered when interpreting each estimate; an overall statement about all message strategies may not correspond to any one reported contrast.

Multiplicity is relevant because several outcomes or contrasts can increase the chance of at least one apparently notable result when all are tested. The registry does not report a multiplicity strategy, such as a prespecified hierarchy or adjustment to significance thresholds. The displayed p-values are unadjusted, so they should not be interpreted as accounting for the full set of comparisons. The number of analyses posted does not by itself identify how many tests were planned or how the sponsor addressed them.

Finally, general statistical interpretation depends on assumptions about outcome classification, missing observations, and the analysis population. The registry names several sources for vaccination data and describes eligibility restrictions, but it does not report how missing records, conflicting sources, exclusions after randomization, or departures from assigned strategies were handled. Those omissions limit how fully the estimates can be evaluated from the registry record.

10. Confidence Intervals and Precision

Precision concerns how narrowly the data locate an estimated effect. A confidence interval gives a range of values compatible with the data under the method used; a narrow interval indicates greater precision than a wide one. It does not guarantee that the true effect lies within the interval, and it does not describe the probability that a particular effect is true.

For the unadjusted risk-difference comparisons, every posted interval includes zero. The estimates therefore do not rule out no absolute difference at the interval's stated level, and the intervals also allow effects in both directions. The posted p-values are 0.446, 0.907, 0.626, 0.88, 0.626, and 0.527. These are unadjusted values and should not be read as evidence that the strategies are equivalent. A failure to distinguish an effect from zero may reflect limited precision, a small underlying effect, or both.

The sponsor-reported adjusted estimates have 95% confidence intervals. Their displayed bounds are close to the null value of 1, and some include or touch that value at the reported precision. Rounded limits can conceal finer distinctions, so the displayed figures should not be used to infer more detail than they show. The adjustment method and uncertainty calculation are not reported in the registry record, which limits independent assessment of precision.

The posted sample-size illustrations indicate that detecting effects at a fraction of the observed effect can require substantially larger planning numbers. That pattern is expected: smaller target effects are harder to distinguish from random variation. These illustrations are conditional on their stated assumptions and are not a substitute for a precision assessment based on a fully specified original design and analysis plan.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size illustrations are planning calculations based on the observed effects in the posted unadjusted comparisons. They give the number of participants per arm, not a total across arms or total events. They use a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The target effect is the stated fraction of the observed effect: the full observed effect, 75% of it, or 50% of it. These are planning assumptions for the illustrations; the trial's own original design assumptions are not reported in the registry record.

The 80% power column gives the estimated participants per arm under those assumptions for a design with an 80% chance of detecting the specified effect if that effect is the true one. The 90% power column gives the corresponding planning number for a 90% chance. Higher target power generally requires more participants. A smaller target effect also requires more participants because it is harder to distinguish from random variation.

For example, for the comparison of text reminders without the added pre-appointment reminder against the reference group, the illustration based on 100% of the observed effect gives 590,851 participants per arm for 80% power and 790,981 participants per arm for 90% power. Planning for 75% of the observed effect gives 1,050,477 participants per arm for 80% power and 1,406,290 participants per arm for 90% power. The corresponding numbers vary across comparisons, reflecting the different observed effects and comparison proportions.

These are hypothetical planning figures, not observed power. They do not describe the probability that the completed trial detected an effect, and they are not the trial's enrollment targets or original justification. The registry does not report the trial's original sample-size calculation.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.474; treatment proportion p₁ = 0.471; average p̄ = 0.472.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.7060 = 1.3838.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.7060 = 0.5942.
4. n = (1.3838 + 0.5942)² / (-0.0026)² = 590,850.6, rounded up to 590,851 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.473627581154709                          # control proportion
p1 <- 0.471054318952093                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(5525494)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 590,851 (80% power), 790,981 (90% power)
 75% of observed effect: n per arm = 1,050,477 (80% power), 1,406,290 (90% power)
 50% of observed effect: n per arm = 2,363,740 (80% power), 3,164,377 (90% power)
power.prop.test: 590850.6 
Simulated power at n = 590,851 per arm: 0.802 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Analyses Reported by the Sponsor

The registry reports four sponsor analyses as adjusted risk ratios with 95% confidence intervals. The estimates are 0.99 (95% CI 0.98 to 1.01), 1.00 (95% CI 0.98 to 1.01), 1.01 (95% CI 1.00 to 1.02), and 1.00 (95% CI 0.99 to 1.01). Each is associated with a different grouping of trial arms. The registry does not identify the adjustment method, covariates, or model used, and does not report p-values for these analyses.

The recalculated comparisons use the posted outcome counts to form unadjusted contrasts. They are useful as a transparent check on simple group-level differences, but they do not reproduce the sponsor's adjusted analyses. Adjustment can change an estimate and its uncertainty, and the unreported methods prevent a direct like-for-like comparison. The adjusted risk ratios should therefore be described as the sponsor's own reported results, while the recalculations should be labeled unadjusted.

No discrepancy is flagged between the available results. Even so, the adjusted and unadjusted estimates should not be collapsed into one summary number: they use different reported scales and analysis approaches. More detail on the sponsor's model and the rationale for the specific groupings is not reported in the registry record.

14. Multiplicity, Missing Data and Analysis Populations

The trial involves multiple message strategies and nested components, and the registry posts several comparisons and adjusted analyses. When many hypotheses are evaluated, the chance of at least one result appearing statistically notable can rise. Methods such as a prespecified hierarchy, family-wise error control, or false-discovery control are general ways to address multiplicity. The registry record does not report whether any such approach was planned or used, nor does it identify a primary contrast among the posted analyses. The unadjusted p-values should therefore be interpreted as unadjusted evidence for their respective comparisons, not as multiplicity-controlled findings.

The registry names an analysis population for the primary outcome: active UCLA Health MyChart users meeting the stated login criterion, with people not affiliated with a primary care practice excluded from the primary analysis. It does not report a complete analysis-population definition for each posted contrast, including whether analyses followed original assignment or how exclusions after randomization were treated. The handling of nonadherence, message delivery failures, or crossovers is also not reported in the registry record.

Missing-data procedures are not reported. Although the outcome was to draw on electronic health records, external claims, and pharmacy data, the record does not explain how unavailable data, conflicting records, or vaccination outside those sources were handled. It also does not say whether missing outcome status was treated as unvaccinated, excluded, imputed, or addressed in another way. These choices can affect a binary outcome analysis and its interpretation.

The registry does not report the sponsor's statistical model, covariate adjustment set, standard-error method, handling of the factorial and nested randomizations, or any interim analyses. The study's original sample-size justification and analysis plan are not reported in the registry record. This limits assessment of whether the posted analyses match prespecified methods and whether multiplicity, missingness, and the analysis population were addressed as planned.

15. What the Result Does, and Does Not, Mean

The posted results describe the recorded influenza vaccination outcome among the registry's defined index-patient population during the stated season. They show that the observed arm-level counts were accompanied by unadjusted comparisons whose intervals include no absolute difference. The sponsor's adjusted risk-ratio estimates are close to 1, with confidence intervals near the null value. Taken together, the posted statistics do not identify a clear separation between the compared strategies in these analyses.

That interpretation is not the same as proving that all strategies produce identical outcomes. A confidence interval that includes the null remains compatible with a range of effects, and the unadjusted intervals permit values on both sides of no difference. The trial's estimates should be read with their uncertainty rather than as definitive evidence of equivalence. The registry does not report an equivalence or noninferiority framework, including a prespecified margin.

The results also do not establish why recorded vaccination outcomes were similar across groups. The registry does not report whether participants opened messages, used a scheduling link, received a message successfully, or obtained vaccination through a particular route. Such process measures could help explain the relationship between assignment and recorded outcomes, but they are not included in the posted results.

Interpretation is specific to the participating health system, its patient portal users, the eligibility restrictions, and the study season. It should not be generalized automatically to patients outside this analysis population, other health systems, different reminder approaches, or other seasons. The record's adjusted estimates are not fully reproducible because the adjustment methods are not reported, and the unadjusted recalculations do not account for that adjustment.

The registry lists zero recorded serious adverse events, other adverse events, and deaths in each group. This describes the posted safety counts, not a complete assessment of how adverse events were sought or classified. Safety ascertainment details are not reported in the registry record.

16. Why This Trial Matters Statistically

This trial is statistically informative because its design separates several practical features of reminder delivery: text versus portal messaging, direct scheduling links, tailored follow-up after a pre-commitment question, and a pre-appointment reminder. That structure makes the interpretation of each contrast important. An estimate for one nested component is not automatically an estimate of every reminder strategy combined, and the registry's different groupings need to be read as distinct comparisons.

The endpoint is common in the posted groups, and the reported contrasts are close to their null values. In this setting, even a large overall enrollment does not guarantee precise estimation of small differences in each nested comparison. The planning illustrations show how sample requirements grow when the target effect is smaller than the observed effect. This is a general feature of statistical planning, not evidence about the probability that the completed trial detected an effect.

The case also highlights why adjusted results, unadjusted recalculations, and design details should be kept distinct. Sponsor-reported adjusted risk ratios summarize analyses whose methods are not reported, while unadjusted comparisons are based on posted arm-level counts. The combination can be useful, but readers need the model, estimand, and multiplicity plan to evaluate how the analyses relate to the factorial design.

Finally, a registry outcome assembled from health records, claims, and pharmacy data raises practical questions about outcome capture and missing information. The record identifies the intended sources but not the rules for reconciling or missing records. Those details matter when interpreting whether the measured endpoint represents vaccination across the eligible population.

17. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

Several trial-specific details needed for a full evaluation are not reported in the registry record. The primary analysis is restricted to active patient-portal users and excludes people not affiliated with a primary care practice. Results therefore describe that defined group, not necessarily all patients who might be eligible for influenza vaccination. The registry does not report further participant characteristics that would help assess how representative the analyzed population is.

The outcome was to be assembled from electronic health records and external claims and pharmacy data, but data-linkage procedures, coverage, and rules for resolving inconsistent records are not described. The treatment of missing vaccination status is also not reported. These matters are particularly relevant to interpreting a binary outcome based on records from multiple sources.

The registry describes a factorial design with nested randomizations, but the posted adjusted analyses use selected groupings whose rationale and statistical model are not reported. It is unclear from the record how the sponsor handled the factorial structure, adjusted for any baseline variables, accounted for multiple comparisons, or defined the analysis population for each contrast. The original sample-size justification and interim-analysis plan are not reported in the registry record.

The intervention descriptions indicate that messages could include scheduling links and that some participants could receive up to 3 reminders. The record does not report message receipt, engagement, scheduling-link use, or other process outcomes, so the pathway from assignment to recorded vaccination cannot be assessed from these results. Nor are details of the masking procedure beyond the “single” designation reported.

Finally, the registry lists zero adverse events in the posted categories but does not describe how events were collected, monitored, or classified. The zero counts should be read as registry entries, not as proof that events were systematically assessed and absent.

18. Related Tutorials

19. Related Statistical Calculators

20. Sources

• ClinicalTrials.gov: Patient Portal Flu Vaccine Reminders (5) — NCT05525494.
• Registry record last updated 2024-08-07; status last verified 2024-07.
• Results first posted 2024-08-07.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

21. Record Summary

The registry describes a completed randomized, single-masked factorial trial of text and patient-portal influenza-vaccine reminders at UCLA Health clinics in Los Angeles. Enrollment is listed as 262085. The study began on 2022-09-07, with primary completion on 2023-04-01.

The primary outcome was participants with annual influenza vaccination over the stated 6-month timeframe, assessed using electronic health record, claims, and pharmacy data. The population was restricted to active UCLA Health MyChart users meeting the login criterion, and people without a primary care practice affiliation were excluded from the primary analysis.

Posted results include arm-level vaccination counts, unadjusted comparisons, and four sponsor-reported adjusted risk ratios with 95% confidence intervals. The unadjusted intervals include no risk difference, while adjusted estimates are close to 1. The registry does not report the adjustment methods, multiplicity strategy, missing-data procedures, or original sample-size justification.