Patient Portal Flu Vaccine Reminders_RCT 5 (LADHS): Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Patient Portal Flu Vaccine Reminders_RCT 5 (LADHS) (NCT05535777), a randomized na study. The primary outcome is Receipt of the Annual Influenza Vaccine Among Adult and Pediatric Index Patients.

2. Clinical Question

The trial addressed reminder delivery rather than comparing vaccines or vaccine formulations. All arms were intended for patients due for influenza vaccination. The enhanced approaches added a route for interaction with call-center staff: one through a phone callback after a text response, and the other through a back-and-forth text conversation. Standard reminders provided contact and self-scheduling routes without the described agent interaction.

The primary endpoint was the percentage of adult and pediatric index patients with annual influenza vaccination between 9/1/22 and 4/1/23. Vaccination status was assessed using electronic health record extraction together with external claims and pharmacy data. The registry record also lists receipt of annual influenza vaccine among pediatric index patients as a secondary outcome, but its result is not reported in the registry record.

The registry record identifies the study as a comparative effectiveness trial in Los Angeles Department of Health clinics. The central comparison is whether adding personalized follow-up to text reminders was associated with a different observed vaccination outcome than standard text reminders. That is a question about these reminder strategies and this enrolled patient population; it is not, by itself, a test of vaccine effectiveness or of clinical outcomes after vaccination.

3. Trial Design

The registry record describes a randomized, single-masked, parallel-group study. Enrollment was 246295, and the study status is completed. The record lists the start date as 2022-09-13 and primary completion as 2023-03-31. The listed phase is NA. The sponsor is the University of California, Los Angeles.

Patients were assigned to one of three reminder approaches. The callback arm received enhanced text reminders, with a phone call from a call-center agent if the patient pressed “1” in response. The agent’s stated role was to help schedule a clinical visit. The bidirectional-text arm offered a response from a call-center agent by text after the patient pressed “1”; the description says the agent could answer questions and help arrange an appointment through text conversation. The standard-text arm received reminders about influenza vaccination, with a clinic callback number and patient-portal self-scheduling described as ways to arrange vaccination.

Each arm’s description refers to up to three text messages. The brief summary says reminders were sent to patients who were due for influenza vaccine. The registry record does not state which party or parties were masked, so the meaning of single masking in practice is not reported in the registry record.

The primary outcome was assessed over six months and covered vaccination between 9/1/22 and 4/1/23. Its defined population included adult and pediatric index patients who were patients of the Los Angeles Department of Health system. For the primary analysis, patients were assigned through managed care plans, self-pay patients linked with a provider, or uninsured patients assigned to a provider. Individuals not affiliated with a primary care practice were excluded from the primary analysis.

The posted summary results give counts for each arm and unadjusted comparisons of each enhanced approach with standard reminders. The registry record does not report the original sample-size justification, the detailed randomization procedure, or the full operational description of the masking.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Receipt of the Annual Influenza Vaccine Among Adult and Pediatric Index Patients, measured over 6 months.

The primary endpoint is receipt of the annual influenza vaccine among adult and pediatric index patients. It is expressed as a percentage of patients meeting the vaccination definition during the stated interval, with a six-month timeframe. The record says vaccination status was assessed by extracting electronic health record data and using external claims and pharmacy data. Combining these sources is intended to capture recorded vaccinations that may not all appear in one clinical record system; the registry record does not give details about how conflicting or incomplete records were handled.

This endpoint is useful for evaluating a reminder strategy because it measures a patient action that the intervention is designed to prompt: receipt of vaccination. Comparing the outcome across randomized reminder groups can assess whether observed vaccination differed by assigned reminder approach within the trial population. It does not measure the quality of the text interaction, whether a patient read or answered a message, or the clinical consequences of vaccination.

The endpoint’s denominator is defined by the index-patient population eligible for the primary analysis, including the specified relationship to a Los Angeles Department of Health primary care practice. People without affiliation to a primary care practice were excluded from the primary analysis. The registry record does not report the number excluded for that reason or give further detail on endpoint adjudication.

A secondary outcome is listed for annual influenza vaccination among pediatric index patients. Its result is not reported in the registry record. The posted primary outcome should therefore be understood as the combined adult-and-pediatric index-patient result, rather than as a separate estimate for either age group.

7. Results Posted in the Registry

• Enhanced Text Reminders With Callback by Person: 67.1% (4641/6914) versus 66.6% (120137/180423) on control; risk difference 0.5 percentage points (95% CI -0.6 to 1.7), risk ratio 1.01 (0.99–1.03); unadjusted two-proportion z-test p = 0.352.
• Enhanced Bidirectional Text Reminders: 67.1% (4629/6901) versus 66.6% (120137/180423) on control; risk difference 0.5 percentage points (95% CI -0.6 to 1.6), risk ratio 1.01 (0.99–1.02); unadjusted two-proportion z-test p = 0.396.

8. Statistical Interpretation

9. Statistical Methods Explained

The posted summary compares a binary outcome: whether an index patient had an annual influenza vaccination recorded during the specified assessment period. For a binary outcome, a risk is the proportion of patients with the event in a group. A risk difference compares two group risks by subtracting one from the other; a positive estimate means the event proportion is higher in the enhanced-reminder group than in the standard-reminder group. The reported risk-difference intervals for both enhanced approaches include zero, the value corresponding to no difference between groups.

What does a confidence interval describe?

A confidence interval conveys the uncertainty around an estimate under the assumptions of the calculation. For these comparisons, the interval gives a range of risk differences compatible with the observed summary data and the method used. An interval that crosses zero includes both a possible positive and a possible negative difference, as well as no difference. It is not a probability statement that the true value lies in the particular calculated interval, and it does not account for sources of bias that were not measured or modeled.

How should the p-values be read?

The reported unadjusted p-values are 0.352 for enhanced texts with callback by a person and 0.396 for enhanced bidirectional texts. In general, a p-value measures how unusual data at least as incompatible with a specified null model would be if that model and the analysis assumptions held. It does not measure the size or importance of an effect, the probability that the null hypothesis is true, or the probability that a treatment works. Here, the p-values accompany unadjusted comparisons and should be interpreted alongside the estimates and intervals rather than as stand-alone conclusions.

What does unadjusted mean?

In general, an unadjusted comparison uses the observed group summaries without controlling for other variables in a regression or other adjustment model. Random assignment is intended to support fair comparisons, but unadjusted results do not account analytically for chance imbalances or for other sources of variation. The registry record does not report an adjusted model or a detailed statistical analysis plan. The posted risk differences and p-values are simple unadjusted recalculations from the reported summary statistics.

Why can the unequal group sizes matter?

The reported groups are markedly unequal in size: each enhanced-reminder group is smaller than the standard-text group. General statistical reasoning indicates that the precision of a comparison depends on the amount of information in both groups, not only on the larger group. A much larger control group cannot fully compensate for limited information in a smaller comparison group. This is relevant when reading the intervals and when considering whether a future study using a different allocation would have different precision.

What does randomization contribute?

In general, random assignment helps make groups comparable on average at the start of a trial, including with respect to factors that were not measured. It does not guarantee exact balance in a particular trial, and it does not remove concerns about missing outcome data, nonadherence, or departures from the assigned intervention. The registry record identifies randomized allocation but does not report the method used to generate or implement the assignment sequence.

What is the outcome count?

The arm results report participants and counts of participants with the vaccination event. For a binary endpoint, the group count and denominator describe the observed event frequency. These summaries do not provide individual-level information about timing, repeated contacts, message receipt, or the path by which a vaccination record was identified. The record describes use of electronic health records and external claims and pharmacy data, but data linkage and missing-record procedures are not reported in the registry record.

What are the relative measures?

General statistical interpretation distinguishes a risk difference from relative measures such as a risk ratio or odds ratio. A risk difference describes an absolute change in event proportion, while a risk ratio compares proportions multiplicatively and an odds ratio compares event odds. The registry results include relative-measure summaries, but their numerical values are not repeated. Each measure answers a different scale-specific question, and none resolves uncertainty or establishes causality beyond the design and analysis assumptions.

The trial’s posted summary does not report adjustment for multiple comparisons, handling of missing outcomes, an intention-to-treat definition, or a per-protocol analysis. Those omissions limit what can be inferred about the analytic population and the robustness of the estimates. The separate planning illustrations are not analyses of observed power; they use assumptions stated with those illustrations and are discussed separately.

10. Confidence Intervals and Precision

Precision concerns how narrowly the data locate an effect estimate. The confidence interval for enhanced texts with callback by a person extends from -0.006 to 0.017 for the risk difference. The interval for enhanced bidirectional texts extends from -0.006 to 0.016. Both span zero. Thus, each interval includes the possibility of a small negative difference, no difference, and a positive difference. The estimates are not precise enough to rule out differences in either direction within the respective intervals.

The point estimates are positive for both comparisons, but a point estimate is only the center of an uncertain range. It should not be interpreted independently of the confidence interval. The corresponding unadjusted p-values are 0.352 and 0.396. Those results are consistent with the intervals crossing zero; neither comparison gives clear evidence against a null difference under the unadjusted calculation.

Precision is influenced by the information in both comparison groups. In this trial, the standard-text arm is much larger than either enhanced-reminder arm. The larger control group contributes information, but the smaller enhanced groups remain relevant to uncertainty. The registry record does not provide a prespecified precision target or the original sample-size rationale, so it is not possible to judge the intervals against a stated design goal.

Confidence intervals describe statistical uncertainty under the calculation’s assumptions. They do not account for possible outcome misclassification, missing records, selection into the primary analysis, or differences in how the interventions were implemented unless those features are explicitly addressed. The registry record does not report enough detail to determine how such issues affected precision. The intervals are informative about sampling uncertainty in the posted unadjusted comparisons, but they are not a complete measure of uncertainty about generalizing the findings.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size illustrations are planning calculations based on the observed effect, not a report of the trial’s original design rationale. The assumptions are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. Each planning scenario uses the stated fraction of the observed effect as the effect to detect. The trial’s own original design assumptions and sample-size justification are not reported in the registry record.

The figures are participants per arm for 80% power and participants per arm for 90% power. In plain language, the 80% column gives the estimated number of participants needed in each of two equally sized groups for the calculation to have an 80% chance of detecting the assumed effect under the stated assumptions. The 90% column applies a higher chance of detection and therefore calls for more participants per arm for the same assumed effect. These are planning illustrations, not observed power.

Rows for 100% of the observed effect, 75% of the observed effect, and 50% of the observed effect illustrate how the calculation changes when the effect to detect is smaller than the observed estimate. The 100% rows list participants per arm for 80% power of 120,009 for callback and 144,375 for bidirectional texts; the 90% figures are 160,658 and 193,277, respectively. For 75% of the observed effect, the corresponding 80% figures are 213,568 and 256,906, and the 90% figures are 285,907 and 343,924. For 50% of the observed effect, the 80% figures are 481,018 and 578,575, and the 90% figures are 643,946 and 774,547.

The pattern illustrates a general planning principle: smaller effects are harder to distinguish from random variation and require more participants for a given power target. The calculations are based on the observed unadjusted effect and specified assumptions; they are not a substitute for a prospective design calculation using a justified target effect and the trial’s actual design features. The reported enrollment allocation was not equal, whereas these planning illustrations assume equal allocation.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.666; treatment proportion p₁ = 0.671; average p̄ = 0.669.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.6657 = 1.3048.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.6657 = 0.5603.
4. n = (1.3048 + 0.5603)² / (0.0054)² = 120,008.8, rounded up to 120,009 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.665862999728416                          # control proportion
p1 <- 0.671246745733295                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(5535777)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 120,009 (80% power), 160,658 (90% power)
 75% of observed effect: n per arm = 213,568 (80% power), 285,907 (90% power)
 50% of observed effect: n per arm = 481,018 (80% power), 643,946 (90% power)
power.prop.test: 120008.8 
Simulated power at n = 120,009 per arm: 0.801 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Multiplicity, Missing Data and Analysis Populations

The trial has two enhanced-reminder comparisons with the standard-text control, and a pediatric vaccination outcome is listed as secondary. Testing multiple outcomes or comparisons can increase the chance of at least one apparently notable result arising by chance. In general, multiplicity procedures can control that risk through prespecified adjustments or a defined testing hierarchy. The registry record does not report a multiplicity adjustment, testing hierarchy, or plan for interpreting the two primary comparisons jointly. The unadjusted p-values should therefore be understood as individual comparisons, not as values adjusted for the set of outcomes or contrasts.

The registry record lists no analysis entries. It does not report how missing vaccination information was handled, whether any patients were excluded because outcome data were unavailable, or what assumptions were used for incomplete records. Because vaccination status was determined using electronic health records and external claims and pharmacy data, incomplete linkage or unavailable records could affect outcome ascertainment; the extent of any such issue is not reported in the registry record.

The primary population excludes individuals not affiliated with a primary care practice in the system, as described in the endpoint definition. The record does not give the count or characteristics of excluded people, or clarify whether all randomized patients were included in the posted arm summaries and primary comparisons. An intention-to-treat definition, per-protocol population, and treatment of crossover or nonreceipt of messages are not reported in the registry record.

These omissions do not establish that the analyses were inappropriate, but they restrict independent evaluation of the analysis population and potential sources of bias. The posted comparisons are unadjusted recalculations from the summary results. No claim about adjustment, missing-data handling, or multiplicity control can be made from the registry record.

14. What the Result Does, and Does Not, Mean

The posted results describe recorded annual influenza vaccination in adult and pediatric index patients assigned to reminder strategies in the Los Angeles Department of Health system. The point estimates for both enhanced approaches are positive relative to standard text reminders. However, both confidence intervals include zero, and the unadjusted p-values do not provide clear evidence of a difference in these comparisons. The results are uncertain rather than definitive evidence of either an increase or an absence of effect.

The comparison concerns a package of communication features as implemented in the trial. For the callback approach, the offer of a call from an agent followed a text response. For bidirectional texting, agent assistance was offered through text conversation. The results do not isolate which feature—message wording, response option, human contact, scheduling assistance, or another aspect—may account for any observed difference. Nor do they show whether patients saw or responded to the messages.

The outcome is recorded vaccination, identified through electronic health records and external claims and pharmacy data. It does not measure all possible benefits or harms of vaccination, and it is not a direct measure of influenza illness. It also does not establish that a reminder strategy would have the same outcome in other health systems or in patients outside the defined primary-analysis population.

The primary outcome combines adults and children. Although a pediatric-only secondary outcome is listed, its result is not reported in the registry record. The primary comparisons therefore cannot establish an effect for children or adults separately. The record also does not report results by clinic, age group, or other patient characteristics.

These data should be read as a comparison of observed vaccination frequencies under the trial’s reminder assignments, with meaningful statistical uncertainty and limits in the analysis details available in the registry record. They do not justify medical advice, a claim that an approach works or fails, or a conclusion that the groups are equivalent.

15. Why This Trial Matters Statistically

This trial is statistically informative because it compares practical communication strategies using a binary outcome that can be assessed across a health system. It also illustrates a common challenge in comparative effectiveness research: the outcome may be relatively frequent, while the effect being studied is a modest difference between groups. Detecting smaller differences reliably generally requires more information than detecting larger ones, which is why precision and planning assumptions matter.

The allocation was unequal, with the standard-text group much larger than either enhanced-reminder group. This makes it important to consider the information contributed by both sides of each comparison rather than focusing only on total enrollment. The planning illustrations use equal allocation, so their assumptions differ from the allocation reported for the trial. Their role is to show how hypothetical study requirements change with effect size and power targets, not to reconstruct the trial’s original design.

The endpoint also raises issues of measurement and population definition. Vaccination status was assessed through several data sources, while the primary analysis excluded people without affiliation to a primary care practice. These features affect which patients the findings describe and how outcome ascertainment should be understood. The registry record does not provide enough analysis detail to assess missingness, subgroup effects, or the effect of data-linkage choices.

Finally, the positive point estimates and intervals crossing zero demonstrate why effect size, uncertainty, and analysis context should be considered together. A point estimate alone cannot settle whether a difference is distinguishable from chance variation, and a nonsignificant comparison does not establish equivalence. This is particularly relevant when evaluating interventions whose practical implementation may depend on staffing and communication workflows.

16. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record provides only limited detail on trial conduct and statistical analysis. Although the study is identified as randomized and single-masked, the masking details and randomization procedure are not reported in the registry record. The record does not provide a full statistical analysis plan, model specifications, or the original sample-size justification. It also lists no analysis entries.

The posted comparisons are unadjusted recalculations from summary statistics. The record does not report an adjusted analysis or provide information about baseline balance. It does not explain whether the reported analysis followed assignment regardless of intervention receipt, or how message nonresponse, intervention nonreceipt, or other departures were treated.

Outcome ascertainment drew on electronic health records and external claims and pharmacy data, but procedures for linking sources, resolving conflicting records, or handling unavailable information are not reported in the registry record. Missing outcome data and the number of people excluded from the primary analysis are also not reported. These gaps limit assessment of possible outcome misclassification and selection.

The primary analysis was restricted to index patients affiliated with a primary care practice in the system. Consequently, the results do not directly describe patients outside that defined group. The primary endpoint combines adults and pediatric patients, and the posted primary comparisons do not provide separate estimates by age group. Although a pediatric secondary outcome is listed, its result is not reported in the registry record.

The comparator was standard text reminders, so the results address the incremental comparison with that approach, not reminders versus no reminders. The trial record does not report details on how consistently staff delivered the enhanced follow-up, how often patients responded, or how many scheduling conversations occurred. Those implementation details are relevant to interpreting how the strategies operated in practice.

17. Related Tutorials

18. Related Statistical Calculators

19. Sources

• ClinicalTrials.gov: Patient Portal Flu Vaccine Reminders_RCT 5 (LADHS) — NCT05535777.
• Registry record last updated 2025-04-11; status last verified 2025-03.
• Results first posted 2025-04-11.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

20. Record Summary

The registry record describes a completed randomized, single-masked, parallel-group comparative effectiveness trial sponsored by the University of California, Los Angeles. Enrollment was 246295. The study compared enhanced text reminders with a callback by a person, enhanced bidirectional texts with call-center assistance, and standard text reminders in Los Angeles Department of Health clinics.

The primary endpoint was receipt of annual influenza vaccine among adult and pediatric index patients over six months, assessed using electronic health records and external claims and pharmacy data. The primary analysis excluded people not affiliated with a primary care practice in the system.

Both enhanced approaches had positive unadjusted risk-difference estimates against standard reminders, but each confidence interval crossed zero. The unadjusted p-values were 0.352 for callback and 0.396 for bidirectional texts. The registry record does not report an adjusted analysis, the original sample-size justification, missing-data procedures, or the result for the listed pediatric secondary outcome.