The Doctors for Coronavirus Prevention Project Thanksgiving / Christmas Messaging Campaign: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for The Doctors for Coronavirus Prevention Project Thanksgiving / Christmas Messaging Campaign (NCT04644328), a randomized na study. The primary outcome is Facebook Movement Metric.

2. Clinical Question

3. Trial Design

The registry record describes a randomized, unmasked study with a sequential model. It lists the study as completed and records enrollment of 820. The sponsor is the National Bureau of Economic Research, Inc. The recorded start date is 2020-11-12, and the primary completion date is 2021-02-11. The record lists the phase as NA.

The intervention was behavioral advertising on Facebook. The high-intensity treatment arm received approximately 3 physician-recorded video ads over a 2-week period. The low-intensity control arm did not receive ads containing short physician-recorded videos. These are descriptions of the campaign conditions; the record does not establish that the control arm received no other information or messaging.

The arm descriptions refer to counties and describe randomization in relation to counties, including exclusions related to data limitations and the holiday periods. The descriptions in the registry record are incomplete, so the full allocation procedure, unit of randomization, and any differences in implementation between the holiday periods are not reported in the registry record. The study is marked as unmasked.

The registry record lists six outcome measures and several secondary outcomes concerning recall of the ad, travel intentions, mask wearing, beliefs about travel, and mobility. It records no analyses. The primary endpoint has several categories or time points and is not reduced to a single comparison in the registry results. Consequently, the record does not give a numerical treatment estimate, a confidence interval, or a test result for the movement endpoint.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: The 13 states which centrally report COVID-19 cases at the ZCTA level contained 829 counties. At Thanksgiving, 9 counties were excluded due to data limitations; at Christmas, 62 counties were excluded due to data limitations and potential negative impacts of treatment in rural, conservative areas resulting from polarization in the wake of the 2020 presidential election.

Thanksgiving Period

Christmas Period

Analysis population for the primary outcome, as defined in the registry: Rural, conservative counties were excluded during Christmas due to potential negative impacts of treatment resulting from polarization in the wake of the 2020 presidential election. The two periods have been entered as rows to avoid double-counting counties.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: The experiment was run during two periods: Thanksgiving (Period 1) and Christmas (Period 2). During Thanksgiving, 820 counties were analyzed (410 high-intensity, 410 low-intensity). Of these counties, 767 were analyzed during Christmas (386 high-intensity, 381 low-intensity). Fewer counties were analyzed at Christmas because we removed counties in rural, conservative areas from the study due to potential negative impacts resulting from polarization in the wake of the 2020 presidential election.

6. Primary Endpoint

The registry lists the primary outcome as Facebook Movement Metric, measured over November 26, 2020 (Thanksgiving); December 24-25, 2020 (Christmas); February 2-29, 2020 (Baseline benchmark).

The primary endpoint is the Facebook Movement Metric, expressed as percent change relative to February 2020. It measures change in the distance covered in a day compared with the same day of the week in the benchmark period of February 2-29, 2020. The measure is defined for people who started the day in a particular location.

The registry record identifies November 26, 2020 as the Thanksgiving date and December 24-25, 2020 as the Christmas period for the endpoint. It defines holiday travel as travel during the three days preceding each holiday. Because the mobility data describe behavior throughout the day among people who were in each county that morning, the record says that the campaign’s potential effect on travel away from home could be captured, but return travel after the holiday could not. The campaign was targeted based on home location.

Using a movement measure provides an indicator of observed mobility, rather than a direct measure of infection or individual health. Comparing movement with a pre-pandemic benchmark gives context for changes relative to that period, but the benchmark is not itself a concurrent randomized control group. The registry record does not report the endpoint estimates or explain the precise aggregation and analysis procedures used to produce them.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The registry record identifies the allocation as randomized and the masking as none. Random allocation is a method for assigning study units to conditions so that, in expectation, pre-existing factors are not systematically chosen to favor one condition. It does not guarantee that all characteristics will be balanced in a particular study. The record does not provide enough detail to establish the exact allocation procedure or the unit randomized. The arm descriptions discuss counties, but the complete randomization description is not reported in the registry record.

What does the mean represent?

The primary parameter is a mean of a continuous measure. The measure is the percent change in distance covered in a day relative to the same day of the week in the February 2-29, 2020 benchmark period. A mean summarizes the central value of the measured movement outcome across the observations included in an analysis. It does not describe every person’s movement, and it can conceal variation among counties or individuals. The registry record does not state the aggregation details or the weighting used to calculate the mean.

The benchmark is a reference period, not a concurrently assigned control condition. A comparison against a benchmark can describe how movement differs from that reference, but interpretation depends on the definition of the measured population, the benchmark, and the analysis. The record defines travel as movement during the three days preceding each holiday. It also explains that the mobility data cannot measure return travel after the holiday, because the data follow the behavior of people who were in a county at the start of the day, while campaign targeting was based on home location.

How should the confidence interval be understood?

The registry record names a 95% confidence interval as the dispersion measure. In general, a confidence interval gives a range of estimates compatible with the observed data and the statistical procedure, under the method’s assumptions. A narrower interval generally indicates greater precision than a wider interval, although precision is not the same as freedom from bias. The registry record does not include interval bounds, so the actual precision of the estimated mean cannot be judged. It also does not state the exact variance estimator or other interval-construction details.

Randomization and statistical uncertainty address different issues. Randomization concerns how treatment conditions are assigned; a confidence interval concerns uncertainty in an estimated quantity under an analysis. Neither alone resolves measurement limitations, missing data, or departures from the analysis assumptions. The registry record does not describe those issues in sufficient detail to assess how they were handled.

What does the sequential model tell the reader?

The registry record labels the model as sequential. That label alone does not specify how observations were ordered, how the model was fitted, what variables it included, or whether the analysis was adjusted. Those details are not reported in the registry record. No particular statistical procedure should be inferred from the model label alone.

The study is unmasked, meaning masking is recorded as none. Lack of masking does not by itself determine the validity of a result, but it can matter when outcomes involve judgment or behavior affected by awareness of assignment. The primary movement endpoint is described as a Facebook-derived measure; the record does not give enough information to evaluate data coverage, classification, or measurement error by arm.

What can planning calculations show?

The record describes sample-size planning illustrations based on the observed effect, using a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The target effect is described as a stated fraction of the observed effect. Such calculations are prospective planning illustrations: they indicate the sample size associated with specified assumptions. They are not observed power and do not demonstrate whether a treatment effect exists. The registry record does not report the observed effect or numerical planning results in the information available for this interpretation.

In general, power depends on the target effect, outcome variability, analysis method, allocation, and significance criterion. Planning based on an observed effect can be sensitive to the uncertainty in that estimate. It should not be read as independent confirmation of the original finding. Because the underlying outcome estimates and their uncertainty are not reported in the registry record, the planning assumptions cannot be evaluated against a reported result.

Why do multiple periods and outcomes matter?

The primary outcome includes several categories or time points, and the record lists secondary outcomes related to message knowledge and mobility. When a study examines multiple outcomes or comparisons, the chance of finding at least one apparently unusual result can increase if each is tested separately without accounting for multiplicity. General approaches include defining a primary comparison in advance, adjusting inference across a family of tests, or clearly treating secondary analyses as exploratory. The registry record does not report a multiplicity strategy or a complete statistical analysis plan, so it is not possible to determine how the several periods and outcomes were handled.

10. Multiplicity, Missing Data and Analysis Populations

The registry record lists a primary movement outcome with several categories or time points, as well as secondary outcomes concerning recall of the ad, intent to travel, mask wearing, beliefs about travel, and mobility measures. It records six outcome measures but no analyses. Because multiple outcomes and time-specific comparisons can create multiple opportunities for an apparently notable result, an analysis plan typically clarifies which comparisons are primary and how multiplicity is addressed. A multiplicity adjustment or prespecified hierarchy is not reported in the registry record.

Missing-data handling is not reported in the registry record. The outcome description acknowledges limits in the mobility data: return travel after a holiday could not be computed, and the record notes exclusions for data limitations at Thanksgiving and exclusions of rural, conservative counties during Christmas because of concern about potential negative impacts of treatment amid polarization following the 2020 presidential election. The registry record does not describe how those exclusions affected the analysis, whether additional observations were unavailable, or how missing data were handled.

The analysis populations are also not reported in the registry record. The enrollment field records 820, but the record does not explain how that enrollment relates to the county-level descriptions or to the observations contributing to each endpoint estimate. It does not state whether analyses followed assigned groups, included only observations with available mobility data, or used another population definition. Without those details, readers cannot assess exclusions after randomization, the analysis denominator, or how the analytic population corresponds to the intended study population.

11. What the Result Does, and Does Not, Mean

The registry record defines a randomized campaign evaluation and an outcome based on movement relative to a February 2020 benchmark. It does not report the numerical treatment comparison, the confidence interval bounds, or an analysis result. The evidence in the registry record therefore supports a description of the intended comparison and measurement, not a conclusion about the size or direction of any campaign-associated change in movement.

The endpoint is a mobility measure. It is not a measure of infection, illness, or individual clinical outcomes. Even a clearly estimated difference in movement would need to be interpreted as a difference in the measure used, rather than automatically as evidence of a change in health outcomes. The record does not provide results for the listed knowledge, travel-intention, mask-wearing, belief, or additional mobility outcomes.

The benchmark comparison is also distinct from a comparison with a contemporaneous non-campaign group. The study does describe randomized treatment and control arms, but the numerical arm comparison and its analysis are not reported in the registry record. The available definition also limits the travel measure to the period before each holiday; it does not capture return travel after the holiday.

These boundaries are important: the absence of results in the registry record is not evidence that the campaign had no effect, just as the design description is not evidence that it did. Statistical interpretation requires effect estimates, uncertainty, and sufficient analysis details. Those are not reported in the registry record. No medical or behavioral recommendation follows from the design description alone.

12. Why This Trial Matters Statistically

This study is statistically notable because it applies a randomized campaign design to a population-level mobility question during two holiday periods. The intervention used physician-recorded Facebook ads, while the comparator did not receive those ads. The primary endpoint is a continuous movement measure expressed relative to a pre-pandemic benchmark, which makes clear how the outcome was operationalized while also raising important questions about the meaning of a benchmark-based change.

The registry description also illustrates that mobility outcomes depend on where people are observed and when. The measure follows people who started the day in a particular location, and the registry explains why return travel after a holiday could not be captured. That detail is central to interpreting the scope of the endpoint: measured movement away from home is not a complete account of all travel related to a holiday.

Several analytical questions remain important for readers: the exact randomization procedure, the analysis unit, the handling of exclusions and unavailable data, and the approach to multiple time points and outcomes. Those aspects affect how estimates should be interpreted, but they are not reported in the registry record. The record also contains no analyses, so the educational value lies in distinguishing a defined outcome and trial design from evidence about an estimated effect.

13. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record does not provide numerical estimates for the primary movement endpoint, confidence interval bounds, or analysis results. It therefore does not permit an assessment of effect direction, size, or precision. The record’s note that the primary outcome has several categories or time points and is not reduced to a single comparison further limits interpretation of the posted results.

The outcome captures distance covered during a day by people who started that day in a particular location. As the record explains, it cannot measure return travel after the holiday. It is also a movement measure relative to the February 2-29, 2020 benchmark, rather than a direct clinical outcome. The registry record does not report details of the measurement process, data coverage, or aggregation used to create the endpoint.

The population description notes exclusions during the holiday periods: counties were excluded at Thanksgiving due to data limitations, and rural, conservative counties were excluded during Christmas because of concern about possible negative impacts of treatment related to polarization after the 2020 presidential election. The registry record does not explain the full implications of these exclusions for the target population or the analysis.

Other limitations include the absence of a reported statistical analysis plan, multiplicity approach, missing-data method, and analysis-population definition. The allocation description is incomplete, and the record does not state the precise unit randomized or how the listed enrollment relates to endpoint observations. Masking is recorded as none. These uncertainties make it difficult to evaluate the analysis, representativeness, and precision from the registry record alone.

14. Related Tutorials

15. Related Statistical Calculators

16. Sources

• ClinicalTrials.gov: The Doctors for Coronavirus Prevention Project Thanksgiving / Christmas Messaging Campaign — NCT04644328.
• Registry record last updated 2021-12-21; status last verified 2021-12.
• Results first posted 2021-12-21.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

17. Record Summary

The registry record describes a completed randomized, unmasked study of Facebook ads with physician-recorded videos focused on holiday travel and mask use during the COVID pandemic. The intervention arm received approximately 3 ads over a 2-week period; the control arm did not receive ads containing short physician-recorded videos. The record lists enrollment of 820 and identifies the model as sequential.

The primary outcome is a mean Facebook movement metric, measured as percent change in daily distance relative to February 2-29, 2020. The endpoint description covers Thanksgiving and Christmas and specifies that holiday travel was defined as travel during the three days preceding each holiday. The record notes that return travel could not be measured and describes county exclusions for data limitations and concerns about potential negative treatment impacts.

Six outcome measures are listed, but the registry record reports no analyses or numerical outcome estimates. Details of the statistical analysis, missing-data handling, multiplicity, and analysis populations are not reported in the registry record.