Using Behavioral Economics to Enhance Appointment Reminders and Reduce Missed Visits: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Using Behavioral Economics to Enhance Appointment Reminders and Reduce Missed Visits (NCT03850431), a randomized na study. The primary outcome is No-Show Rate.

2. Clinical Question

3. Trial Design

The registry describes a randomized, parallel-group trial with no masking. The intervention was delivered through clinic-assigned appointment letters, and the study is identified as cluster-randomized. In a cluster-randomized design, groups such as clinics, rather than individual participants, are assigned to study conditions. Outcomes from people within the same clinic may be more alike than outcomes from people in different clinics, so the analysis needs to account for the design’s grouping structure.

The behavioral reminder approaches included social norms and behavioral instructions; caring and consequences for others with behavioral instructions; caring and consequences for self with behavioral instructions; and a combination of all the listed nudges. Usual care used a letter containing basic appointment information. The registry describes the study’s aims as developing behavioral-economics-informed messages, evaluating enhanced reminders, and identifying potential barriers and facilitators to widespread implementation.

The registry lists enrollment of 34516, a start date of 2019-12-17, primary completion on 2021-10-14, and a completed status. The results note that the number analyzed for the primary outcome exceeds the Participant Flow total because the analyses include additional pooled Primary Care and Mental Health intervention groupings. Those pooled groupings were not arms to which participants were individually randomized. The registry does not report the details of clinic allocation procedures, the number of clinics assigned to each condition, or the method used to account for clustering in each regression analysis.

The primary endpoint timeframe was 1 year. Cancellation Rate was listed as a secondary outcome. The registry lists 2 outcome measures and 20 analyses. It does not report a detailed statistical analysis plan or explain how all of the listed comparisons correspond to specific intervention contrasts in human-readable terms.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Only the first 6 of 10 groups are shown.

Overall Study

Analysis population for the primary outcome, as defined in the registry: The number of participants analyzed is greater than listed in Participant Flow because these results include two additional arms listed for analysis: all 4 Primary Care intervention arms, and all 4 Mental Health intervention arms. These were not individual arms to which participants were randomized but rather a combination of arms for purposes of analysis. The number of appointments analyzed is greater than the number of participants because some participants had multiple appointments.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: Some participants had appointments with clinics that were randomized to different arms. In this table, participants are only counted once within the 5 primary care arms and once within the 5 mental health arms, according to their first primary care or mental health appointment. This table's total column includes the 4014 participants who had appointments in both primary care and mental health twice - once in each. Summary statistics for age \& gender are based on the 32,946 unique participants.

Only the first 6 of 11 groups are shown.

6. Primary Endpoint

The registry lists the primary outcome as No-Show Rate, measured over 1 year.

The primary endpoint was the percentage of total appointments classified as a no-show over 1 year. The registry defines the numerator as appointments marked as a no-show and appointments canceled by the patient or clinic after the appointment time. The denominator consists of no-shows and completed appointments. The endpoint is therefore an appointment-level measure of missed visits, not a count of unique participants who missed an appointment.

The registry also states that canceled appointments are not included in the number of units analyzed for calculation of missed appointments. This wording distinguishes cancellations from the scheduled appointments used in the no-show calculation, while also specifying that cancellations after the appointment time are counted in the numerator. The exact handling of any other cancellation circumstances is not reported in the registry record.

A no-show percentage is useful for describing the frequency of missed appointments relative to scheduled appointment activity. It can be clinically and operationally relevant in a study whose stated focus is appointment attendance and access. The posted primary analyses, however, are logistic-regression odds ratios, not direct reports of the endpoint percentage in each arm. They therefore describe relative odds in the coded comparisons and do not, on their own, show the absolute no-show rate or the absolute difference in rates between groups.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

What does an odds ratio express?

General explanation: An odds ratio compares the odds of an outcome between two groups. Odds are related to, but are not the same as, the probability or percentage of an outcome. An odds ratio of 1 represents equal odds in the groups being compared. An odds ratio above 1 represents higher odds in the group used as the numerator of the comparison; an odds ratio below 1 represents lower odds. To say which reminder group had higher odds requires knowing the ordering and labels used for the comparison. The registry’s reported OG codes do not identify those groups in human-readable terms in the analysis entries, so the direction cannot be assigned to a named reminder approach with confidence.

What does logistic regression do?

General explanation: Logistic regression is a method for modeling an outcome with two categories, such as whether an appointment was classified as a no-show. The model relates the outcome to group membership and any other terms included in the model. The registry identifies the method as logistic regression, but does not report the full model formula, covariates, or any adjustment details. It is therefore not possible to determine from the registry record whether the posted estimates are adjusted, or exactly which factors were included.

How should a confidence interval be read?

General explanation: A confidence interval conveys uncertainty around an estimated effect. A narrower interval indicates greater precision than a wider interval, while an interval that includes the null value leaves the direction and size of the effect uncertain. For an odds ratio, the null value is 1. All 10 reported 95% confidence intervals include 1, and the intervals extend to both sides of that value. The posted estimates thus do not rule out equal odds, or differences in either direction, at the interval’s stated confidence level.

What does the p-value add?

General explanation: A p-value describes how compatible the observed data are with a specified null model, given the assumptions of the analysis. It is not the probability that the null hypothesis is true, and it does not measure the clinical importance of an effect. The posted p-values are 0.145, 0.298, 0.127, 0.286, 0.183, 0.209, 0.463, 0.074, 0.306, and 0.373. None is below 0.05. These results do not provide strong evidence against equal odds under the reported tests, but they do not establish that the reminder groups are equivalent.

Why does the cluster design matter?

General explanation: When clinics are randomized, observations from the same clinic may share characteristics and outcomes. Treating every appointment as fully independent can make standard errors too small and confidence intervals too narrow if within-clinic similarity is not handled appropriately. The registry explicitly notes that a simple individual-level re-analysis would be misleading and indicates that only the sponsor’s own analysis is shown. It does not describe the cluster-adjustment method, so the details needed to assess this aspect of the models are not reported in the registry record.

Over-dispersion is a general statistical concern when observed variation exceeds what a model assumes. Accounting for extra variation can affect uncertainty estimates. The registry record does not report whether over-dispersion was assessed or how it was addressed, so no conclusion about that modeling issue can be drawn from the reported information.

What can be concluded from the comparisons?

The results report relative odds, not the no-show percentages in each group. Without those percentages, the baseline level and absolute size of any difference are not available. The coded comparison groups also limit interpretation of which specific behavioral content is represented by each estimate. The trial’s cluster-randomized design is relevant to the validity of the analyses, but the registry does not supply enough model detail to independently assess the handling of clustering. The estimates and their uncertainty should therefore be read as the sponsor’s reported comparisons, without translating them into absolute attendance changes or attributing them to a particular nudge combination.

10. Analyses Reported by the Sponsor

The registry reports the sponsor’s logistic-regression odds ratios, confidence intervals, and p-values for the primary outcome. It does not report a separate recalculation of those results. The registry also identifies the study as cluster-randomized and explicitly cautions that a simple individual-level re-analysis would be misleading; only the sponsor’s own analysis is shown. Accordingly, there is no independent unadjusted estimate to compare with the sponsor’s analysis in the posted results.

This distinction matters because appointments within the same clinic may not be statistically independent. A recalculation based only on summary statistics that ignored clinic-level clustering could give misleading uncertainty estimates. The method the sponsor used to account for clustering, along with the full model specification, is not reported in the registry record. The sponsor’s analysis therefore takes precedence over a simple individual-level recalculation, but the available record does not provide enough detail to independently evaluate its modeling choices.

The sponsor-reported confidence intervals all include the odds-ratio null value of 1, and the reported p-values are all greater than 0.05. Those features describe the precision and statistical evidence in the posted analyses; they do not establish equivalence. Because the analysis entries use coded groups without a human-readable mapping, interpreting a specific estimate as a comparison between a named nudge and usual care is also not supported by the entries alone.

11. Multiplicity, Missing Data and Analysis Populations

The registry lists 20 analyses and 2 outcome measures, with No-Show Rate as the primary outcome and Cancellation Rate as a secondary outcome. It reports 10 logistic-regression comparisons for the primary outcome in the results. Conducting multiple comparisons raises the possibility of finding an apparently notable result by chance, even when no group differences exist. Statistical approaches to address that issue can include prespecifying a primary contrast or adjusting inference across a set of tests. The registry does not report whether any multiplicity strategy was planned or applied, how the 20 analyses relate to the reported comparisons, or whether the reported p-values were adjusted.

The registry record does not report how missing outcome data were defined or handled, whether any appointments or clinics were excluded, or whether sensitivity analyses addressed missingness. The primary endpoint description gives rules for classifying no-shows, completed appointments, and some cancellations, but it does not describe the handling of all possible incomplete or unavailable appointment records. No conclusions about missing-data bias can be made from the posted information.

The analysis population requires particular care. The registry states that the number analyzed for the primary outcome is greater than the Participant Flow total because the results include two additional arms listed for analysis: all 4 Primary Care intervention arms and all 4 Mental Health intervention arms. These pooled analyses were not individual arms to which participants were randomized. The exact number analyzed in each comparison and the mapping between the coded analysis groups and the named trial arms are not reported in the analysis entries. It is therefore important not to interpret every analysis grouping as a randomized arm or assume that each contrast represents a direct comparison of one named intervention with usual care.

The registry does not report an interim-analysis plan, stopping rules, a detailed prespecified analysis plan, or a full account of analysis-population definitions. These omissions limit assessment of the relationship between planned and reported analyses. They do not, by themselves, demonstrate that analyses were improperly conducted; they mean that those details are not reported in the registry record.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

Most frequent non-serious adverse event terms

The registry reports serious adverse event counts of 0 in each of the five listed groups: Usual Care, Caring + Consequences for Self + Behavioral Instructions, Caring + Consequences for Others + Behavioral Instructions, Social Norms + Behavioral Instructions, and Combination of All Nudges. The listed number at risk differs by group, while each group has 0 serious adverse events reported. These figures describe the recorded events for the listed groups and should not be interpreted as proof that no serious events occurred outside the registry’s reported data.

For other adverse events, the registry reports 1 event in the Social Norms + Behavioral Instructions group and 0 in each of the other listed groups. The adverse-event record does not provide a serious-event term, because no serious terms are listed. These sparse counts do not establish comparative safety: the registry does not report a statistical comparison of adverse events, and the study’s primary focus was appointment no-shows rather than a detailed safety evaluation.

The counts should be read alongside the corresponding at-risk numbers and the study’s recording procedures, which are not described in detail in the registry record. Zero reported serious events is a statement about the posted record, not a general conclusion about safety.

13. What the Result Does, and Does Not, Mean

The posted results show that the registry reports logistic-regression odds ratios for a set of no-show comparisons. Each reported 95% confidence interval includes 1, and each reported p-value is greater than 0.05. In the reported tests, the data do not provide strong statistical evidence against equal odds. The estimates remain uncertain: their confidence intervals include the null value and values on either side of it.

These findings do not show that the reminder approaches are equivalent, nor do they establish that any approach has no effect. A non-significant result can arise when the estimated difference is small, when uncertainty is substantial, or both. The registry does not report a prespecified equivalence margin or a non-inferiority framework. No equivalence conclusion is supported by these results alone.

The odds ratios are not absolute no-show percentages or percentage-point differences. Since the registry entries do not provide arm-specific no-show percentages, they do not reveal the absolute level of missed appointments in each group or the practical size of any difference. The human-readable identities of the coded comparison groups are also not given in the result entries, limiting attribution to individual behavioral messages.

Interpretation is further constrained by the cluster-randomized design and the limited model detail in the registry. The record says that a simple individual-level re-analysis would be misleading and reports only the sponsor’s own analysis. The cluster-adjustment method and full model specification are not reported. The results should therefore be understood as the sponsor’s reported statistical comparisons, not as a basis for converting odds ratios into absolute changes or drawing conclusions about a specific nudge’s performance.

Overall, the record presents uncertain relative-odds estimates, without enough arm-level outcome information to describe absolute attendance patterns. It does not establish that the groups have the same outcomes, and it does not quantify an absolute improvement or worsening in appointment attendance.

14. Why This Trial Matters Statistically

This trial is statistically notable because the intervention was assigned at the clinic level while the primary endpoint was calculated from appointments. That structure creates a distinction between the unit of randomization and the unit used to define the outcome. Analyses need to reflect the clinic grouping so that uncertainty is not understated by treating appointments from the same clinic as wholly independent. The registry itself flags this issue by warning that a simple individual-level re-analysis would be misleading.

The study also illustrates why relative effects need to be read alongside absolute outcome information. The primary outcome is defined as a percentage of appointments, but the reported comparisons are odds ratios. Without the group-specific percentages, the reader cannot gauge baseline no-show levels or translate the relative estimates into absolute differences. Reporting both would make the statistical magnitude more directly interpretable.

Multiplicity is another important consideration. The registry lists 20 analyses, while the reported primary-outcome entries include several comparisons and a secondary outcome is also listed. The record does not explain the planned hierarchy or any adjustment for multiple testing. Transparent reporting of which comparisons were prespecified and how inference was handled would help readers understand the evidential strength of individual results.

Finally, the primary-outcome analysis includes pooled Primary Care and Mental Health intervention groupings that were not randomized arms. Distinguishing randomized assignments from analytic groupings is essential for interpreting the estimand and the comparisons. Clear group labels, absolute outcome rates, detailed cluster-adjusted methods, and analysis-population definitions would support more reproducible and informative interpretation.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry’s reported analysis entries identify groups with OG codes but do not give a human-readable mapping for each comparison. This prevents confident attribution of individual odds ratios to named reminder approaches or to usual care. The registry also does not report arm-specific no-show percentages, so the absolute event levels and absolute differences cannot be assessed from these entries.

The trial is cluster-randomized, but the reported information does not describe how clustering was handled in the logistic-regression models. The registry warns that individual-level re-analysis would be misleading and provides only the sponsor’s analysis. The full model specification, including covariates and any cluster-level adjustments, is not reported. The analysis cannot therefore be independently assessed in detail from the registry record.

The primary outcome population includes pooled Primary Care and Mental Health intervention groupings beyond the Participant Flow total, and the registry notes that these were not arms to which participants were randomized. The exact population for each reported comparison is not reported in the analysis entries. This makes it difficult to determine precisely which population each estimate represents.

The registry lists 20 analyses but does not report a detailed multiplicity strategy, a comprehensive analysis plan, or a mapping of all analyses to the reported results. Missing-data methods and sensitivity analyses are also not reported. The endpoint description gives some rules for classifying appointments, but does not resolve every potential data-handling question. These reporting limits constrain interpretation; they do not establish that the analyses were incorrectly performed.

Finally, the reported odds ratios and confidence intervals do not directly communicate practical differences in no-show percentages. The record does not provide an equivalence framework, so the non-significant tests should not be read as proof that the reminder approaches yield identical outcomes.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: Using Behavioral Economics to Enhance Appointment Reminders and Reduce Missed Visits — NCT03850431.
• Registry record last updated 2024-04-19; status last verified 2023-10.
• Results first posted 2024-04-19.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

The registry describes a completed, randomized, parallel-group, unmasked, cluster-randomized study of behavioral-economics-informed appointment reminder letters. Enrollment was 34516. The primary endpoint was the percentage of appointments classified as no-shows over 1 year, and Cancellation Rate was listed as a secondary outcome.

The posted primary results include 10 logistic-regression odds-ratio comparisons. All reported 95% confidence intervals include 1, and all reported p-values are greater than 0.05. The record notes that the primary outcome analysis includes pooled Primary Care and Mental Health intervention groupings beyond the Participant Flow total; those pooled groupings were not randomized arms.

The registry cautions that a simple individual-level re-analysis would be misleading and reports only the sponsor’s own analysis. The mapping of coded comparison groups, arm-specific no-show percentages, full model details, missing-data methods, and multiplicity procedures are not reported in the registry record. Serious adverse event counts were 0 in each listed group, with 1 other adverse event reported in the Social Norms + Behavioral Instructions group.