Clinical Reminder Changes to Increase Smoking Cessation Treatment in Outpatient Psychiatry: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Clinical Reminder Changes to Increase Smoking Cessation Treatment in Outpatient Psychiatry (NCT04071795), a randomized na study. The primary outcome is Number of Patients Screened for Smoking.

2. Clinical Question

The trial asks whether changing the design of a tobacco-use clinical reminder changes screening and treatment-related care in outpatient psychiatry. The comparison is between an opt-in reminder, which encourages the psychiatrist to offer cessation medication and counseling referral to patients interested in quitting, and an opt-out reminder, which includes a standing nicotine replacement therapy order and counseling referral that generate automatically unless the provider opts out.

3. Trial Design

This was a randomized, parallel, cluster-randomized study with no masking. The unit assigned to a reminder condition was the psychiatrist, rather than each patient treated by that psychiatrist. The study description identifies 20 psychiatrists at the VA New York Harbor Healthcare System and says half received the opt-in reminder while the other half received the opt-out reminder. The registry lists enrollment of 140, but that figure should not be treated as the number of psychiatrists: the record separately identifies the psychiatrists as the clusters and describes patient electronic health record data for outcome analysis.

Both groups received a one-hour training on tobacco treatment and individual education outreach, termed academic detailing, before reminder implementation. The description says this outreach demonstrated the clinical reminder and allowed questions. The opt-in condition prompted psychiatrists to offer cessation medication and counseling referral to interested patients. The opt-out condition incorporated a standing nicotine replacement therapy order and counseling referral that would generate unless the provider actively opted out.

The registry lists the study as completed, with a start date of 2020-01-06 and a primary completion date of 2021-02-24. The phase is listed as NA. These administrative details establish the registered status and timeline; they do not establish what the outcome comparison found.

Cluster assignment matters to interpretation because patients cared for by the same psychiatrist may have outcomes that are more alike than outcomes for patients cared for by different psychiatrists. Consequently, patient records are not necessarily independent observations. A valid comparison needs to account for assignment and dependence at the psychiatrist level. The registry record does not report the sponsor's statistical model, the number of clusters analyzed, or the handling of any variation in the number of patients associated with each psychiatrist.

The primary endpoint is described as a patient count based on medical records, with a timeframe of up to Month 6. The record also lists three outcome measures overall, but the primary outcome is the only one described in detail in the available outcome information. It does not give the observed primary endpoint count in either group or a between-group estimate.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: In this study, psychiatrists practicing at the VA New York Harbor Healthcare System were enrolled and randomized to either the Opt-Out Reminder group (n=58) or Opt-In Reminder group (n=82). A waiver of informed consent was granted to extract data on patients (Opt-Out Reminder, n=240; Opt-In Reminder, n=303) who were seen by the participating psychiatrists. These patients were not considered to be enrolled participants

Overall Study

Analysis population for the primary outcome, as defined in the registry: A waiver of HIPAA authorization was granted to abstract patient electronic health record (EHR) data from VA administrative data for the purposes of analysis of the primary outcomes. The overall number of participants analyzed represents patients (Participants) with available EHR data (Opt-Out Reminder Group, n = 240; Opt-In Reminder Group = 303) who visited a participating psychiatrist.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: Baseline data were only collected from participating psychiatrists. A waiver of informed consent was granted to extract data on patients who were seen by the participating psychiatrists. No baseline data were collected for these patients.

6. Primary Endpoint

The registry lists the primary outcome as Number of Patients Screened for Smoking, measured over Up to Month 6.

The primary endpoint is the number of patients with documented current smoking in the medical record who were screened for tobacco use during a psychiatry visit. The source is described as medical records, and the registered timeframe is up to Month 6. The measure therefore concerns documented screening activity, rather than a patient-reported change in smoking or a count of patients who quit.

Screening is relevant to the trial's aim because both reminder designs are intended to affect clinical action around tobacco use. A record of screening can indicate whether tobacco use was addressed during a visit, and the contrast between reminder designs can be assessed through the recorded counts. However, a count alone does not explain why a screening entry is absent, whether the patient was eligible for screening at that visit, or whether the care recorded elsewhere in the medical record was captured by the endpoint definition.

The registry describes the primary outcome population as patients with available electronic health record data, but that description is truncated before the full population detail is given. It does not report the observed screening counts by group, the number of eligible records, or an effect estimate. Accordingly, the endpoint definition can be described, but its observed value and the precision of the comparison are not reported in the registry record.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The central statistical feature is cluster randomization. The study description says psychiatrists, not individual patients, were assigned to the reminder conditions. Patients associated with one psychiatrist may share features of care, documentation practices, or clinical workflow. As a result, their recorded outcomes can be correlated. Standard methods that assume every patient observation is independent may treat the information in the dataset as greater than it really is and can produce uncertainty estimates that are too narrow.

Why does cluster assignment affect analysis?

Analysis needs to reflect the level at which the intervention was assigned. A cluster-aware method accounts for similarities among observations within each psychiatrist's practice. Common approaches in general include models with cluster-level effects or methods that adjust uncertainty for clustering. These are general statistical possibilities, not claims about the method used in this trial. The registry record does not report the sponsor's model, the correlation assumptions, or other analysis details. It does caution that a simple individual-level re-analysis would be misleading and that only the sponsor's own analysis is shown.

The number and size of clusters can also matter. With cluster trials, the amount of independent information is influenced by how many clusters are studied and by how similar outcomes are within a cluster. Larger groups of patients under a single psychiatrist do not necessarily provide as much additional independent information as adding another psychiatrist. The record identifies 20 psychiatrists in the study description, but does not report the number included in the primary analysis or cluster-specific patient counts. No estimate of within-cluster correlation is reported.

What does the endpoint count represent?

The primary outcome is a count of patients screened, based on medical-record documentation. Counts summarize how many recorded observations meet the endpoint definition, but interpretation depends on the eligible population and the way each group contributes records. A count is not automatically a rate or a proportion. Those require a clearly defined denominator, and the registry record does not give the group-specific eligible totals for the primary endpoint. No rate, percentage, or adjusted contrast can therefore be reconstructed from the endpoint description alone.

For a binary outcome, each eligible record can be classified according to whether the event occurred, and the resulting counts can be compared between groups. General methods may estimate a difference or another contrast and account for clustering. Here, the record identifies a count endpoint and a binary outcome classification, but it does not provide the observed primary outcome counts or the analysis estimate. It is therefore not possible to describe the estimated effect or its precision from the registry record.

How should uncertainty be understood?

In general, a confidence interval communicates the statistical uncertainty around an estimated effect under the assumptions of the analysis. A wider interval indicates less precision than a narrower interval, while the interval's location describes the range of effects compatible with the model and data. A p-value addresses the compatibility of the observed data with a specified null model; it does not quantify the probability that a treatment is effective. Neither an interval nor a p-value is reported for the primary comparison in the registry record.

Over-dispersion is a general concern when observed variation is greater than a model assumes. In cluster trials, ignoring dependence can likewise make standard errors too small. More broadly, model assumptions should match the design and outcome scale. These are general considerations, not reported features of the sponsor's analysis. The registry record does not state whether over-dispersion was assessed, how model fit was evaluated, or which variance estimator was used.

What can planning calculations tell readers?

Sample-size calculations are planning tools, not observations of trial performance. The described planning illustration uses a two-sided significance level of 0.05, equal allocation between groups, and a normal approximation. It is based on the observed effect, with the effect to detect expressed as a stated fraction of that observed effect. For outcomes other than time-to-event outcomes, the illustration is expressed as participants per arm. These assumptions describe the calculation approach, not the original design rationale.

The registry record does not report a sample-size justification for the trial, and no planning result can establish what the study actually observed. Planning figures should not be read as observed power. Because the primary outcome effect is not reported in the registry record, the numerical planning illustrations and their inputs cannot be interpreted as evidence about the comparison's result. Any planning calculation is distinct from the completed trial's endpoint data and from the analysis accounting for psychiatrist-level clustering.

Finally, the registry lists zero analyses. This means the registry record does not supply a formal analysis description alongside the outcome information. The design cautions against independent patient-level recalculation, so an unadjusted recalculation from summary counts would not be a suitable substitute for a cluster-aware sponsor analysis. No recalculated effect or precision measure is presented.

10. Exact Estimates for Each Arm

No two-arm comparison is shown for this trial, but each arm's result can still be summarized with exact interval estimates computed from the posted counts. The Clopper-Pearson interval is exact and conservative; the Wilson interval is a common closer-to-nominal alternative; the Bayesian credible interval uses a uniform Beta(1, 1) prior. For small groups the three can differ noticeably.

R code and output

arm    <- c("Opt-Out Reminder - Patients of Participating Psych", "Opt-In Reminder - Patients of Participating Psychi")
n      <- c(240, 303)
events <- c(240, 303)
for (i in seq_along(n)) {
  ci <- binom.test(events[i], n[i])$conf.int          # Clopper-Pearson exact interval
  cr <- qbeta(c(0.025, 0.975), 1 + events[i], 1 + n[i] - events[i])   # Bayesian, uniform Beta(1,1) prior
  wi <- (function(x, n) { p <- x / n; z <- qnorm(0.975); d <- 1 + z^2 / n
    c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d, (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d) })(events[i], n[i])
  cat(sprintf("ARM|%s|%d|%d|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f\n", arm[i], n[i], events[i], 100 * events[i] / n[i],
              100 * ci[1], 100 * ci[2], 100 * wi[1], 100 * wi[2], 100 * cr[1], 100 * cr[2]))
}

ARM  Opt-Out Reminder - Patients of Participating Psych  240  240  100.0  98.5  100.0  98.4  100.0  98.5  100.0
ARM  Opt-In Reminder - Patients of Participating Psychi  303  303  100.0  98.8  100.0  98.7  100.0  98.8  100.0

11. Multiplicity, Missing Data and Analysis Populations

The registry lists three outcome measures, while the primary measure is the number of patients screened for smoking. The other outcome measures are not described in the available record details. A complete account of which outcomes were designated primary or secondary, and how they relate to one another, is therefore not available beyond the stated primary endpoint and the total number of outcomes.

When a study evaluates multiple outcomes or makes multiple statistical comparisons, the chance of a false-positive finding can increase if each is assessed without an appropriate plan. General approaches include prespecifying a primary comparison, adjusting for multiple testing, or interpreting secondary findings as exploratory. The registry record does not report a multiplicity procedure, a testing hierarchy, or any adjustment. It also lists zero analyses, so no analysis plan or results for the additional outcome measures are detailed.

The primary endpoint description says the population analyzed consists of patients with available electronic health record data, but the description is truncated. The record does not explain how missing or incomplete records were handled, whether records were excluded, or whether any sensitivity analysis was conducted. It does not report an intention-to-treat definition for the patient-level outcome or explain how provider-level assignments were retained in analysis if patient records were unavailable. Those details are not reported in the registry record.

These omissions matter because analysis populations and missing-data methods can affect the interpretation of a comparison. A count among records with available data may differ from a count in a broader eligible population if availability is related to screening or care. This is a general concern, not evidence that such bias occurred in this trial. Without the full population definition, group-specific denominators, and missing-data approach, readers cannot determine the extent of missingness or its possible effect on the endpoint estimate.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The adverse-event section reports zero serious adverse events among 34 participants at risk in the Opt Out Reminder group and zero among 55 participants at risk in the Opt In Reminder group. It also reports zero deaths in each group. No top serious-event terms are listed. These are the safety figures recorded for the groups; they are separate from the primary screening endpoint.

A count of zero means that no event of the specified type was reported in the listed group during the period covered by the registry entry. It does not establish that an event could never occur, nor does it show that the groups are equivalent in safety. The registry record does not give further detail about the safety observation period, event definitions, or the method used to collect and classify adverse events. Those details are not reported in the registry record.

The at-risk group sizes in the adverse-event section should not be used as denominators for the primary outcome unless the record explicitly establishes that they are the same analysis population. The primary outcome population is described separately and incompletely. The zero serious-event and death counts therefore inform only the posted safety summary, not the comparative effect on screening.

13. What the Result Does, and Does Not, Mean

The record documents a completed randomized, parallel, cluster-randomized study of two clinical reminder designs in outpatient psychiatry. It describes how the reminders differed, how psychiatrists were trained, and how the primary outcome was defined. Those details allow readers to understand the planned clinical workflow comparison and the type of documentation used to assess screening.

The record does not report the observed primary endpoint counts, a between-group estimate, a confidence interval, or a test result. It therefore does not establish whether the opt-out reminder was associated with more or fewer documented screenings than the opt-in reminder, nor how large or precise any difference might have been. The completion status and enrollment figure are not substitutes for those results.

The primary endpoint measures documented screening, not smoking cessation itself. Even if screening counts were reported, they would not on their own establish whether patients accepted medication, attended counseling, changed smoking behavior, or experienced a health outcome. The trial's reminder strategies include treatment-related orders and referral, but those components should not be conflated with a demonstrated change in smoking behavior.

The design also limits what can be inferred from simple comparisons of patient records. Because assignment occurred at the psychiatrist level, a valid analysis needs to respect clustering. The registry cautions against an individual-level re-analysis, and the sponsor's analysis details and result are not reported in the registry record. The safety summary reports zero serious adverse events and deaths in both listed groups, but the detail available does not establish broader safety equivalence.

The evidence that can be described from the registry is therefore about study structure, endpoint definition, and posted safety counts. The primary comparative result and its precision are not reported in the registry record. No conclusion about the reminder strategies' effects on screening or cessation should be drawn from the administrative completion status alone.

14. Why This Trial Matters Statistically

This trial illustrates why the unit of randomization is a core part of statistical interpretation. A study can collect patient-level outcomes while assigning an intervention at the psychiatrist level. The apparent number of patient records is not the same as the number of independent intervention assignments. Analyses need to account for the clustering created by the design, or their uncertainty may be misleading.

It also distinguishes an intervention's workflow mechanism from the outcome used to evaluate it. The reminders differ in whether treatment orders and referrals are actively prompted or generated unless a provider opts out. The primary endpoint focuses on documented screening, which is a clinical-process measure. Keeping those elements distinct helps avoid treating a reminder's intended mechanism as proof of an outcome effect.

Registry transparency is important for interpretation. The record identifies the primary endpoint and overall number of outcomes but does not report the primary counts or statistical estimate. It also does not provide analysis details, a complete endpoint population description, or the approach to missing data and multiplicity. These gaps prevent readers from evaluating the magnitude and precision of the primary comparison from the registry record.

The posted serious-event counts are likewise interpreted in context: zero reported events are useful descriptive information, but they do not answer the effectiveness question or establish equivalence. Taken together, the design and reporting details show why readers need the analysis estimate, its uncertainty, the cluster-aware methods, and the outcome denominators before drawing a statistical conclusion.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The central limitation for interpretation is that the registry record does not report the primary outcome counts or a comparison estimate. Without those results, the magnitude, direction, and precision of the screening comparison cannot be assessed. The record lists three outcome measures but gives no analysis results, and it lists zero analyses.

The primary population description is truncated after referring to patients with available electronic health record data. The full definition of the analysis population, group-specific endpoint denominators, and the handling of unavailable records are not reported in the registry record. This leaves uncertainty about which records contributed to the primary endpoint and how the analyzed population relates to the enrollment figure.

The study uses psychiatrist-level cluster randomization, but the registry record does not describe the sponsor's statistical model, cluster-adjusted uncertainty method, number of clusters included in the analysis, or any assessment of within-cluster dependence. The record cautions that a simple patient-level re-analysis would be misleading. Without the sponsor's estimate and method details, the statistical comparison cannot be independently evaluated from the information recorded.

The record also does not report a sample-size justification, a multiplicity procedure, or missing-data methods. It identifies a primary outcome timeframe of up to Month 6, but does not give further detail about outcome ascertainment timing or the safety observation period. These are reporting limitations; they do not by themselves show that the study was conducted improperly. They do constrain what can be concluded from the registry record.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: Clinical Reminder Changes to Increase Smoking Cessation Treatment in Outpatient Psychiatry — NCT04071795.
• Registry record last updated 2023-04-28; status last verified 2023-04.
• Results first posted 2023-04-28.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT04071795 is a completed, randomized, parallel, unmasked, cluster-randomized study sponsored by NYU Langone Health. It compares opt-out and opt-in tobacco-use reminders for outpatient psychiatrists at the VA New York Harbor Healthcare System. The study description identifies 20 psychiatrists, and the registry lists enrollment of 140.

The primary endpoint is the number of patients with documented current smoking who were screened for tobacco use during a psychiatry visit, using medical-record data over a timeframe of up to Month 6. The registry record does not report the primary outcome counts, a comparison estimate, or analysis details. It lists zero analyses and three outcome measures overall.

The adverse-event summary reports zero serious adverse events and zero deaths in each group: 34 participants at risk in the opt-out group and 55 in the opt-in group. These safety counts do not supply the missing primary outcome result.