An Initial Study of the Implementation of Digital Therapeutics for Substance Use Disorders in Primary Care: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for An Initial Study of the Implementation of Digital Therapeutics for Substance Use Disorders in Primary Care (NCT04907045), a non-randomized na study. The primary outcome is Reach of the Digital Therapeutic to Patients in the Primary Care Clinic.

2. Clinical Question

3. Trial Design

This was a completed, non-randomized pilot study of implementation in primary care. The registry lists an enrollment of 1552 and describes implementation in two primary care clinics. The design is parallel, with no masking. The record does not identify a randomized allocation process, and it explicitly cautions that the arms do not form a like-for-like control comparison.

The study description sets out two successive periods in the same clinics. In the first 3-month period, the clinics used a standard approach to implementing reSET and reSET-O. In the second 3-month period, the study tested and refined two additional strategies: health coaching directed to patients and practice coaching directed to clinicians. The arm descriptions specify standard implementation plus health coaching, and standard implementation plus health coaching and practice facilitation. The registry does not report a separate allocation mechanism or enough detail to establish how clinic, period, and strategy effects were distinguished analytically.

The stated purpose was to inform statistical design and data collection for a subsequent cluster-randomized trial. The pilot also aimed to test economic data-collection tools and gather qualitative data for formative evaluation. These aims make implementation feasibility and measurement processes central to interpreting the study, rather than treating it as a randomized efficacy comparison.

The record lists a 6-month timeframe for the primary reach outcome. It does not report a statistical analysis plan, analysis populations, interim analyses, or an explanation of how observations from the sequential periods were handled. The registry lists no analysis entries. These omissions limit what can be concluded about comparative effects from the posted record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: Of the 18,577 primary care patients with visits to the eligible clinics, 1,552 were eligible for the analysis.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Reach of the Digital Therapeutic to Patients in the Primary Care Clinic, measured over 6 months.

The primary endpoint is titled reach of the digital therapeutic to patients in the primary care clinic. It concerns initiation of reSET or reSET-O, defined in the registry as opening the app, entering the prescription code, and using a treatment module. The timeframe is 6 months. Although the endpoint is described as a proportion, the registry also identifies its unit as participants and its measure as a count of participants. No numerator, denominator, or reach estimate is reported in the record.

Reach is an implementation outcome: it describes whether intended users begin using the digital treatment, rather than whether symptoms or other clinical outcomes improve. For an implementation pilot, tracking initiation can help assess whether a treatment is being taken up in routine care and whether implementation strategies warrant further study. It does not by itself establish sustained use, adherence, clinical benefit, or comparative effectiveness. The record does not define the expected follow-up or additional engagement requirements beyond the specified initiation steps.

Because the endpoint is recorded as both a proportion and a participant count, interpretation would depend on knowing the denominator used to calculate any proportion and the rules for identifying eligible patients. Those details and the observed endpoint result are not reported in the registry record. The stated primary outcome therefore identifies what the pilot intended to measure, but the posted information does not quantify reach.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

Several features of this record affect how statistical evidence should be read. General statistical explanations below describe concepts; they do not imply that a particular analysis was performed in this pilot. The registry record lists no analysis entries and does not report a statistical analysis plan.

What non-randomized allocation means

In a randomized study, allocation by chance is intended to make comparison groups similar on average with respect to measured and unmeasured characteristics. Here, allocation is listed as non-randomized, and the record warns that the arms are not a like-for-like control comparison. A difference between groups would therefore be descriptive unless the design and analysis adequately addressed confounding. Confounding occurs when a factor related to both the strategy received and the outcome can contribute to an apparent group difference. Clinic practices, patient characteristics, or changes over time are general examples of factors that can matter in implementation research; the record does not say how these factors were measured or adjusted for.

Why the sequence matters

The study description places standard implementation in the first 3-month period and testing of added coaching strategies in a second 3-month period in the same clinics. A before-and-after comparison can be affected by calendar-time changes as well as the implementation change. General methods for repeated or sequential observations may account for time structure, but the registry record does not report which methods, if any, were used. It also does not explain whether outcomes were analyzed at the patient, clinic, or period level.

Reach and denominators

The endpoint describes initiation as opening the app, entering the prescription code, and using a treatment module. It is described as a proportion and as a participant count. For a proportion to be interpretable, the numerator and denominator need clear definitions: the numerator would identify people meeting the initiation definition, while the denominator would identify the eligible population at risk of initiating. This is a general explanation, not a claim about the pilot's calculation. Those counts and the denominator definition are not reported in the registry record. The total enrollment of 1552 is not necessarily the outcome denominator, since enrollment and eligibility for a particular measure can differ.

Uncertainty and confidence intervals

General statistical interpretation uses a confidence interval to describe the range of estimates compatible with the data and the assumptions of the method. A narrow interval indicates more precision than a wide interval, but does not eliminate possible bias from a non-randomized design or unclear measurement. The registry record does not report an estimate or confidence interval for reach, so its precision cannot be assessed. A p-value, if reported, would address compatibility with a specified statistical model and null hypothesis; it would not on its own resolve confounding or establish practical importance. No p-values are reported in the registry record.

Planning illustrations are not observed power

Sample-size planning calculations can illustrate how many participants per arm would be needed under assumptions about a target effect, significance level, allocation, and statistical approximation. The stated planning approach uses a two-sided significance level of 0.05, equal allocation between two arms, a normal approximation, and an effect defined as a stated fraction of the observed effect. These are assumptions for an illustration, not a description of the pilot's actual design or proof that it had a particular probability of detecting an effect. The registry does not report its own sample-size justification. Any planning values should not be described as observed power.

In general, planning based on an observed effect can be sensitive to how that effect was estimated. A small or unstable pilot estimate may not be a reliable value for designing a later trial. Considering multiple fractions of an effect is one way to examine how assumptions change planning requirements, but it cannot substitute for clinical judgment, a prespecified target, or a design that accounts for the intended analysis. No new planning values can be inferred from the registry record.

Binary outcomes and event counts

The primary measure is classified as a binary outcome and its meaning is participants with an event, recorded as counts. In general, a binary endpoint records whether a defined event occurred for each participant; counts can then be compared using suitable methods. The endpoint description is specifically an initiation definition, and the registry does not report its count. The safety record separately gives counts of serious and other adverse events and deaths for the named groups. Those small group-specific safety figures should be interpreted as recorded counts, not as evidence that the broader study population had the same event profile.

Statistical methods for counts depend on the data structure and assumptions. Over-dispersion is a general issue in count models: it means the variability in counts is greater than a model assumes, which can make standard errors too small if ignored. The registry record does not report a count model, an over-dispersion assessment, or any analysis of the primary endpoint. No such method should be attributed to the study on the basis of the posted record.

Finally, the registry lists seven outcome measures, a secondary outcome of effectiveness, and zero analysis entries. It does not report handling of missing data, analysis populations, multiplicity adjustments, or interim analyses. These are important features of a statistical plan, but their absence from the registry record does not establish what investigators did outside the record.

10. Exact Estimates for Each Arm

No two-arm comparison is shown for this trial, but each arm's result can still be summarized with exact interval estimates computed from the posted counts. The Clopper-Pearson interval is exact and conservative; the Wilson interval is a common closer-to-nominal alternative; the Bayesian credible interval uses a uniform Beta(1, 1) prior. For small groups the three can differ noticeably.

R code and output

arm    <- c("Standard Implementation Plus Health Coaching", "Standard Implementation Plus Health Coaching and P")
n      <- c(296, 1256)
events <- c(1, 4)
for (i in seq_along(n)) {
  ci <- binom.test(events[i], n[i])$conf.int          # Clopper-Pearson exact interval
  cr <- qbeta(c(0.025, 0.975), 1 + events[i], 1 + n[i] - events[i])   # Bayesian, uniform Beta(1,1) prior
  wi <- (function(x, n) { p <- x / n; z <- qnorm(0.975); d <- 1 + z^2 / n
    c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d, (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d) })(events[i], n[i])
  cat(sprintf("ARM|%s|%d|%d|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f\n", arm[i], n[i], events[i], 100 * events[i] / n[i],
              100 * ci[1], 100 * ci[2], 100 * wi[1], 100 * wi[2], 100 * cr[1], 100 * cr[2]))
}

ARM  Standard Implementation Plus Health Coaching  296  1  0.3  0.0  1.9  0.1  1.9  0.1  1.9
ARM  Standard Implementation Plus Health Coaching and P  1256  4  0.3  0.1  0.8  0.1  0.8  0.1  0.8

11. Multiplicity, Missing Data and Analysis Populations

The registry lists seven outcome measures and identifies effectiveness as a secondary outcome. It does not report the full outcome definitions or analysis results for those measures in the record. When a study examines multiple outcomes or multiple comparisons, the chance of at least one apparently notable result can increase. General statistical approaches may prespecify a primary outcome, control the family-wise error rate, or use other strategies appropriate to the study aim. The registry record does not report a multiplicity strategy, so no adjustment should be assumed.

Missing-data handling is also not reported. In general, missing outcomes can affect estimates if the availability of data is related to participant characteristics, treatment access, or the outcome itself. Methods may include explicit assumptions and sensitivity analyses, but the record does not state whether these were used. It also does not define an analysis population, such as all enrolled patients or only those eligible for the app. Enrollment of 1552 should not be assumed to equal the denominator for every outcome.

The registry lists zero analysis entries and does not report interim analyses or a statistical analysis plan. It is therefore not possible to determine from the record which comparisons were prespecified, how missing observations were handled, or whether any multiplicity control was planned. These details are not reported in the registry record; their absence should not be interpreted as evidence that no analyses occurred outside the record.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

For Standard Implementation Plus Health Coaching, the registry records 6 participants at risk for serious adverse events and 0 affected; it records 6 at risk for other adverse events and 0 affected, with 0 deaths. For Standard Implementation Plus Health Coaching and Practice Facilitation, it records 8 at risk for serious adverse events and 0 affected; it records 8 at risk for other adverse events and 0 affected, with 0 deaths. The serious adverse event terms are not listed.

These are the safety counts recorded for the named groups. They should not be treated as a complete safety profile for all 1552 enrolled participants: the record's at-risk counts are 6 and 8 for the respective groups, and it does not explain in the information available how those safety populations were selected or followed. Zero recorded events in these groups is a count, not proof that no events occurred outside the recorded population or that an intervention is safe. The registry does not report comparative safety analyses or uncertainty estimates. Because the study was non-randomized, these figures also do not support a like-for-like safety comparison between strategies.

13. What the Result Does, and Does Not, Mean

The record establishes that the DIGITS pilot was designed to examine implementation of reSET and reSET-O in primary care and to develop information for a subsequent cluster-randomized trial. It describes a standard implementation period followed by a period testing and refining patient health coaching and clinician-facing practice coaching. It also identifies reach, defined through app-initiation steps, as the primary outcome. These details explain what the study set out to learn.

The registry information does not establish the extent of reach. No outcome numerator, denominator, estimate, confidence interval, or p-value is reported for the primary measure. The overall enrollment figure cannot be substituted for an outcome-specific denominator. Without those results, the record cannot show how many patients initiated the apps or how precisely reach was estimated.

Nor does the record establish whether one implementation strategy produced better outcomes than another. Allocation was non-randomized, and the registry cautions that the arms are not a like-for-like control comparison. The sequential timing of the implementation periods also means that a simple contrast could mix strategy differences with time and clinic context. No comparative analysis is reported.

The zero event counts in the listed safety groups describe the recorded participants in those groups only. They are not a guarantee of safety and do not characterize all enrolled participants. Likewise, planning illustrations based on fractions of an observed effect are not observed evidence and should not be read as power achieved by the study.

In general, a pilot can inform whether measurement and implementation procedures are workable and help shape a later trial. The registry describes those aims, but it does not report the corresponding outcome estimates or qualitative findings. The record therefore supports conclusions about the pilot's stated design and purpose, not conclusions about clinical effectiveness, comparative implementation success, or population-wide safety.

14. Why This Trial Matters Statistically

Implementation questions are distinct from questions about whether a treatment changes a clinical outcome. A digital therapeutic can only be evaluated in routine care if eligible patients can access and initiate it. Measuring reach therefore addresses an important step in the pathway between offering an intervention and evaluating its subsequent effects. The DIGITS pilot defined initiation through observable app-use steps, giving the primary outcome an operational meaning.

The study also focused on how implementation could be organized in primary care. It included standard implementation and described additional patient health coaching and clinician-facing practice facilitation. Testing data-collection tools and gathering qualitative information were stated aims alongside the implementation work. These features can inform the design of later research even when a pilot is not structured as a definitive comparative trial.

Statistically, the record highlights why design and analysis need to match the question. The pilot was non-randomized, used successive periods in the same clinics, and explicitly cautioned against treating its arms as a like-for-like control comparison. A subsequent cluster-randomized trial, as described in the pilot's aims, would address a different design question. The registry says the pilot's analytic goals were to inform that later trial's statistical design and data collection processes.

The record's limitations also illustrate the distinction between identifying an endpoint and reporting its evidence. A clear definition of reach is useful, but an outcome estimate, denominator, and measure of uncertainty are needed to assess the observed result. Those endpoint results are not reported in the registry record. The pilot's statistical importance therefore lies in its implementation and future-design aims, while the posted information does not permit quantitative conclusions about reach or comparative effects.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The main limitation for interpretation is that the registry record does not report the primary outcome result. Although reach is defined and assigned a 6-month timeframe, no count, denominator, proportion, or uncertainty estimate is given. The record lists enrollment of 1552, but does not establish that this is the population used to calculate reach.

The non-randomized design and the registry's explicit warning that the arms are not like-for-like comparisons limit causal interpretation. The study description also places standard implementation in an initial 3-month period and testing of additional strategies in a second 3-month period in the same clinics. The record does not report how possible time effects, clinic context, or differences in patient composition were addressed.

There is limited information about the populations contributing to outcomes. The registry does not describe eligibility criteria, baseline characteristics, outcome-specific denominators, or the analysis population. It does not report missing-data methods, an analysis plan, interim analyses, or multiplicity handling. It lists seven outcome measures, a secondary outcome of effectiveness, and zero analysis entries, but does not provide outcome estimates in the record.

The safety figures apply to small, specified at-risk groups, with 6 and 8 participants listed for the two arms. The registry record does not explain the selection or follow-up of those groups, so the counts should not be generalized to all enrolled participants. The absence of listed serious adverse event terms does not establish that the record provides a complete safety assessment.

These limitations concern the posted registry information and the design described there. They do not establish what analyses, data collection, or additional findings may exist outside the registry record.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: An Initial Study of the Implementation of Digital Therapeutics for Substance Use Disorders in Primary Care — NCT04907045.
• Registry record last updated 2025-08-06; status last verified 2025-07.
• Results first posted 2025-08-06.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT04907045 is a completed, non-randomized pilot sponsored by Kaiser Permanente, with a parallel model and no masking. The registry lists enrollment of 1552 and describes implementation of reSET and reSET-O in two primary care clinics. The study used a 3-month standard implementation period followed by a 3-month period testing and refining health coaching and practice facilitation.

The primary outcome is reach, defined by app opening, prescription-code entry, and use of a treatment module, with a 6-month timeframe. The registry does not report its estimate or an analysis result. It lists seven outcome measures, effectiveness as a secondary outcome, and zero analysis entries. The safety groups record no serious adverse events, other adverse events, or deaths among their listed at-risk participants. The stated purpose was to inform design and data collection for a subsequent cluster-randomized trial.