Implementing and Evaluating Computer-Based Interventions for Mental Health: Testing an Implementation Strategy for VA Outpatient Care: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Implementing and Evaluating Computer-Based Interventions for Mental Health: Testing an Implementation Strategy for VA Outpatient Care (NCT03151083), a na na study. The primary outcome is Program Engagement: Completion of the First Program Module.

2. Clinical Question

The study concerns implementation of SHUTi, a self-guided internet program for insomnia, in VA Connecticut primary care. Its central question is whether different approaches to putting the program into routine care are associated with patient engagement, provider referral, and program completion. The registry describes a sequence of implementation phases rather than a randomized comparison, so the record does not establish a like-for-like comparison between intervention and control groups.

The record’s summary describes intended attention to patient engagement, provider adoption through referral, and patient completion. It also describes a proposed comparison with a low-intensity strategy over a six-month active implementation period. The phase descriptions, however, identify implementation windows of eight months each. The record does not provide enough reported outcome information to assess whether the primary question was answered or whether engagement differed across phases.

3. Trial Design

This is a completed, sequential study of implementation approaches for an internet-based insomnia intervention in VA outpatient care. The registry lists an enrollment of 77, a start date of 2017-05-30, and a primary completion date of 2020-01-01. The listed phase is NA, allocation is NA, masking is NONE, and the model is SEQUENTIAL. These entries do not describe a randomized trial with masked outcome assessment.

The registry describes three implementation phases. Phase 1, Implementation As Usual, used research-team implementation activities between June 2017 and January 2018. Phase 2, Primary Care Coached Digital CBTi Implementation, used activities executed by primary care teams between June 2018 and January 2019. Phase 3, Primary Care Mental Health Collaborative Care Implementation, also used activities executed by primary care teams, between April 2019 and November 2019. Each phase description calls its implementation window eight months. Provider education and reminders are named among the activities; the third phase description also names patient advertising or information.

Because the approaches occurred sequentially, calendar time and implementation approach are entangled in the design. A difference across phases could reflect changes in implementation, changes over time, or other factors; the registry record does not report a design that separates these explanations. It also does not report participant-level allocation procedures, a protocol for comparing phases, or a detailed statistical analysis plan. The sponsor is the VA Office of Research and Development. The registry status is COMPLETED, which describes the study’s status, not whether its primary outcome was achieved.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Analysis population for the primary outcome, as defined in the registry: Unique Veterans treated in VA Connecticut primary care over the respective intervention periods.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Program Engagement: Completion of the First Program Module, measured over 3 weeks following subject enrollment.

The primary endpoint is program engagement, defined as completion of the first program module. The description counts participants who engage by completing at least one module among unique Veterans treated in VA Connecticut primary care during the respective intervention periods. Its timeframe is three weeks following subject enrollment.

This measure is an early-use indicator: it distinguishes participants who begin the program, as defined by completing a module, from those who do not meet that threshold within the specified period. In implementation research, such a measure can help describe whether a service reaches participants and whether they take an initial step in using it. It is not, by itself, a measure of insomnia symptom change, sustained program use, or clinical benefit. The registry lists Insomnia Severity Index as a secondary outcome, not as the primary endpoint.

The record provides the endpoint definition but no reported count, proportion, estimate, or uncertainty interval for it. It therefore does not show how many participants completed a module, or how engagement varied among the implementation phases. The precise analysis denominator and handling of participants with incomplete follow-up are not reported in the registry record.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The registry record describes a sequential implementation study, but does not report a detailed statistical analysis plan or any analyses. General statistical concepts can clarify what would be needed to interpret the outcomes; they should not be mistaken for methods confirmed to have been used in this study.

What does the primary outcome count?

The primary outcome is a binary participant-level event: whether a participant completed at least one program module during the stated three-week period. A binary endpoint can be summarized by counting participants with the event and relating that count to a clearly defined denominator. The record specifies the intended population as unique Veterans treated in VA Connecticut primary care over the respective intervention periods, but does not report phase-specific event counts or denominators. It also does not report how repeated enrollment, eligibility changes, or incomplete observation were handled.

A proportion is often useful for a binary outcome because it expresses the frequency of the event among those included in the denominator. Comparing proportions across groups or periods requires clear definitions of both the numerator and denominator, as well as a stated analysis method. No such numerical results or analysis details are reported in the registry record. The endpoint definition alone cannot be used to reconstruct an observed proportion.

What does a confidence interval communicate?

General explanation: a confidence interval describes uncertainty around an estimate under a specified statistical procedure. For a proportion or a difference in proportions, it helps readers judge how precisely the data locate the underlying quantity. A narrow interval indicates greater precision than a wide interval, though precision does not by itself establish that an effect is important. The registry record reports no confidence interval for the primary outcome, and no interval should be inferred from the enrollment alone.

A p-value, when reported, addresses the compatibility of data with a specified statistical null model. It does not state the size or practical importance of an effect, and it cannot replace an estimate and its uncertainty. No p-value or inferential test is reported in the registry record. A conclusion about statistical significance would therefore be unsupported.

Why does the sequential design affect analysis?

The three implementation phases occurred in different periods. General explanation: a sequential design can be useful for studying implementation as practice changes, but contrasts between periods may reflect both the implementation strategy and time-related influences. A statistical adjustment can address only factors that are measured and appropriately incorporated; it cannot guarantee removal of unmeasured differences. The registry record does not report a model, covariate set, or strategy for accounting for time or other potential differences between phases.

The record lists allocation as NA and the model as SEQUENTIAL. It also explicitly notes that this is not a randomized comparison and that the phases are not a like-for-like control comparison. These design features limit causal interpretation of any between-phase contrast. The record does not report a concurrent comparison group, randomization procedure, masking, or a prespecified contrast for estimating the implementation effect.

What do the planning assumptions mean?

General explanation: sample-size calculations for a planned comparison depend on assumptions about the outcome, allocation, significance level, target power, and effect to detect. A planning calculation is not an observed result and does not demonstrate that a study had a particular probability of detecting an effect after the data were collected. The record describes an illustration based on the observed effect, with a two-sided significance level of 0.05, equal allocation, and a normal approximation; it says the effect to detect is a stated fraction of the observed effect. The numerical planning estimates are not reported in the registry record. Because this study is not described as a randomized two-arm trial, those generic assumptions should not be read as a description of the actual sequential phase design.

Over-dispersion is a general concern when count data vary more than a model assumes; ignoring it can make uncertainty estimates too optimistic. The primary endpoint is described as a participant count for a binary event, but no model or dispersion assessment is reported. There is therefore no basis to say whether over-dispersion was assessed or relevant to an analysis actually used.

Overall, the registry supplies an endpoint definition and design description but not the analysis outputs required to evaluate effect size, precision, or model adequacy. Statistical interpretations beyond those design and measurement details are not reported in the registry record.

10. Exact Estimates for Each Arm

No two-arm comparison is shown for this trial, but each arm's result can still be summarized with exact interval estimates computed from the posted counts. The Clopper-Pearson interval is exact and conservative; the Wilson interval is a common closer-to-nominal alternative; the Bayesian credible interval uses a uniform Beta(1, 1) prior. For small groups the three can differ noticeably.

R code and output

arm    <- c("Phase 1: Implementation As Usual (Implementation T", "Phase 2: Primary Care Coached Digital CBTi Impleme", "Phase 3: Primary Care Mental Health Collaborative ")
n      <- c(12658, 13592, 13572)
events <- c(17, 12, 16)
for (i in seq_along(n)) {
  ci <- binom.test(events[i], n[i])$conf.int          # Clopper-Pearson exact interval
  cr <- qbeta(c(0.025, 0.975), 1 + events[i], 1 + n[i] - events[i])   # Bayesian, uniform Beta(1,1) prior
  wi <- (function(x, n) { p <- x / n; z <- qnorm(0.975); d <- 1 + z^2 / n
    c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d, (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d) })(events[i], n[i])
  cat(sprintf("ARM|%s|%d|%d|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f\n", arm[i], n[i], events[i], 100 * events[i] / n[i],
              100 * ci[1], 100 * ci[2], 100 * wi[1], 100 * wi[2], 100 * cr[1], 100 * cr[2]))
}

ARM  Phase 1: Implementation As Usual (Implementation T  12658  17  0.1  0.1  0.2  0.1  0.2  0.1  0.2
ARM  Phase 2: Primary Care Coached Digital CBTi Impleme  13592  12  0.1  0.0  0.2  0.1  0.2  0.1  0.2
ARM  Phase 3: Primary Care Mental Health Collaborative   13572  16  0.1  0.1  0.2  0.1  0.2  0.1  0.2

11. Multiplicity, Missing Data and Analysis Populations

The registry lists six outcome measures, including Program Completion, Provider Adoption, and Insomnia Severity Index as secondary outcomes. It lists zero analyses. The record does not report which outcomes were analyzed, which comparisons were planned, or whether any outcomes were designated as secondary or exploratory in a detailed analysis plan beyond the listed endpoint labels.

General explanation: when several outcomes or comparisons are tested, the chance of at least one apparently unusual result can increase. A multiplicity strategy may define a primary test, adjust inference across tests, or treat secondary findings as exploratory. The registry record does not report a multiplicity adjustment or other strategy for handling multiple outcomes and comparisons. Since no analyses or inferential results are posted, there are no reported p-values to assess in light of multiplicity.

Missing-data handling is also not reported. The endpoint is defined within three weeks following enrollment, but the record does not describe how participants without a recorded module-completion status were handled, whether follow-up was complete, or whether any imputation or sensitivity analysis was used. It does not specify an analysis population such as all enrolled participants or a subset defined by eligibility or follow-up.

These omissions matter because denominators and missing observations affect interpretation of a binary engagement outcome. Without reported counts, analysis populations, and missing-data methods, readers cannot determine whether the outcome represents all enrolled participants or a narrower group. The registry record does not provide enough information to assess the possible influence of missing outcome data.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry’s serious adverse event summary is for All Enrollees. It reports 77 participants at risk, 0 serious adverse events, 0 other adverse events, and 0 deaths. These figures describe the posted safety counts for this group.

A recorded count of zero means that no events of the listed type appear in this registry summary. It does not establish that no adverse events occurred outside the reporting scope, that every participant had complete safety follow-up, or that the intervention has no risk. The record does not provide event terms, follow-up duration for safety, or additional detail about how adverse events were collected. The absence of listed serious-event terms should therefore be read as a limit of the posted safety information, not as proof of absence of harm.

These safety counts do not answer the primary engagement question, which concerns completion of a program module. They also do not provide evidence about insomnia symptoms or implementation outcomes.

13. What the Result Does, and Does Not, Mean

The record establishes that a sequential implementation study of SHUTi in VA Connecticut primary care was completed and that 77 participants were enrolled. It identifies completion of the first program module within three weeks as the primary engagement outcome. It also describes three implementation phases, with activities led first by the research team and later by primary care teams.

What the registry does not establish is the observed engagement rate, a difference between phases, or an effect of the implementation approach. No primary outcome result, effect estimate, confidence interval, or p-value is reported. The status COMPLETED and the presence of an endpoint definition cannot substitute for those results. Nor does the record provide a reported analysis of Insomnia Severity Index, provider adoption, or program completion.

General explanation: a description of an implementation strategy and an outcome definition can show what researchers intended to evaluate, while a result requires observed data and a stated analysis. To judge how precisely an outcome was estimated, readers need both an estimate and its uncertainty. To attribute a contrast to the strategy, they also need a design and analysis that address alternative explanations. The registry record does not report these elements for an outcome comparison.

The sequential phases are not a randomized, like-for-like control comparison. Accordingly, even if phase-specific outcome figures were available, temporal differences would need careful interpretation. The registry record does not provide figures or an analysis that would support such an interpretation.

The safety summary records no serious adverse events, other adverse events, or deaths among the 77 listed as at risk. This is a report of the registry counts, not a guarantee of safety or a comprehensive account of events. Overall, the record describes the study’s implementation context and intended measures but does not report the results needed to determine whether engagement, referral, completion, or symptoms changed.

14. Why This Trial Matters Statistically

This study is statistically relevant because implementation outcomes are not the same as clinical outcomes. Whether a participant starts a self-guided program, whether a provider refers a patient, and whether a participant completes the program describe different stages of care delivery. The registry names these implementation-related outcomes and separately lists Insomnia Severity Index. Keeping those measures distinct helps prevent an engagement measure from being mistaken for a symptom or treatment outcome.

The sequential design also illustrates an important interpretive issue. When implementation approaches occur in separate periods, a difference between periods may coincide with changes in staffing, workflow, patient mix, or other time-related circumstances. General explanation: a statistical association between phase and outcome does not by itself isolate the effect of the implementation approach. The record explicitly identifies the study as nonrandomized and cautions that the phases are not a like-for-like control comparison.

Clear reporting of event counts, denominators, estimates, and uncertainty would be important for understanding engagement. It would also help readers assess how much information the enrolled sample contributes and how stable a phase comparison might be. The registry identifies a primary measure but reports no outcome analysis, so those questions remain unanswered in the posted record.

The record also highlights the value of distinguishing a completed study from a fully reported statistical result. Study status, enrollment, endpoint definitions, and safety counts describe different aspects of a trial record. None alone establishes the size, precision, or direction of an implementation effect. This distinction is especially important when readers are considering a sequential implementation study rather than a randomized comparison.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The most important limitation for interpretation is that outcome results are not reported in the registry record. The primary endpoint is defined, but there is no count or proportion for participants completing a module, no phase-specific estimate, and no uncertainty interval. The registry lists zero analyses, so no statistical comparison can be evaluated from the posted results.

The design is sequential and nonrandomized, and the registry states that the phases are not a like-for-like control comparison. The implementation approaches took place in different time periods. This makes it difficult to distinguish any implementation effect from changes over time, even if phase-level outcomes were available. The record does not report an adjustment strategy for possible differences among periods.

Information on analysis populations and missing outcomes is not reported. The endpoint description names the intended population and timeframe, but does not specify how participants lacking a module-completion record or complete follow-up would be handled. The registry also does not report a detailed statistical analysis plan, model, covariate strategy, or multiplicity approach.

The summary describes an intended six-month active implementation period, whereas the descriptions of the three phases each identify an eight-month implementation window. The registry record does not explain how these durations relate. The phase descriptions identify some implementation activities, but do not provide enough detail to characterize every component or its fidelity.

Finally, the safety summary provides counts but limited context about collection and follow-up. Zero recorded events should be interpreted as the posted count, not as evidence that risk was absent. These reporting limitations restrict conclusions about engagement, symptoms, provider adoption, patient completion, and safety.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: Implementing and Evaluating Computer-Based Interventions for Mental Health: Testing an Implementation Strategy for VA Outpatient Care — NCT03151083.
• Registry record last updated 2023-07-21; status last verified 2023-07.
• Results first posted 2021-01-27.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

The registry describes a completed, sequential implementation study of SHUTi, an internet-based self-guided insomnia program, in VA Connecticut primary care. It reports enrollment of 77 and identifies three implementation phases occurring between June 2017 and November 2019. The primary outcome is completion of at least one program module within three weeks following enrollment.

The record lists six outcome measures and zero analyses. It does not report the primary outcome result, phase comparisons, or statistical uncertainty. The study is not described as randomized, and the registry notes that its phases are not a like-for-like control comparison. The safety summary reports 77 participants at risk, 0 serious adverse events, 0 other adverse events, and 0 deaths. These posted counts and design details describe the record, but do not establish an implementation effect or clinical outcome.