Using Mobile Technology to Enhance Multisystemic Therapy (MST) Outcomes: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Using Mobile Technology to Enhance Multisystemic Therapy (MST) Outcomes (NCT02320903), a na na study. The primary outcome is App Usage - Viewed Notifications Regarding Behavioral Expectations Met/Not Met.

2. Clinical Question

3. Trial Design

The registry describes a completed, Phase I, single-group study of the VillageWhere app prototype. It lists an enrollment of 88 and identifies Evidence-Based Practice Institute in Seattle, Washington, as the sponsor. The recorded study start is 2014-04, and primary completion is 2016-03. Allocation is listed as NA, masking as none, and the model as single group. These details indicate that the registry does not describe a randomized, masked, controlled comparison.

The brief summary describes an open 45-day feasibility trial with N=20 caregiver and teen dyads. It says that caregivers and teens would be assessed before receiving the app and at the end of app use, with therapists assessed at the end. This account does not match the registry’s listed enrollment of 88. The registry record does not explain the difference, so the two figures should not be treated as interchangeable or reconciled by assumption.

The prototype was intended to support smartphone location monitoring using GPS, teen self-appraisal of behavioral expectations, and a points system tied to rewards. Participants were asked to use the app as often as they wished. The record lists 21 outcome measures and 16 analyses, but does not report the allocation of participants across analysis populations or a detailed statistical plan. No control condition is described; changes observed over time, if reported, would therefore not by themselves establish that the app caused those changes.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: Initial App Set Up: Clinicians were solely responsible for assisting families to set up their app; however, many clinicians were uncomfortable with this because they were not "tech savvy". Solution: Phase II will include a self-guided setup tool that will empower users (parents, youth) to complete the set up independently.

Overall Study

Reasons for not completing this period:

Analysis population for the primary outcome, as defined in the registry: All parent/caregiver and teen participants analyzed.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as App Usage - Viewed Notifications Regarding Behavioral Expectations Met/Not Met, measured over up to 45 days.

The primary endpoint is App Usage - Viewed Notifications Regarding Behavioral Expectations Met/Not Met. It counts participants who viewed notifications that a teen was or was not meeting a behavioral expectation, with both parents and teens able to use this feature. The registry describes the outcome as a percentage, while its analysis parameter is a count of participants. The timeframe is up to 45 days, and the stated population is all parent or caregiver and teen participants analyzed.

This measure is useful for assessing whether participants engaged with a central app feature. For a feasibility study, feature use can help characterize whether the prototype was used as intended and whether its functions were accessed during the study period. But viewing a notification is not equivalent to changing monitoring behavior, delivering a reward, or improving a teen’s behavior. Those are distinct constructs and would require their own reported results and definitions.

The registry record does not report the number of participants who viewed the notifications, the denominator used for the percentage, or a confidence interval. It therefore does not establish how common notification viewing was or how precisely it was estimated. The endpoint is a usage measure, not evidence on its own of clinical benefit.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

Outcome scale and count data. The primary outcome is defined as a count of participants who viewed a notification, while the description frames it as a percentage. For a binary participant-level outcome, each participant is classified according to whether the specified event occurred or did not occur. A count records how many participants had the event; a percentage expresses that count relative to a defined denominator. The denominator is essential to interpreting a percentage, and it is not reported for this endpoint. No count or percentage is reported either, so the observed level of use cannot be assessed.

Precision and confidence intervals. General statistical explanation: a confidence interval describes the uncertainty around an estimate under the assumptions of the method used. For a proportion, it can indicate how compatible different underlying rates are with the observed data. A narrow interval generally indicates greater precision than a wide interval, but precision depends on the amount and pattern of data as well as the method. The registry record does not report a confidence interval or the endpoint data needed to assess precision. A point estimate alone, even if it were available, would not show how uncertain the estimate is.

Single-group interpretation. The study model is single group, with no randomized comparator identified. General statistical explanation: a one-group estimate can describe the observed experience of participants in that group, but it does not estimate a treatment contrast against a control group. Repeated measurements before and after app use can describe within-participant change if such results are reported, but change over time can have explanations other than the intervention. The record describes planned baseline and end-of-use assessments in its brief summary, but the relevant observed results and analysis details are not reported in the registry record.

Model and distribution choices. The primary endpoint is a participant count, and the record labels the outcome as binary. General statistical explanation: binary data are commonly analyzed using methods designed for event proportions or counts of events and non-events. The choice of method should reflect the unit of analysis, the denominator, repeated observations, and any clustering, such as multiple participants within a caregiver-teen dyad. The registry record does not report the statistical model, whether dyads were analyzed jointly, or how dependence between caregiver and teen observations was handled. There is no basis for attributing a particular model or adjustment to the investigators.

Over-dispersion is a general concern when count data vary more than a chosen model expects. If present, it can make standard errors too small and confidence intervals or tests too optimistic unless the analysis accounts for the extra variability. This concern is more directly relevant to some count models than to a simple binary participant outcome. The registry record does not report an over-dispersion assessment or a model for the primary endpoint, so no claim can be made about whether it was relevant or addressed.

Testing and adjustment. A significance test evaluates data against a stated null model; a p-value summarizes how unusual the data would be under that model, given the test’s assumptions. It does not measure the probability that a hypothesis is true or the size of an effect. The record does not report a p-value, a null hypothesis, or an inferential test for the primary endpoint. It also does not report an adjusted or unadjusted effect estimate. Thus, there is no reported test result to interpret.

The registry lists 21 outcome measures and 16 analyses. When many outcomes or analyses are considered, the chance of finding at least one apparently unusual result can increase if each is tested separately without accounting for multiplicity. General approaches can include designating a primary outcome in advance, adjusting inference across a defined family of tests, or treating secondary analyses as exploratory. Which approach was used is not reported in the registry record. The count of outcomes and analyses alone does not show whether a multiplicity adjustment was necessary or performed.

Planning illustrations. The stated assumptions for sample-size illustrations are a two-sided significance level of 0.05, equal allocation between two arms, a normal approximation, and an effect to detect expressed as a stated fraction of the observed effect. The instructions describe participant counts per arm for non-time-to-event outcomes and total events for time-to-event outcomes. These are planning illustrations, not observed power. The registry describes the actual study as single group, and the record does not report a comparator or the observed effect values needed to evaluate the illustrations. They should not be read as the actual trial’s design justification or as evidence about the precision of its endpoint estimate.

Finally, the registry does not report the endpoint’s analysis denominator, handling of missing values, or the analysis population used for each result. Those details matter because a participant who did not view a notification may differ from a participant with no recorded usage information. Without knowing how such cases were classified or excluded, the intended interpretation of any usage estimate would remain incomplete.

10. Exact Estimates for Each Arm

No two-arm comparison is shown for this trial, but each arm's result can still be summarized with exact interval estimates computed from the posted counts. The Clopper-Pearson interval is exact and conservative; the Wilson interval is a common closer-to-nominal alternative; the Bayesian credible interval uses a uniform Beta(1, 1) prior. For small groups the three can differ noticeably.

R code and output

arm    <- c("Parents/Caregivers", "Youth/Teens")
n      <- c(16, 16)
events <- c(14, 14)
for (i in seq_along(n)) {
  ci <- binom.test(events[i], n[i])$conf.int          # Clopper-Pearson exact interval
  cr <- qbeta(c(0.025, 0.975), 1 + events[i], 1 + n[i] - events[i])   # Bayesian, uniform Beta(1,1) prior
  wi <- (function(x, n) { p <- x / n; z <- qnorm(0.975); d <- 1 + z^2 / n
    c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d, (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d) })(events[i], n[i])
  cat(sprintf("ARM|%s|%d|%d|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f|%.1f\n", arm[i], n[i], events[i], 100 * events[i] / n[i],
              100 * ci[1], 100 * ci[2], 100 * wi[1], 100 * wi[2], 100 * cr[1], 100 * cr[2]))
}

ARM  Parents/Caregivers  16  14  87.5  61.7  98.4  64.0  96.5  63.6  96.2
ARM  Youth/Teens  16  14  87.5  61.7  98.4  64.0  96.5  63.6  96.2

11. Multiplicity, Missing Data and Analysis Populations

The record lists 21 outcome measures and 16 analyses, including secondary measures of aggression, depression, rule-breaking, parental supervision, positive parenting, parental consistency, parental effectiveness, and perceived stress. Multiple measures can be useful in a feasibility study because they cover different aspects of app use and participant experience. They also create a need to distinguish the principal question from exploratory questions and to explain how missing observations and multiple tests are handled.

Whether the investigators specified a multiplicity strategy, such as prioritizing a primary endpoint or adjusting for multiple comparisons, is not reported in the registry record. The listed number of measures and analyses does not establish that all were tested or that any particular correction was used. Without a prespecified analysis plan and reported results, it is not possible to determine how to interpret any isolated statistical finding across the outcome set.

Missing-data handling is not reported in the registry record. The record does not state how participants without app-use information, incomplete assessments, or incomplete follow-up were treated. It also does not provide outcome-specific denominators or explain whether analyses included all enrolled participants or only those with available observations. These omissions limit assessment of potential bias, especially when engagement with an app could affect whether outcome data are observed.

The primary outcome description names all parent or caregiver and teen participants analyzed, but does not define the analyzed population or its size. A full account would ordinarily clarify who was eligible for each analysis, how missing data were handled, and whether analyses followed a prespecified plan. Those details are not reported in the registry record.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry lists one group, Use of VillageWhere App Prototype. For serious adverse events, 32 participants were at risk and 0 were affected. For other adverse events, 32 were at risk and 0 were affected. No serious adverse event terms are listed. These are counts recorded for the study group; they are not estimates comparing the app with a control condition.

A zero count means that no affected participants were recorded in the stated group for the corresponding category. It does not establish that an event is impossible or that the intervention has no safety concerns. Interpreting the counts requires context about the participants included, observation period, event definitions, and completeness of reporting. The record does not provide further detail on these points, and its primary endpoint timeframe is described as up to 45 days.

The at-risk counts are 32 for both categories, but the registry’s overall enrollment is 88. The record does not explain the relationship between these figures, so the safety denominator should not be assumed to represent all enrolled participants. The zero counts are therefore best read as the posted safety figures for the recorded group, not as proof about risks outside that group or period.

13. What the Result Does, and Does Not, Mean

The record establishes that VillageWhere was studied as a prototype in a completed, single-group Phase I trial, and that the primary endpoint concerned whether caregivers or teens viewed notifications about behavioral expectations. It also identifies the app’s intended functions and several additional outcome domains. These details clarify what the investigators set out to examine, but they do not supply the primary endpoint result.

Because no observed count or percentage is reported for notification viewing, the record does not show how often participants accessed that feature. It also does not provide an interval or test that would describe statistical uncertainty. The secondary outcome names indicate areas of interest, not evidence that those outcomes changed. Without reported estimates and their denominators, no conclusions about the direction, size, or precision of changes are supported.

The design further limits what can be inferred. All enrolled caregivers and teens were assigned to use the prototype, and no control group is described. Even if pre-to-post changes were available, the single-group design would not distinguish app-related change from other explanations. It would also not show how use compares with another approach.

The registry’s safety figures record 0 serious adverse events and 0 other adverse events among 32 participants at risk in each category. These figures describe what was recorded in that group; they do not provide a comparative risk estimate. The enrollment figure of 88 and the brief summary’s N=20 are not reconciled in the record, adding uncertainty about how the study’s participant counts relate to the described feasibility trial.

Thus, the record can be read as documenting the study question, app, design, and a limited set of safety counts. It does not establish an effect on behavior, demonstrate efficacy, or quantify engagement with the primary app feature.

14. Why This Trial Matters Statistically

This trial is statistically informative as an example of the difference between a feasibility question and an effectiveness question. Its primary endpoint is about app-feature use: whether participants viewed notifications about behavioral expectations. That kind of measure can characterize engagement with a prototype, but it is not interchangeable with a behavioral outcome or a comparison of clinical benefit.

The single-group structure also makes the distinction between description and causal inference especially important. Counts and percentages from one group can describe what happened among participants with recorded data. They cannot by themselves show what would have happened without the app. A randomized comparator, if the goal were to estimate a causal contrast, would address a different question from this open feasibility design.

The record also illustrates why endpoint definitions and denominators matter. A percentage is interpretable only when its numerator and denominator are clear, and app use can be difficult to summarize if participants have different opportunities or levels of use. Here, the outcome is described, but the observed count, percentage, and analysis denominator are not reported.

Finally, the mismatch between the registry enrollment of 88 and the brief summary’s N=20, together with the smaller safety at-risk count of 32, emphasizes the value of explaining participant flow and analysis populations. The registry gives useful design context, but incomplete results and population details limit the statistical conclusions that can be drawn.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The primary limitation for interpreting the posted results is that the primary endpoint has no reported observed value. There is no count of participants who viewed notifications, no percentage, no confidence interval, and no test result. As a result, the central app-use question cannot be answered quantitatively from the registry record.

The single-group design has no like-for-like control comparison. It can describe participants’ experience if outcome data are reported, but it does not separate app-related changes from other causes. The registry does not report a detailed statistical plan, sample-size justification, interim analyses, or a protocol for handling missing data. These items are not reported in the registry record and cannot be inferred from the study description.

Participant counts are not fully reconciled: enrollment is listed as 88, while the brief summary describes an open trial with N=20 caregiver and teen dyads. The safety group lists 32 at risk for each adverse-event category. The record does not explain how these figures relate to one another, or which participants were included in the primary endpoint analysis.

The endpoint’s percentage denominator is not stated, and caregiver and teen participants may have different opportunities to view notifications. The record does not explain how those observations were combined or how dependence within caregiver-teen dyads was handled. It also does not report results for the listed secondary outcomes. Together, these gaps limit assessment of endpoint precision, representativeness, and the range of conclusions supported by the study.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: Using Mobile Technology to Enhance Multisystemic Therapy (MST) Outcomes — NCT02320903.
• Registry record last updated 2018-02-14; status last verified 2017-11.
• Results first posted 2018-02-14.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT02320903 is a completed Phase I, single-group study of the VillageWhere app prototype, sponsored by Evidence-Based Practice Institute in Seattle, Washington. The record lists Child Behavior Disorders, enrollment of 88, a start of 2014-04, and primary completion of 2016-03. The brief summary describes a 45-day open trial with N=20 caregiver and teen dyads, a figure that the registry does not reconcile with its listed enrollment.

The primary endpoint concerns participants viewing notifications about whether a teen met behavioral expectations. Its observed count, percentage, denominator, and statistical uncertainty are not reported in the registry record. The design has no control group, and the record does not report the statistical plan or missing-data methods. Safety figures list 0 serious adverse events and 0 other adverse events among 32 at risk in each category. The posted information describes the study’s aims and design but does not quantify the primary outcome or establish a comparative treatment effect.