Mobile After-Care Support App: Pilot RCT: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Mobile After-Care Support App: Pilot RCT (NCT03955250), a randomized na study. The primary outcome is Client Satisfaction Questionnaire (CSQ).

2. Clinical Question

3. Trial Design

This was a randomized, parallel-group pilot trial comparing the Mobile After-Care Support (MACS) app with a mobile app attention control. The registry lists the study as single masked; which people were masked is not reported in the registry record. Both groups downloaded an app to a mobile phone using a third-party software platform. The MACS app was designed to provide ecological momentary assessment and intervention, while the attention control app was designed to provide ecological momentary assessment and psychoeducation about illness.

The registry records enrollment of 42 and lists the study status as completed. The brief summary describes the planned pilot randomized clinical trial as n = 60, so the planned enrollment described there differs from the recorded enrollment. A reason for the difference is not reported in the registry record. The start date is 2019-05-31 and the primary completion date is 2022-05-31.

The primary outcome was the Client Satisfaction Questionnaire (CSQ), assessed at one month. The posted outcome summaries include 10 participants in the MACS arm and 12 in the attention control arm. The registry notes that fewer participants were analyzed for this measure than were enrolled because of missing data on the measure. Reasons for missingness, the analysis population definition, and the handling of missing values are not reported in the registry record.

Brief Psychiatric Rating Scale (BPRS), Brief Adherence Rating Scale (BARS), and Brief Cope Inventory (Brief COPE) are listed as secondary outcomes. The registry records four outcome measures and seven analyses. It does not report the original sample-size justification or the trial's full statistical analysis plan.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Analysis population for the primary outcome, as defined in the registry: The number of participants analyzed is lower than the number enrolled due to missing data on this measure.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Client Satisfaction Questionnaire (CSQ), measured over 1 month.

The primary endpoint is the Client Satisfaction Questionnaire (CSQ) score at one month. It is a self-report measure of patient satisfaction with treatment. The total score is the sum of items and ranges from 8 to 32; higher scores indicate greater satisfaction. The registry describes the outcome as a mean on a continuous scale, with standard deviation used to describe spread.

Satisfaction is relevant to a feasibility-focused pilot because it describes participants' appraisal of the treatment experience. It is distinct from symptom ratings, adherence, coping, or clinical events. The study also lists the Brief Psychiatric Rating Scale, Brief Adherence Rating Scale, and Brief Cope Inventory as secondary outcomes, but the primary comparison is the CSQ score.

The posted mean CSQ score was 23.7 in the MACS arm and 22.5 in the mobile app attention control arm. The registry indicates that the number analyzed was lower than enrollment because of missing data on this measure. The analyzed group sizes were 10 and 12, respectively. The record does not report the pattern or reasons for missing CSQ data, or whether missingness was addressed by an imputation or other method.

A satisfaction score captures a participant-reported outcome at the stated assessment time. It does not by itself describe longer-term satisfaction, treatment adherence, or changes in symptoms. Nor does this score alone establish whether the apps differ on the study's broader aims.

7. Results Posted in the Registry

• Mobile After-Care Support (MACS) App: mean 23.70 versus 22.50 on control; difference 1.20 (95% CI -3.89 to 6.29), standardized effect d = 0.19; z-test p = 0.644.

8. Statistical Interpretation

9. Statistical Methods Explained

What does the mean difference represent?

A mean is the average score within a group. For this continuous endpoint, the posted means are 23.7 in the MACS group and 22.5 in the attention control group. The unadjusted mean difference is reported as 1.2 points, calculated in the direction of MACS minus attention control. A positive estimate means the posted average is higher in the MACS group; it does not by itself establish that the difference is reliable or important.

The standard deviations are 4.4 for MACS and 7.6 for attention control. A standard deviation describes the spread of individual scores around a group mean. Greater spread means participant scores vary more within that group. It is not the same quantity as the uncertainty around the estimated difference, which is represented by the confidence interval.

What does the t-test assess?

The registry reports a two-sided t-test. In general, a two-sided test evaluates whether the data provide evidence of a difference in either direction, rather than testing only whether one group is higher. A t-test compares a difference in means with its uncertainty under assumptions about the data and analysis. The sponsor's reported result is p > 0.05. This means the registry does not report a result meeting the conventional threshold associated with statistical significance; it does not prove that the group means are equal.

The registry does not post the sponsor's exact p-value, test statistic, or confidence interval. A separate unadjusted recalculation from the posted summary statistics gives p = 0.644. Because the record marks a discrepancy, the sponsor's own analysis takes precedence. The recalculation can be useful for understanding the posted summaries, but should not replace the trial's reported analysis.

How should the confidence interval be read?

In general, a confidence interval expresses the uncertainty around an estimated quantity under the method used to construct it. The recalculated interval for the mean difference runs from -3.892 to 6.292. Values below zero correspond to a lower mean in MACS than in attention control, while values above zero correspond to a higher mean in MACS. Since the interval spans zero, the summaries do not rule out no mean difference. Its width also indicates that the estimate is not highly precise.

The registry does not post the confidence interval from the sponsor's analysis. The recalculated interval is an unadjusted reconstruction from the posted group sizes, means, and standard deviations. It should be distinguished from a sponsor-reported interval, particularly because the record flags a discrepancy between the posted analysis and the recalculation.

What does over-dispersion mean in general?

For a continuous outcome, variation among individual scores is represented by the standard deviation. When scores vary substantially relative to the observed mean difference, the estimated group contrast tends to be less precise. In general, more variability makes it harder to distinguish a true group difference from random variation. Here, the standard deviations are 4.4 and 7.6, while the analyzed groups are small. Those features help explain why a point estimate alone is not enough to support a firm conclusion.

Over-dispersion is often discussed for count outcomes when observed variability exceeds what a simple model assumes. The primary outcome in this study is a continuous satisfaction score, so that count-data concept is not the principal model issue for the posted primary comparison. The record does not report diagnostic checks or alternative models.

What is not described about the analysis?

The record does not report whether the primary comparison was adjusted for baseline measures, how randomization was implemented, or which analysis population was used. It also does not describe the handling of missing CSQ values, the model for secondary outcomes, or any multiplicity adjustment. The registry lists four outcome measures and seven analyses, but does not provide enough detail to reconstruct every analysis or determine how the analyses were related. These omissions matter when interpreting a pilot study with multiple outcomes and missing endpoint data.

10. Confidence Intervals and Precision

The unadjusted recalculation estimates that the MACS group's mean CSQ score was 1.2 points higher than the attention control group's mean. The recalculated confidence interval ranges from -3.892 to 6.292. This interval is broad relative to the point estimate and crosses zero, so the posted summary statistics allow for a difference in either direction as well as no difference.

A confidence interval is a way to express statistical uncertainty around an estimate, given the method and assumptions used to calculate it. It is not a range containing every participant's score, nor does it show the probability that a particular value is true. In general, a wider interval indicates less precision. The interval reflects uncertainty in the group comparison, and its breadth is consistent with the small analyzed groups and variation in scores.

The registry's own analysis reports a two-sided t-test with p > 0.05, but does not report its interval estimate. The interval above is an unadjusted recalculation from the posted summaries, not an interval attributed to the sponsor. Because the record flags a discrepancy, the sponsor's analysis takes precedence. The sponsor's result and the recalculated interval both leave the comparison unresolved, but they are not interchangeable outputs.

The CSQ scale ranges from 8 to 32, but the registry does not state what size of group difference would be important to patients or clinicians. Statistical precision therefore cannot be translated into a judgment about practical importance from this record alone. The confidence interval describes uncertainty; it does not establish equivalence or rule out potentially relevant differences.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations based on the observed effect in this pilot, not the trial's own design assumptions. The original design assumptions and sample-size justification are not reported in the registry record. The planning calculation assumes a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. It uses the stated fraction of the observed effect as the effect to detect.

The figures are participants per arm for 80% power and participants per arm for 90% power. In plain terms, the 80% column gives the estimated number per group for a design with an 80% chance of detecting the assumed effect under the planning assumptions if that effect is present. The 90% column uses a higher target chance of detection, which generally requires more participants. These are design illustrations, not observed trial results, and this is not observed power.

The rows use 100% of the observed effect, 75% of the observed effect, and 50% of the observed effect. For example, planning for 75% of the observed effect gives 785 participants per arm for 80% power and 1,051 participants per arm for 90% power. Planning for 50% of the observed effect gives 1,766 participants per arm for 80% power and 2,364 participants per arm for 90% power. The figures for 100% of the observed effect are 442 participants per arm for 80% power and 591 participants per arm for 90% power. Smaller effects require more participants to detect under the same planning assumptions.

Because the observed effect comes from a small pilot comparison and is uncertain, these illustrations should not be treated as a definitive estimate of the sample size needed for a future study. They do not replace the trial's original design rationale, which is not reported in the registry record.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, Δ is the difference in means to detect and σ is the common standard deviation.

Worked calculation for this trial

1. Observed difference Δ = 1.200; pooled standard deviation σ = 6.362.
2. n = 2 × (1.96 + 0.842)² × 6.362² / 1.200² = 441.3, rounded up to 442 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
delta <- 1.2                      # observed difference in means
sigma <- 6.36238948823475                     # pooled standard deviation
n_per_arm <- function(power, frac = 1) {
  ceiling(round(2 * (z_a + qnorm(power))^2 * sigma^2 / (delta * frac)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (t-based, so very slightly larger)
cat("power.t.test:", round(power.t.test(delta = delta, sd = sigma, power = 0.80)$n, 1), "\n")

# Simulation: exact two-sample t-test using simulated means and variances
set.seed(3955250)
n    <- n_per_arm(0.80)
reps <- 20000
m_diff <- rnorm(reps, delta, sigma * sqrt(2 / n))             # difference in sample means
s2     <- sigma^2 * rchisq(reps, 2 * (n - 1)) / (2 * (n - 1))   # pooled sample variance
tstat  <- m_diff / sqrt(s2 * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(tstat) > qt(0.975, 2 * (n - 1))), reps))

Output from running the code

100% of observed effect: n per arm = 442 (80% power), 591 (90% power)
 75% of observed effect: n per arm = 785 (80% power), 1,051 (90% power)
 50% of observed effect: n per arm = 1,766 (80% power), 2,364 (90% power)
power.t.test: 442.2 
Simulated power at n = 442 per arm: 0.803 (20000 replicates)

The simulation draws many sets of group means and variances at the planned sample size and counts how often the two-sample t-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Analyses Reported by the Sponsor

The registry reports the sponsor's primary comparison as a two-sided t-test with p > 0.05. It does not post the exact p-value, test statistic, or a sponsor-reported confidence interval. The sponsor's result is therefore the primary reported inferential finding for this endpoint, and it does not indicate a statistically significant difference under the reported test.

An unadjusted recalculation from the posted group summaries estimates a mean difference of 1.2 points, with a confidence interval from -3.892 to 6.292 and p = 0.644. The record flags a discrepancy between the sponsor's analysis and the recalculation. The sponsor's own analysis takes precedence. The recalculation is best read as a secondary check based on the posted means, standard deviations, and analyzed group sizes, not as a correction to the sponsor's result.

The sponsor's result is reported only as p > 0.05, so it cannot be compared with the recalculated p-value at greater precision. Both outputs are consistent with the record not establishing a statistically significant difference, but their methods or inputs may differ; the reason for the discrepancy is not reported in the registry record. Neither result demonstrates that the groups are equivalent or that no meaningful difference exists.

14. Multiplicity, Missing Data and Analysis Populations

The registry lists a primary CSQ outcome and three secondary outcomes: the Brief Psychiatric Rating Scale, Brief Adherence Rating Scale, and Brief Cope Inventory. It records four outcome measures and seven analyses. When a study examines multiple outcomes or performs multiple analyses, the chance of obtaining at least one apparently unusual result can increase. In general, a multiplicity plan explains whether and how that issue is addressed. A correction or prespecified hierarchy is not reported in the registry record, so the interpretation of results across the full set of analyses cannot be assessed from the listed information.

Missing data affect the primary endpoint. The registry states that fewer participants were analyzed than enrolled because of missing data on this measure. The posted CSQ summaries include 10 participants in the MACS arm and 12 in the attention control arm. The record does not report why CSQ data were missing, whether missingness differed between groups, or whether the analysis used imputation or another approach. It also does not report sensitivity analyses addressing missing outcomes. If the observed scores differ systematically from unobserved scores, a comparison based only on available data may not represent the randomized groups fully; this is a general concern, not a claim about what happened in this trial.

The analysis population is not reported in the registry record. It is therefore unclear whether the primary analysis followed an intention-to-treat approach, a per-protocol approach, or another definition. The registry does not explain how participants were handled if they did not use an assigned app or had other protocol deviations.

These reporting gaps do not prove that the analysis was inappropriate. They limit the ability to evaluate the assumptions behind the sponsor's result and to judge how missingness and multiple analyses affect interpretation. The primary comparison should be read alongside these uncertainties.

15. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

Most frequent serious adverse event terms

Serious events with the highest arm-level rate: rehospitalization (47.4%), suicide attempt (10.5%).

The posted serious adverse event figures list 11 affected participants among 19 at risk in the MACS arm and 9 affected participants among 23 at risk in the mobile app attention control arm. The record also lists zero other adverse events among 19 at risk in MACS and zero among 23 at risk in the attention control arm. Deaths are listed as zero in both groups.

The serious-event terms identified as most common are Rehospitalization, at 47.4%, and Suicide attempt, at 10.5%. The registry does not assign these term percentages to a particular arm in the information reported for those terms. They should not be interpreted as arm-specific rates. The record does not state whether these terms refer to events or participants, or provide further detail about timing, severity, relatedness to the app, or how events were coded.

These counts are descriptive. They do not by themselves establish that the apps caused or prevented events, and the record does not report a statistical comparison of safety outcomes. The at-risk numbers in the safety listing differ from the analyzed numbers for the CSQ endpoint; they refer to the safety reporting, not the primary outcome analysis. Further safety definitions and follow-up details are not reported in the registry record.

16. What the Result Does, and Does Not, Mean

The primary posted result is a comparison of treatment satisfaction one month after treatment. The CSQ means were 23.7 for the MACS app and 22.5 for the mobile app attention control. The unadjusted mean difference is 1.2 points in the direction of a higher score for MACS. The sponsor's reported two-sided test gives p > 0.05, and the registry does not report its exact p-value or confidence interval.

The recalculated interval, from -3.892 to 6.292, spans zero and is wide enough to leave the direction and size of a possible difference uncertain. The result does not establish a statistically significant difference, but a non-significant result does not demonstrate that the groups are identical. The registry also does not define a threshold for a clinically or practically important CSQ difference, so the point estimate cannot be classified as meaningful or unimportant on this record alone.

The trial is described as a pilot focused on feasibility, acceptability, and preliminary effects. The primary satisfaction comparison does not answer every broader study aim. The record lists secondary measures of psychiatric symptoms, adherence, and coping, but the detail does not establish conclusions about those outcomes. The reported satisfaction result also does not establish effects on hospitalization or other clinical outcomes.

Interpretation is limited by the small analyzed groups and missing CSQ data. The registry does not report the missing-data strategy, analysis population, multiplicity approach, or a complete statistical plan. These omissions make it difficult to assess how robust the reported comparison is. The trial results are best understood as preliminary evidence from a pilot comparison, not a definitive determination about the relative effects of the two apps.

17. Why This Trial Matters Statistically

This trial is statistically informative because it illustrates the difference between an observed point estimate and the certainty of that estimate. The MACS group had a higher posted mean CSQ score than the attention control group, but the recalculated interval is broad and includes zero. Reading the estimate together with its uncertainty gives a more balanced account than focusing on the direction of the means alone.

The record also distinguishes a sponsor-reported analysis from an unadjusted recalculation based on posted summaries. The sponsor reports p > 0.05, while the recalculation gives p = 0.644 and an interval from -3.892 to 6.292. The registry flags a discrepancy, so the sponsor's own analysis takes precedence. This is a reminder that summary statistics may not reproduce an analysis if the underlying methods or inputs differ, and that recalculations should be labeled and interpreted cautiously.

Missing endpoint data and multiple listed outcomes are also important statistical considerations. The CSQ analysis includes fewer participants than were enrolled, and the record does not describe missingness handling. The registry lists four outcome measures and seven analyses, but does not report a multiplicity plan. These details affect how confidently readers can interpret a single result in the context of the full study.

Finally, the planning illustrations show how sample-size requirements depend on the effect chosen for planning. They use the observed pilot effect and explicit assumptions, rather than the original trial design rationale, which is not reported in the registry record. Such illustrations can inform statistical planning, but the pilot estimate is uncertain and should not be treated as a precise forecast of a future study's effect.

18. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The primary outcome analysis is based on 10 participants in the MACS arm and 12 in the attention control arm, and the registry says fewer participants were analyzed than enrolled because of missing data on this measure. The reasons for missingness, the amount and pattern of missing data beyond the posted group summaries, and the method used to handle missing values are not reported in the registry record. This limits assessment of possible bias from incomplete outcome data.

The study is labeled a pilot, and the record's brief summary describes a planned trial with n = 60 while the recorded enrollment is 42. The reason for this difference is not reported. The original sample-size rationale and statistical design assumptions are also not reported, so the planning illustrations derived from the observed effect should not be mistaken for the trial's design calculations.

The registry reports the sponsor's result as p > 0.05 but does not provide the exact p-value, test statistic, or sponsor-reported interval. A discrepancy is flagged between that analysis and an unadjusted recalculation from posted summaries. The reason for the discrepancy is not reported. The sponsor's result takes precedence, while the recalculation gives a view of uncertainty based on the summary statistics.

Further limitations include the absence of a reported multiplicity plan, analysis population definition, details about randomization implementation, and information about who was masked. The record also does not give a threshold for an important CSQ difference or enough detail to assess generalizability beyond the described population. These are reporting limitations; they do not establish that the study procedures were inadequate.

19. Related Tutorials

20. Related Statistical Calculators

21. Sources

• ClinicalTrials.gov: Mobile After-Care Support App: Pilot RCT — NCT03955250.
• Registry record last updated 2024-03-19; status last verified 2024-02.
• Results first posted 2023-08-02.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

22. Record Summary

NCT03955250 is a completed randomized, parallel-group, single-masked pilot RCT sponsored by Butler Hospital. It compares the Mobile After-Care Support (MACS) app with a mobile app attention control for patients following hospitalization related to psychotic-spectrum disorder. Recorded enrollment is 42; the brief summary describes a planned trial with n = 60. The study start date is 2019-05-31, and the primary completion date is 2022-05-31.

The primary outcome is the Client Satisfaction Questionnaire score at one month. Posted means are 23.7 for MACS and 22.5 for attention control. The registry reports a two-sided t-test with p > 0.05. An unadjusted recalculation estimates a mean difference of 1.2, with a confidence interval from -3.892 to 6.292; because a discrepancy is flagged, the sponsor's analysis takes precedence. The record notes missing CSQ data and does not report the missing-data method or the original sample-size justification.