A Pilot Study Testing a Life Skills Application to Address Interpersonal Relationships in College: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for A Pilot Study Testing a Life Skills Application to Address Interpersonal Relationships in College (NCT03316911), a randomized na study. The primary outcome is 3-Month Usability.

2. Clinical Question

3. Trial Design

The registry identifies this as a randomized, parallel-group pilot study with no masking. It lists the phase as NA and the status as completed. Enrollment was 261. The study was sponsored by the University of Michigan and focused on adapting a web-based application for use with college students. The adaptation process was intended to address the needs of a diverse student population and to emphasize life skills and holistic self-care in support of primary prevention programming for incoming students. The intervention arm was assigned MKit, while the control arm was assigned the usual standard of care training package for incoming students. The registry describes that usual package as including the Haven online module about healthy relationships and sexual violence and Relationship Remix, an interactive peer activity. The intervention-arm description also begins with this usual package, but the registered description is truncated. Details beyond the listed material are not reported in the registry record. The primary endpoint was three-month usability, collected in the intervention group only. It was not registered as a direct comparison of usability scores between the randomized arms. The listed secondary outcomes were three-month retention and five-month retention. The brief summary says retention was to be monitored over the academic year, but the registry record does not report retention results. The study began on 2017-10-22, with primary completion on 2018-04-28. The registry record does not report the allocation ratio, a sample-size justification, or additional details of the randomization procedure. It also does not report a statistical analysis plan. Random assignment describes the design; without reported outcome analyses, it does not establish what results were observed or how precisely they were estimated.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Reasons for not completing this period:

Analysis population for the primary outcome, as defined in the registry: Only intervention participants were asked the items on the System Usability Scale.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as 3-Month Usability, measured over 3-months.

The primary endpoint was three-month usability of the WebApp, measured with the 10-item System Usability Scale in the intervention group only. Each item is rated from 1 (Strongly Disagree) to 5 (Strongly Agree). To form a final score, the scoring procedure subtracts one from each odd-item response, subtracts each even-item response from 5, sums the resulting values, and multiplies that sum by 2.5. The resulting score ranges from 0 to 100. The registry description states that a score above 68 is average. Usability is relevant to an application intended to reinforce existing prevention programming: participants’ assessments can indicate whether they find the application usable. This is a questionnaire-based measure of participants’ ratings, not by itself a measure of prevention outcomes or of whether the application changes behavior. The registered endpoint also does not compare usability ratings with a control-group score, because only intervention participants were asked the System Usability Scale items. The registry record does not report the mean score, its standard deviation, the number of participants contributing scores, or an uncertainty interval. It therefore does not establish the level or precision of usability ratings in this study. Nor does the description of the scale’s above-average threshold tell readers whether the study’s participants scored above that threshold.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

What does the registered primary measure capture?

The primary outcome is a continuous score from the System Usability Scale, collected at three months in the intervention group only. A continuous measure can take values across a range rather than belonging to a small set of categories. The registry specifies that the outcome is summarized as a mean with a standard deviation. The mean describes the average score in the analyzed group; the standard deviation describes how spread out individual scores are around that average. These summaries answer different questions: an average does not show whether participants’ ratings were tightly clustered or varied widely.

The scale is constructed from 10 item responses, each rated from 1 to 5, using the scoring steps described in the endpoint definition. Its final range is 0 to 100. The score’s bounded range matters when interpreting summaries: observations cannot extend beyond the scale limits, and a distribution may be uneven or concentrated near a limit. In general, the mean and standard deviation are useful descriptive summaries, but their interpretation depends on the pattern of individual observations. The registry record does not report that pattern or the score summaries for this study.

Why is this not a treatment-effect comparison?

Randomized parallel-group designs can support a comparison between assigned groups when an outcome is measured in both groups and the analysis is appropriate. Here, the registry says that only intervention participants were asked the usability items. The registered endpoint therefore describes usability ratings in that group; it does not provide a randomized contrast with Standard of Care. A difference between groups cannot be estimated for this endpoint from the registry description because the control group’s usability score is not part of the measure.

In general, a confidence interval around an estimate communicates its statistical uncertainty under the chosen method and assumptions. A narrow interval indicates greater precision than a wide interval, while neither width nor statistical significance establishes practical importance on its own. No estimate or confidence interval for this primary outcome is reported in the registry record. A p-value is also not reported. Without these results, readers cannot evaluate the observed score, its precision, or any formal test concerning the primary endpoint.

What can randomization and masking tell readers?

The registry lists randomized allocation, a parallel model, and no masking. Random allocation is a design feature intended to make assigned groups comparable on average, though it cannot ensure balance in every realized study. The record does not report the allocation ratio or the randomization procedure, so those details should not be inferred. No masking means the registry does not describe the study as blinded. The record does not state which people, if any, knew assignment during outcome collection or analysis beyond that designation.

These design details need to be interpreted alongside the endpoint. Randomization does not make a single-arm usability score into a between-arm effect estimate. Also, because the System Usability Scale is based on participant ratings, responses capture participants’ reported assessments. The registry record does not report whether any additional objective measures of application use were collected or how any such measures were analyzed.

How should missing statistical details be read?

The record lists 16 outcome measures and 0 analyses. It does not report the statistical model, the analysis population, handling of missing questionnaire responses, or an approach to uncertainty estimation. Those details are needed to understand how an outcome summary would be derived from participant data. For example, the number enrolled and the number contributing to an endpoint can differ if some participants do not complete that measure. The record does not report the number contributing to the three-month usability endpoint, so those quantities cannot be equated.

In general, missing data can affect an estimate when the participants with observed measurements differ systematically from those without them. Methods for addressing missingness rely on assumptions, and those assumptions should be described alongside the analysis. No missing-data method is reported in the registry record. The listed retention outcomes are relevant to follow-up, but their presence does not establish how complete follow-up was or how retention was analyzed.

What is the role of planning calculations?

A sample-size calculation is a planning tool based on assumptions about the endpoint, variability, target effect, allocation, and error rates. It is not a result and cannot substitute for an observed estimate. The registry record does not report the trial’s sample-size justification or the assumptions behind its enrollment target. The primary usability endpoint was measured in one arm only, so a two-arm planning calculation for a usability difference would not correspond to that registered endpoint. Any planning illustration should be read as a hypothetical design calculation, not as observed power or evidence about the study result.

More broadly, statistical interpretation requires matching the question, outcome, design, analysis, and reported estimate. For MKit, the registered primary question concerns intervention-group usability, while the randomized design also includes a Standard of Care group. The registry record does not report the analysis needed to assess the primary endpoint or results for the secondary retention outcomes. General statistical principles can clarify what the planned measure means, but they cannot fill in missing trial estimates.

10. Confidence Intervals for Each Arm

No two-arm comparison is shown for this trial, but each arm's mean can be given with a t-based 95% confidence interval computed from the posted mean, spread and number of participants.

R code and output

arm <- c("MKit WebApp Intervention")
n   <- c(93)
m   <- c(67.66)
sp  <- c(14.87)
sd  <- sp      # posted standard deviations
for (i in seq_along(n)) {
  se <- sd[i] / sqrt(n[i]); h <- qt(0.975, n[i] - 1) * se
  cat(sprintf("CARM|%s|%d|%.3f|%.3f|%.3f|%.3f\n", arm[i], n[i], m[i], sd[i], m[i] - h, m[i] + h))
}

CARM  MKit WebApp Intervention  93  67.660  14.870  64.598  70.722

11. Multiplicity, Missing Data and Analysis Populations

The registry record lists 16 outcome measures and 0 analyses. It identifies three-month usability as the primary endpoint and lists three-month and five-month retention as secondary outcomes. The brief summary also describes aims concerning acceptability and willingness to use the application, but the registry entry does not report results for these aims. The names of all outcome measures beyond those specified are not included in the registry record for this account. When a study evaluates multiple outcomes, the chance of at least one apparently unusual result can increase if each is tested separately without accounting for the multiplicity. In general, investigators may address this through a prespecified hierarchy, adjusted significance thresholds, or by emphasizing estimation and uncertainty rather than isolated tests. The registry record does not report whether any such approach was planned. It also does not report an analysis plan specifying how the primary and secondary outcomes were ordered or interpreted together. The number enrolled does not establish how many participants contributed to each outcome. The record does not report the analyzed population for usability or retention, the number of missing responses, reasons for missingness, or any method for handling missing data. It does not state whether analyses followed assignment regardless of participation, included only questionnaire completers, or used another population definition. These omissions limit assessment of possible selection from incomplete follow-up. The listed retention outcomes do not, by themselves, establish how many participants were retained or how retention was calculated.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

For MKit WebApp Intervention, the registry lists 122 participants at risk for serious adverse events and 0 serious adverse events; it also lists 122 at risk for other adverse events and 0 other adverse events. For Standard of Care, it lists 139 participants at risk for serious adverse events and 0 serious adverse events, and 139 at risk for other adverse events and 0 other adverse events. Deaths are listed as 0 in each group. No top serious adverse-event terms are listed. These figures describe what the registry reports; they do not show whether every possible event was collected or whether participants had no experiences outside the recorded categories. A zero count means no events in that category are listed for the stated at-risk group in this record. It does not establish that the intervention caused no harm or that the groups had equivalent safety. The registry record does not report event definitions, severity assessment details, follow-up completeness, or a comparative statistical analysis of adverse events. The at-risk counts are safety denominators and should not be assumed to be the numbers included in the primary usability analysis.

13. What the Result Does, and Does Not, Mean

The study was designed as a pilot of an adapted life skills web application in the context of incoming-student primary prevention programming. Its primary registered outcome focuses on whether intervention participants rated the WebApp as usable at three months. The registry describes the System Usability Scale and its scoring, but it does not report the study’s resulting score. As a result, the record does not establish whether participants’ ratings were high or low, how variable they were, or whether they met the stated scale benchmark. The outcome also has a specific scope. Because usability questions were asked only of intervention participants, the registered primary measure does not estimate a usability difference between the MKit and Standard of Care groups. It is not a direct measure of whether the application prevents sexual violence, changes relationships, or improves other outcomes. The study hypothesis about usability, acceptability, and willingness to use the application should be distinguished from findings; the registry entry does not provide results that confirm or refute those hypotheses. The secondary outcomes of three-month and five-month retention point to follow-up as another study interest. Their results are not reported in the registry record. Enrollment of 261 and the listed safety counts do not reveal completion of follow-up or endpoint-specific participation. With no posted outcome analyses, estimates, or uncertainty measures, the record cannot support conclusions about the size or precision of usability or retention outcomes. More generally, a pilot study can inform questions about implementation and measurement, but its interpretation depends on what was measured and reported. Here, the record describes an adaptation effort, randomized assignment, and a usability assessment, while leaving the numerical outcome findings unavailable. That absence is not evidence that the application succeeded or failed; it means the registered results cannot be evaluated from this record alone.

14. Why This Trial Matters Statistically

This trial is statistically notable because its randomized design and primary endpoint answer different kinds of questions. Random assignment placed participants in parallel intervention and Standard of Care groups, but the primary usability measure was collected only from intervention participants. That measure can describe reported usability in the intervention group, but it does not yield a randomized comparison of usability between groups. Keeping the design distinct from the endpoint prevents an overstatement of what randomization can establish for this particular outcome. The study also illustrates how implementation questions differ from questions about downstream outcomes. The application was adapted for a diverse college population and intended to reinforce existing primary prevention programming. A usability score can address participants’ ratings of the application, while retention can describe continued participation over follow-up. Neither is interchangeable with an outcome measuring prevention effectiveness. The record identifies usability as primary and retention as secondary, but does not report their estimates or analyses. The missing analysis results are consequential for statistical interpretation. Without a mean, standard deviation, participant count for the endpoint, or uncertainty estimate, the observed level and precision of usability cannot be assessed. Without reported retention outcomes, follow-up cannot be characterized from the registry entry. The study therefore offers a clear example of why registry objectives and design features should not be mistaken for results. Statistical conclusions depend on estimates and their context, not only on the existence of a randomized study or a named measurement instrument. The record’s outcome list and analysis count further underscore the distinction between registered outcomes and reported evidence. Sixteen outcome measures are listed, but 0 analyses are reported. Transparent reporting of estimates, denominators, missingness, and analysis methods would be necessary to interpret the pilot’s findings and judge how they relate to its stated aims.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The primary endpoint is limited to intervention participants’ three-month usability ratings. The registry record does not report a corresponding control-group usability assessment, so the endpoint cannot directly estimate a group difference. It is a questionnaire-based measure and reflects participants’ ratings, rather than independently documenting application use or prevention outcomes. The record does not report the observed score, its variability, or the number of participants who completed the measure. Several design and analysis details are not reported in the registry record. These include the allocation ratio, the sample-size justification, the analysis population, the statistical model, the handling of missing responses, and the approach to multiplicity across 16 outcome measures. The record lists 0 analyses, and it does not report results for the primary endpoint or the three-month and five-month retention outcomes. Without those details and results, it is not possible to assess precision, completeness of follow-up, or whether the analyses addressed the registered questions. The intervention was intended as reinforcement to usual primary prevention programming, rather than described as a replacement for it. The registered description of the usual training package is incomplete, and further intervention details are not reported in the registry record. The record also identifies no masking. These features make it important not to infer more about delivery, participant awareness, or outcome assessment than the registry states. Finally, the title identifies the project as a pilot, and the stated aims include adaptation and usability. The available registry entries do not establish whether the adaptation met its aims, whether participants were willing to use the application, or whether retention differed by assignment. Those conclusions require results that are not reported in the registry record.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: A Pilot Study Testing a Life Skills Application to Address Interpersonal Relationships in College — NCT03316911.
• Registry record last updated 2019-05-10; status last verified 2019-05.
• Results first posted 2019-05-10.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT03316911 is a completed, randomized, unmasked, parallel pilot study sponsored by the University of Michigan. It enrolled 261 participants and evaluated MKit, an adapted web application focused on life skills and holistic self-care as reinforcement to incoming-student primary prevention programming. The primary endpoint was three-month usability, measured with the 10-item System Usability Scale among intervention participants only. Three-month and five-month retention were listed as secondary outcomes. The registry record lists 16 outcome measures and 0 analyses. It does not report the primary usability result, retention results, analysis populations, missing-data methods, or a statistical analysis plan. Serious adverse events, other adverse events, and deaths are each listed as 0 in both arms. These entries describe the registered record but do not supply outcome estimates or establish comparative effects.