A Proof-of-concept RCT of Version 3.0 of the Smoking Cessation Smartphone App "Smiling Instead of Smoking" (SiS): Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for A Proof-of-concept RCT of Version 3.0 of the Smoking Cessation Smartphone App "Smiling Instead of Smoking" (SiS) (NCT04672239), a randomized na study. The primary outcome is End of Treatment Group Differences in Smoking Self-Efficacy Questionnaire (SEQ).

2. Clinical Question

3. Trial Design

This was a randomized, parallel-group, open-label proof-of-concept trial with three behavioral conditions: the SiS Version 3 app, the QuitGuide app, and the Clearing the Air brochure. The registry record lists enrollment of 229 and a completed study. It identifies Massachusetts General Hospital as the sponsor. The study start date was 2021-01-28, and primary completion was 2022-08-10.

Each condition involved remote onboarding and a planned 7 weeks of use while participants quit smoking. The registry summary describes SiS as being tested against treatment as usual and against QuitGuide; the intervention list identifies the Clearing the Air brochure as the third condition. The registry record does not provide further detail about what treatment as usual entailed or how it related to the brochure condition.

“Randomized” indicates that assignment to study conditions was intended to use a chance-based process. The specific randomization sequence, allocation concealment procedures, and allocation ratio are not reported in the registry record. “Open-label” means masking was listed as none; the record does not describe any separate masking of outcome assessors or analysts.

The study belongs to a three-study development series, according to the registry summary. The earlier studies are described there as assessing feasibility and acceptability under different onboarding approaches. The current trial is identified as the third study and as proof-of-concept. That designation is important context: it describes the study’s role in developing and testing an intervention, rather than establishing a definitive clinical effect.

The planned primary assessment period extended up to 24 weeks post quit day. Treatment began 1 week before the quit date, with baseline at week -1. The record lists secondary outcomes involving abstinence, cigarette change, satisfaction, time applying materials, strategy use, perceived impact, appreciation, and app usage for SiS and QuitGuide. Specific visit schedules beyond the primary timeframe are not reported in the registry record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: One participant in each treatment arm decided against the study during the onboarding process and before starting any study procedures. These were excluded from analysis in a modified intent to treat approach for an enrolled and active sample of n=226.

Week 2 Survey

Reasons for not completing this period:

Week 6 Survey

Reasons for not completing this period:

3-Month Follow-up Survey

Reasons for not completing this period:

6-Month Follow-up Survey

Reasons for not completing this period:

Analysis population for the primary outcome, as defined in the registry: Some participants had missing data at various assessment time-points, but all enrolled participants provided data for at least one assessment.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: All randomized and starting study procedures (n=226; see Participant flow pre-assignment details). Some measures had missing data.

6. Primary Endpoint

The registry lists the primary outcome as End of Treatment Group Differences in Smoking Self-Efficacy Questionnaire (SEQ), measured over Up to 24 weeks post quit day. Note that the 7 weeks of treatment started 1 week prior to the quit date (baseline = week -1)..

The primary endpoint was the mean score on the Smoking Self-Efficacy Questionnaire (SEQ-12), assessed for end-of-treatment group differences. The questionnaire is described as a 12-item self-report measure of confidence in abstaining from smoking when facing internal or external cues. Responses use a 0-100 slider scale, from not at all confident to extremely confident. The scale score is the mean across items, and higher scores indicate greater self-efficacy to abstain.

Self-efficacy is a psychological construct: it captures a person’s reported confidence in carrying out a behavior under challenging circumstances. It is not itself a direct count of cigarettes smoked or a measure of abstinence. A mean score can summarize responses across the questionnaire items and permits comparisons of average reported confidence across trial groups.

The registry record identifies the endpoint as continuous and specifies the mean as the summary measure, with standard deviation as the dispersion measure. Standard deviation describes how spread out individual scores are around their group mean. It is distinct from the uncertainty in an estimated group mean or in a difference between group means.

The assessment timeframe is listed as up to 24 weeks post quit day, with treatment beginning 1 week before the quit date and baseline at week -1. The record does not report the group means, group difference estimates, confidence intervals, or statistical tests for this endpoint. Consequently, the measure and its intended interpretation can be described, but the magnitude and precision of any observed group differences cannot be assessed from the registry record.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The trial is described as randomized and parallel-group. In a randomized design, participants are assigned to conditions through a chance-based process, which aims to make groups comparable on average at the start of a study. Randomization does not guarantee identical groups, especially in an individual trial. The registry record does not describe the sequence-generation method, allocation concealment, or allocation ratio, so those features cannot be assessed from the posted information.

A parallel-group design assigns participants to one condition and compares outcomes across the conditions. Here, the three listed conditions were SiS, QuitGuide, and the Clearing the Air brochure. The registry record describes the study as open-label, with masking listed as none. Participants and study personnel may therefore have known which material was assigned; the record does not explain whether any outcome assessment or analysis was separately masked.

The primary outcome is continuous, and the registry identifies the mean and standard deviation as its summary statistics. A group mean gives the average questionnaire score. The standard deviation describes variation in participant scores around that average. A comparison of means focuses on the difference between group averages, while also needing an estimate of uncertainty. The record does not report the observed means, standard deviations, or comparative estimates.

What would an uncertainty interval communicate?

In general, a confidence interval expresses the statistical uncertainty around an estimated quantity, such as a difference in means. A narrower interval indicates greater precision under the analysis assumptions; a wider interval indicates less precision. An interval should be interpreted in relation to the scale and the question being asked, not as a guarantee that the true value lies within a particular realized interval. No confidence intervals for the primary comparison are reported in the registry record.

How should a significance test be read?

In general, a p-value measures how unusual results at least as incompatible with a stated null hypothesis would be if that hypothesis and the test assumptions were true. It is not the probability that the null hypothesis is true, nor does it quantify the size or practical importance of an effect. A p-value alone also does not show the precision of an estimate. The registry record does not report p-values for the primary endpoint or identify the test used.

What can the planning information establish?

The registry record describes a planning illustration based on an observed effect, with a two-sided significance level of 0.05, equal allocation between two arms, and a normal approximation. It says the effect to detect is a stated fraction of the observed effect. For time-to-event outcomes, the illustration uses a total number of events; for other outcomes, it uses participants per arm. The numerical planning outputs and the specific effect fractions are not reported in the registry record.

These planning illustrations are not the trial’s original design justification. The record explicitly distinguishes them from the trial’s own design. They should be read as hypothetical planning scenarios, not as observed power or as evidence that the trial had a particular probability of detecting an effect. The original sample-size justification is not reported in the registry record.

What analysis details remain unknown?

The registry lists 13 outcome measures and 21 analyses, but it does not describe the statistical models, covariates, handling of repeated measurements, or the specific comparisons included in each analysis. It also does not report whether the primary outcome was analyzed at a single assessment or across multiple time points. The outcome timeframe is listed as up to 24 weeks post quit day, but the exact primary assessment visit is not stated.

For a continuous outcome, analysts may use methods that compare group means, potentially adjusting for baseline measurements or other prespecified factors. Those are general options, not methods that can be attributed to this trial. The analysis model, assumptions, and any diagnostic checks are not reported in the registry record. Without the estimates and methods, readers cannot assess whether the reported data support a difference, how precise any difference is, or how sensitive the result may be to analysis choices.

10. Multiplicity, Missing Data and Analysis Populations

The registry record lists 13 outcome measures and 21 analyses. When a study examines multiple outcomes, time points, or group comparisons, the chance of obtaining at least one apparently notable result can increase if every test is treated as an isolated finding. General approaches to address this issue include prespecifying a primary comparison, adjusting the significance threshold, or interpreting a family of results together. The registry record does not report a multiplicity strategy or enough detail to determine which analyses were treated as confirmatory.

The primary outcome is identified as end-of-treatment group differences in SEQ scores. However, the record does not report the primary analysis model, the planned pairwise contrasts, or whether any adjustment was intended for the comparisons involving three conditions. It also does not report whether secondary outcomes were considered exploratory or how their results were to be interpreted. Therefore, the number of registered outcomes and analyses is useful context, but it cannot establish how the investigators addressed multiplicity.

Missing data are explicitly noted: some participants had missing data at various assessment time points, while every enrolled participant provided data for at least one assessment. This does not indicate how many observations were missing for the primary endpoint or whether missingness differed across conditions. The method for handling missing outcome data, any assumptions about the reasons for missingness, and any sensitivity analyses are not reported in the registry record.

The analysis population is also not described. The record does not state whether the primary analysis included all randomized participants, only participants with an observed end-of-treatment score, or another defined population. The number analyzed for each outcome is not reported in the registry record. These omissions limit assessment of how missing data and participant inclusion could affect interpretation.

11. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry record reports serious adverse events for the SiS app, QuitGuide app, and Clearing the Air brochure groups. In each group, the reported number of participants at risk was 80, 75, and 71, respectively; the serious adverse event count was 0 for each. Deaths were also reported as 0 in each group. The record lists no serious adverse event terms.

For other adverse events, the record reports 0 in each group, with participants at risk listed as 80 for SiS, 75 for QuitGuide, and 71 for Clearing the Air. These are the posted counts. They do not supply additional detail about event definitions, ascertainment procedures, observation duration for safety, or the handling of participants who did not complete follow-up; those details are not reported in the registry record.

A zero count means no events of the listed type were recorded in the registry results for the stated group. It does not establish that an event is impossible, that every participant was observed for the same length of time, or that the groups have identical safety profiles. The record does not report comparative safety estimates or uncertainty intervals. The figures should therefore be read as descriptive counts rather than as a statistical demonstration of equivalence between conditions.

12. What the Result Does, and Does Not, Mean

The registry record establishes that a randomized, three-condition proof-of-concept study was completed and identifies the main outcome and the materials participants were assigned to use. Its primary question concerned group differences in self-reported confidence to abstain from smoking, measured with the SEQ-12. The study also registered outcomes related to abstinence, cigarette change, experience with the materials, strategy use, and app usage.

The record does not report the primary-outcome estimates or results for the secondary outcomes. It therefore does not show whether mean self-efficacy differed among the groups, whether any difference was statistically distinguishable from chance under a specified analysis, or how precise a difference might be. The stated research objective should not be mistaken for an observed finding.

Even if outcome estimates were available, a self-efficacy score would represent reported confidence, not a direct measure of smoking abstinence. The record’s list of secondary outcomes includes self-reported abstinence and cigarette change, but their numerical results are not reported. Each outcome would answer a different question, and a change in one measure would not automatically establish change in another.

The registry information also does not establish how participants used the assigned materials in practice, beyond the plan that they would be asked to use them for 7 weeks. Actual app usage is listed as a secondary outcome for SiS and QuitGuide, but those results are not reported. Treatment adherence, engagement, and outcome completeness cannot be inferred from enrollment or from the assigned intervention descriptions.

In general, a trial’s results need to be considered alongside the estimate’s uncertainty, the analysis population, missing-data handling, and the prespecified analysis plan. Those details are not reported for the outcome comparisons in this record. The posted information can support a careful description of the study’s design and measures, but not a conclusion that one condition improved smoking cessation or self-efficacy relative to another.

13. Why This Trial Matters Statistically

This trial is statistically informative as an example of a randomized, parallel-group evaluation of behavioral support delivered through different formats: a purpose-built smartphone app, an existing smartphone app, and a brochure. Its proof-of-concept role places the emphasis on testing an intervention and its measurement approach. That context matters when distinguishing an early evaluation from a definitive test of an intervention’s effects.

The primary measure captures a construct that is relevant to quitting behavior but differs from smoking behavior itself. Identifying the SEQ-12 as a continuous outcome, specifying its range and direction, and stating that scores are averaged across items help clarify what a group comparison would mean. This also illustrates why outcome interpretation depends on the measure: a difference in confidence scores is not automatically a difference in abstinence.

The study includes three conditions and multiple registered outcomes and analyses. That makes clear reporting of the primary comparison, analysis choices, and multiplicity handling especially important for readers interpreting results. The registry record identifies the primary endpoint but does not report those analysis details or the estimates needed to evaluate the comparison.

The missing-data statement is another important statistical feature. Participants contributed at least one assessment, but some had missing data at various time points. Reporting the number assessed at each visit and describing the analysis approach would help readers judge the completeness and robustness of outcome comparisons. Those details are not reported in the registry record.

Finally, the posted safety counts and intervention descriptions are distinct from evidence about comparative outcomes. The record is useful for understanding the trial’s structure and registered questions, while its lack of posted efficacy estimates limits what can be concluded about the intervention comparisons. That distinction is central to responsible statistical interpretation.

14. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The principal limitation for statistical interpretation is that the registry record does not report results for the primary endpoint or the secondary outcomes. Group means, comparison estimates, uncertainty intervals, p-values, and numbers analyzed are not reported. As a result, the direction, magnitude, and precision of outcome differences cannot be evaluated.

Some participants had missing data at various assessment time points. Although the record states that all enrolled participants provided data for at least one assessment, it does not report missingness by group or time point, the reason for missing data, or the method used to address it. The analysis population and any sensitivity analyses are not reported in the registry record.

The trial is open-label, with masking listed as none. The registry record does not describe whether outcome assessors or analysts were masked, or how self-reported outcomes were collected. The randomization method and allocation concealment procedures are also not reported, limiting assessment of these design features.

The registry identifies 13 outcome measures and 21 analyses but does not report a multiplicity plan, detailed statistical analysis plan, or the specific comparisons tested. It also does not report an original sample-size justification. The numerical outputs of the planning illustrations are not reported in the registry record, and those illustrations are described as separate from the trial’s own design.

The safety results are reported as counts, without comparative estimates or uncertainty intervals. Definitions, ascertainment details, and follow-up duration for adverse events are not reported. These limitations mean that the posted record supports description of the trial structure and registered outcomes, but not a full appraisal of efficacy, precision, missing-data sensitivity, or comparative safety.

15. Related Tutorials

16. Related Statistical Calculators

17. Sources

• ClinicalTrials.gov: A Proof-of-concept RCT of Version 3.0 of the Smoking Cessation Smartphone App "Smiling Instead of Smoking" (SiS) — NCT04672239.
• Registry record last updated 2024-09-19; status last verified 2024-04.
• Results first posted 2024-09-19.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

18. Record Summary

NCT04672239 is a completed, randomized, parallel-group, open-label proof-of-concept trial sponsored by Massachusetts General Hospital. It enrolled 229 participants with nondaily smoking and compared the SiS Version 3 smartphone app, the QuitGuide app, and the Clearing the Air brochure. The study start date was 2021-01-28, and primary completion was 2022-08-10.

The primary outcome was end-of-treatment group differences in mean SEQ-12 scores, a self-reported measure of confidence in abstaining from smoking. The listed timeframe extended up to 24 weeks post quit day; treatment began 1 week before the quit date, with baseline at week -1. The registry record notes missing data at some assessment time points and states that all enrolled participants provided data for at least one assessment.

The record lists 13 outcome measures and 21 analyses. It reports 0 serious adverse events and 0 deaths in each of the three groups. Numerical primary and secondary outcome results, analysis details, and missing-data methods are not reported in the registry record.