Artificial Intelligence to Measure Adherence to Oral Medication: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Artificial Intelligence to Measure Adherence to Oral Medication (NCT04046497), a randomized phase 4 study. The primary outcome is Service Utilization Report Form.

2. Clinical Question

This is a question about a specific reported service-utilization outcome during a stated follow-up period. It is not a direct measure of medication adherence: the record’s primary outcome is the Service Utilization Report Form, and its description concerns psychiatric emergency-room visits or psychiatric hospitalization. The trial’s stated project aim was to use an AI smartphone app to support medication adherence, but the primary outcome should be interpreted according to what it measures. The record does not describe a separate posted adherence outcome or secondary outcome results. It also does not state whether the primary measure counts a participant once if either type of service use occurred, or how overlapping event types were handled. The available outcome counts can therefore be read as participant counts for the reported measure, without inferring more detailed event patterns.

3. Trial Design

The registry record describes a randomized, parallel-group phase 4 trial, with the status marked completed. The two listed arms were Smartphone App and Usual Care. The app was described as an AI smartphone app intended to support medication adherence; usual care was described as care provided at a CSC clinic. The record lists masking as none, so it does not report masking of participants or other trial parties.

The stated enrollment was 139. For the primary outcome, the posted arm counts are 11.0 participants in the Smartphone App arm and 15.0 in the Usual Care arm, with outcome denominators of 59.0 and 63.0, respectively. The registry record does not explain the relationship between the enrollment figure and the denominators reported for this outcome. These figures should therefore be read as separate registry entries rather than reconciled by assuming a particular analysis population or accounting for participants.

The listed start date is 2020-11-20, and the listed primary completion date is 2025-03-31. Northwell Health is named as the sponsor. The intervention type for both listed arms is OTHER. The record does not provide details about random-sequence generation, allocation concealment, the timing or mechanics of assignment, sites beyond the CSC clinic description, or the intervention’s delivery and adherence monitoring. It also does not report the original design assumptions or a statistical analysis plan. Randomization identifies the intended comparative design, but the record does not give enough operational detail to assess how assignment was implemented or how the final outcome analysis handled participants and missing observations.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Service Utilization Report Form, measured over 12 months.

The primary endpoint is titled Service Utilization Report Form. Its unit is participants, its parameter is a count of participants, and its timeframe is 12 months. The description says it provides participant utilization of psychiatric ER visits or psychiatric hospitalization. In the posted results, the Smartphone App arm has 11.0 participants with the outcome among 59.0, and the Usual Care arm has 15.0 among 63.0.

A participant-level measure can summarize whether the specified service-utilization outcome was recorded for a participant, rather than how many total service events occurred. That distinction matters: the posted measure is a count of participants, not a count of visits or hospitalizations. The registry record does not specify whether one or both types of service use were required, how multiple events were counted, or how the form classified participants with both types of service use.

This endpoint is relevant to the stated project because it records psychiatric service utilization during follow-up. However, it is not itself a direct measurement of medication-taking or app engagement. The trial’s stated aim concerned support for medication adherence, while the primary endpoint measures utilization of psychiatric emergency services or hospitalization. Those are distinct concepts, and the results should be described as service-utilization findings rather than as a direct adherence result.

7. Results Posted in the Registry

• Smartphone App: 18.6% (11/59) versus 23.8% (15/63) on control; risk difference -5.2 percentage points (95% CI -19.6 to 9.3), risk ratio 0.78 (0.39–1.56); unadjusted two-proportion z-test p = 0.486.

8. Statistical Interpretation

9. Statistical Methods Explained

The primary endpoint is binary at the participant level: a participant is counted as having the recorded service-utilization outcome or not. The registry record describes the measure as participant utilization of psychiatric ER visits or psychiatric hospitalization, and the analysis summary labels the outcome as participants with an event. In a binary-outcome comparison, the arm-level count is interpreted in relation to the number of participants with an outcome assessment. The posted denominators for this outcome are 59.0 for Smartphone App and 63.0 for Usual Care.

What does the risk difference describe?

The risk difference compares the event proportions between arms by subtracting one arm’s proportion from the other, according to the comparison’s direction. Here, the reported estimate is -0.052. A negative estimate means the observed proportion is lower in the arm represented as the intervention relative to the control arm under that direction. This is an absolute contrast on the proportion scale. It is not a relative percentage change, and it should not be converted into a percentage or another measure without doing a separate calculation.

The estimate is unadjusted: it compares the posted arm-level outcome summaries without a reported covariate-adjustment model. The registry record lists no analyses. It does not report whether the sponsor used a different primary analysis, which covariates were considered, or whether any adjustments were planned. Thus, the posted comparison can be described as an unadjusted summary, but it cannot stand in for an unreported sponsor analysis.

How should the p-value be read?

The posted unadjusted p-value is 0.486. A p-value describes how compatible the observed data are with a specified null model, under the assumptions of the test used. It is not the probability that the null hypothesis is true, nor does it measure the size or clinical importance of an effect. The registry record does not identify the test used to produce this p-value. It should therefore be read as the posted unadjusted value, not as evidence about an undocumented analysis procedure.

The p-value is not a substitute for the estimated contrast and its interval. The risk-difference interval extends from -0.196 to 0.093, including zero. That interval conveys a range of values compatible with the data under the interval method’s assumptions. Because the interval covers negative and positive contrasts, the direction and magnitude remain uncertain. A p-value above a conventional threshold does not demonstrate equivalence or rule out effects in either direction.

Why are several effect measures reported?

The risk ratio and odds ratio express relative contrasts, whereas the risk difference expresses an absolute contrast. The reported risk ratio is 0.783, with an interval from 0.392 to 1.565. The reported odds ratio is 0.733, with an interval from 0.306 to 1.759. For a risk ratio or odds ratio, the null value is 1; for a risk difference, the null value is 0. The intervals include their respective null values. These measures use distinct scales and should not be treated as interchangeable, particularly when communicating absolute event frequencies.

Over-dispersion is a concern in some count-data settings when the variability exceeds what a model assumes. General statistical explanation: accounting for over-dispersion can prevent uncertainty from being understated when a count model’s variance assumptions do not fit the data. It is not a stated feature of this endpoint, which is recorded as a binary participant outcome, and the registry record does not report an over-dispersion model or adjustment. The concept should not be used to infer what analysis the trial conducted.

What does randomization contribute?

Random assignment is intended to create comparable groups on average, reducing systematic differences at the point of assignment. It does not guarantee that every characteristic is balanced in a particular trial, and it does not remove uncertainty from the observed estimates. The record says the trial was randomized, but it does not report sequence generation, concealment, baseline characteristics, or the details of the implemented randomization procedure. The data therefore support describing the design as randomized, while leaving those implementation details unassessed from the registry record.

The reported outcome denominators are 59.0 and 63.0, while the stated enrollment is 139. The registry record does not explain the difference between these entries. It also does not report whether the analysis followed an intention-to-treat, per-protocol, or other population definition. No assumptions about exclusions, follow-up, or outcome availability should be made in the absence of that information.

10. Confidence Intervals and Precision

The risk-difference estimate is -0.052, with a confidence interval from -0.196 to 0.093. A confidence interval expresses statistical uncertainty around an estimate under the method’s assumptions; it is not a guarantee that the true value lies in the interval. This interval includes zero, so it is compatible with no risk difference as well as with negative or positive differences. Its breadth indicates that the posted estimate is not highly precise on the risk-difference scale.

The relative-effect intervals also include their null values. The risk ratio is 0.783, with an interval from 0.392 to 1.565; the odds ratio is 0.733, with an interval from 0.306 to 1.759. For these relative measures, the null value is 1. Their intervals span values on both sides of that null, reflecting uncertainty about direction and magnitude. These intervals should be interpreted on their own scales, not as interchangeable with the risk-difference interval.

The posted unadjusted p-value is 0.486. The registry record does not identify the interval construction method or the statistical test used, so the precise assumptions behind those summaries are not reported in the registry record. Taken together, the estimates and intervals do not establish equivalence or rule out a meaningful contrast. They describe uncertainty in the posted comparison, and the limited outcome details and unreported analysis population add context to that uncertainty.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations expressed as participants per arm. They are not total-event counts. The assumptions listed for the illustration are a two-sided significance level of 0.05, equal allocation between the two arms, a normal approximation, and an effect to detect set as the stated fraction of the observed effect. The original trial’s own design assumptions are not reported in the registry record.

The 80% power column gives the illustrative number of participants per arm for a design that would have 80% power under those assumptions, if the specified effect were the effect to detect. The 90% power column makes the corresponding illustration for 90% power. These are prospective planning quantities under a stated scenario, not descriptions of the number enrolled in this completed trial and not estimates of its achieved power. This is not observed power.

The rows represent planning for 100% of the observed effect, 75% of the observed effect, and 50% of the observed effect. For those rows, the listed participants per arm for 80% power are 983, 1,787, and 4,105; the corresponding participants per arm for 90% power are 1,315, 2,391, and 5,495. The pattern illustrates a general planning principle: smaller effects are harder to distinguish from sampling variation and therefore require more participants under otherwise fixed assumptions.

The figures depend on the observed effect used as the starting point and on the stated planning assumptions. They should not be read as a recommendation for a future study or as evidence that the trial was designed to detect the observed contrast. No original sample-size justification is reported in the registry record. The accompanying curve should be read as a visual representation of the same planning relationship, not as a separate trial result.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.238; treatment proportion p₁ = 0.186; average p̄ = 0.212.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.5783 = 1.1334.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.5771 = 0.4857.
4. n = (1.1334 + 0.4857)² / (-0.0517)² = 982.6, rounded up to 983 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.238095238095238                          # control proportion
p1 <- 0.186440677966102                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(4046497)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 983 (80% power), 1,315 (90% power)
 75% of observed effect: n per arm = 1,787 (80% power), 2,391 (90% power)
 50% of observed effect: n per arm = 4,105 (80% power), 5,495 (90% power)
power.prop.test: 982.6 
Simulated power at n = 983 per arm: 0.801 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Multiplicity, Missing Data and Analysis Populations

The registry record lists one outcome measure and no posted secondary outcomes. It lists no analyses. It does not report a multiplicity strategy, such as how inference would be handled across multiple outcomes, time points, or comparisons. With no posted secondary results or analysis details, it is not possible to determine whether any unreported multiplicity considerations affected the analysis. No multiplicity adjustment should be inferred from the absence of posted analyses.

Missing-data methods are not reported in the registry record. The record does not say how missing outcome assessments, incomplete follow-up, or participants without a recorded service-utilization form were handled. It also does not give the number or pattern of missing primary-outcome observations. The enrollment figure is 139, while the posted primary-outcome denominators are 59.0 and 63.0; the record does not explain the relationship between these entries. That difference should not be interpreted as a specific amount or pattern of missingness without further information.

The registry record does not define an analysis population. It does not state whether the comparison used an intention-to-treat, per-protocol, or other approach, nor does it describe exclusions or deviations from assigned treatment. Although the design is recorded as randomized, the analysis population and handling of assignment are not specified in the posted analysis information. The record also does not report whether participants were analyzed according to randomized assignment or actual intervention receipt.

These omissions affect how confidently the unadjusted comparison can be interpreted. They do not prove that data were missing, that a particular analysis population was used, or that multiplicity was mishandled. They mean those methods cannot be assessed from the registry record. The posted result can be described as an unadjusted arm comparison of the stated counts, while conclusions about missingness, analysis populations, and the prespecified statistical plan remain unresolved.

14. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

Most frequent serious adverse event terms

Serious events with the highest arm-level rate: psychiatric hospitalization (29.6%).

The registry safety entries list serious adverse events at risk and affected for each arm. For Smartphone App, the record lists 68 at risk and 17 affected. For Usual Care, it lists 71 at risk and 21 affected. These are the registry’s serious-adverse-event figures and should be read separately from the primary service-utilization counts and denominators. The record does not explain the safety denominators or define how the affected counts were tabulated.

Psychiatric Hospitalization is listed as the top serious-event term, with 29.6%. The registry record does not give a count for that term or explain the percentage’s denominator in the information available. It should not be treated as an arm-specific percentage or converted into a count. The listed death count is 0 in each arm. Other adverse events are recorded as 0 at risk and 0 affected in both arms; those entries do not by themselves establish that no other events occurred, since the record does not explain those fields.

These entries describe what the registry reports, not a comparative safety analysis. No adjusted safety comparison, event-level detail, or interpretation of causality is reported. The figures should not be used to infer that the intervention caused or prevented an event.

15. What the Result Does, and Does Not, Mean

The primary result is an unadjusted comparison of participants with a reported psychiatric service-utilization outcome during 12 months. The posted counts are 11.0 of 59.0 for Smartphone App and 15.0 of 63.0 for Usual Care. The estimated risk difference is -0.052, with a confidence interval from -0.196 to 0.093, and the unadjusted p-value is 0.486. The interval includes zero, leaving uncertainty about both the direction and the size of the contrast.

This result does not establish that the app and usual care are equivalent. It also does not demonstrate that there is no difference. A non-significant comparison can arise when the estimate is uncertain; the interval is important because it shows a range of contrasts compatible with the posted data under the method’s assumptions. The registry record does not report an equivalence or non-inferiority framework, and no such conclusion should be inferred.

The endpoint is not a direct measurement of medication adherence, despite adherence support being the project aim. It records participants’ utilization of psychiatric emergency services or hospitalization, rather than app engagement, medication-taking behavior, or the total number of service events. The record does not provide a posted secondary adherence outcome. Accordingly, the primary result cannot answer every question about adherence or about other clinical outcomes.

The record also does not report the original statistical plan, missing-data methods, analysis population, or details that reconcile enrollment with the primary outcome denominators. Those gaps limit assessment of how the result was produced and how broadly it applies. The appropriate reading is narrow: the registry reports these participant counts and an uncertain unadjusted comparison for the specified service-utilization measure. It does not establish a causal explanation for the observed counts or support claims about outcomes that were not posted.

16. Why This Trial Matters Statistically

This trial illustrates why the endpoint must be distinguished from the intervention’s stated aim. The project aimed to use an AI smartphone app to support medication adherence, but its posted primary outcome is participant utilization of psychiatric emergency services or hospitalization. That outcome may be important to the trial’s service-use question, but it is not itself a direct measure of adherence. Statistical interpretation begins by identifying exactly what was measured, rather than allowing the intervention’s purpose to stand in for the endpoint.

The result also illustrates the importance of effect estimates and uncertainty together. The unadjusted risk difference is -0.052, but its interval ranges from -0.196 to 0.093 and includes zero. The point estimate alone would give an incomplete impression. The interval makes clear that the result is compatible with contrasts in either direction, and the unadjusted p-value of 0.486 does not establish equivalence or settle the size of any underlying difference.

Finally, the record demonstrates how reporting details shape the strength of interpretation. Randomization is recorded, but the registry does not provide the analysis population, missing-data handling, original sample-size assumptions, or a statistical analysis plan. It also reports primary-outcome denominators that are not reconciled with the stated enrollment. These omissions do not invalidate the posted result, but they constrain what can be assessed. Clear reporting of endpoint definitions, participant flow, and analysis decisions is essential for readers to distinguish a concise registry comparison from a complete account of a clinical trial’s statistical evidence.

17. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

Several limitations are specific to the information recorded for this trial. The primary endpoint description combines psychiatric ER visits or psychiatric hospitalization, but the registry record does not specify how the outcome classified participants with one or both types of service use, or how multiple events were handled. The reported measure counts participants, so it does not describe the total number of visits or hospitalizations.

The stated enrollment is 139, whereas the primary-outcome denominators are 59.0 for Smartphone App and 63.0 for Usual Care. The registry record does not explain the relationship between these values. It does not report a participant-flow account, the number with missing outcome data, or the analysis population. This prevents a clear assessment of which participants contributed to the endpoint comparison and how the denominators arose.

The record does not report details of sequence generation, allocation concealment, intervention uptake, app use, or adherence support received. Masking is listed as none, but the record does not elaborate on which parties knew assignment. The description of usual care is limited to care provided at a CSC clinic. These omissions limit assessment of implementation and of the similarity of care aside from the assigned intervention.

The registry record lists no analyses and does not report the original statistical plan, sample-size justification, missing-data approach, or multiplicity strategy. The comparison therefore should be treated as the posted unadjusted summary, not assumed to be the full prespecified analysis. These are limits on interpretation of the registry record; they do not establish that any particular methodological problem occurred.

18. Related Tutorials

19. Related Statistical Calculators

20. Sources

• ClinicalTrials.gov: Artificial Intelligence to Measure Adherence to Oral Medication — NCT04046497.
• Registry record last updated 2026-10-13; status last verified 2026-08.
• Results first posted 2026-10-13.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

21. Record Summary

The registry record describes NCT04046497, Artificial Intelligence to Measure Adherence to Oral Medication, as a completed randomized, parallel-group phase 4 trial sponsored by Northwell Health. It lists First Episode Psychosis as the condition and an enrollment of 139. The two arms are an AI smartphone app intended to support medication adherence and usual care provided at a CSC clinic. Masking is listed as none.

The primary endpoint is the Service Utilization Report Form, a participant count over 12 months describing psychiatric ER visits or psychiatric hospitalization. The posted counts are 11.0 of 59.0 for Smartphone App and 15.0 of 63.0 for Usual Care. The unadjusted risk difference is -0.052, with an interval from -0.196 to 0.093; the posted unadjusted p-value is 0.486. No analyses or secondary outcomes are posted in the registry record. The record does not report the original statistical plan, missing-data methods, or analysis population.