VA Integrated Medication Manager: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for VA Integrated Medication Manager (NCT01787175), a randomized na study. The primary outcome is Amount of Time to Complete Assessment and Plan.

2. Clinical Question

This is a question about task completion in a simulated EHR setting. The trial record describes randomized assignment to one of the two systems, not a comparison of patient outcomes. Its stated aim was to advance healthcare informatics and improve medication management through development of the Integrated Medication Manager. The primary measure captures one aspect of the simulated work: the time providers used to review a case and write an assessment and plan. The registry also lists identification of planned monitoring and follow-up encounters in the assessment and plan as a secondary outcome. The record does not describe the results for that secondary outcome in the information.

3. Trial Design

The registry classifies this study as randomized, with a parallel-group model and no masking. It records 58 enrolled providers and a completed study. The intervention was the Integrated Medication Manager; the comparator was the standard EHR, identified in the arm descriptions as the VA’s CPRS. Thirty providers were assigned to the Integrated Medication Manager and 28 to the standard EHR. Participants were experienced providers taking part in EHR simulations, and assignment determined which system they used during the simulation.

The primary endpoint was the amount of time to complete an assessment and plan, measured in minutes. Participants had a maximum of 10 minutes to review the patient case and write the assessment and plan. The registry describes the endpoint as a mean with standard deviation. The posted arm summaries report a mean of 8.5 minutes with standard deviation 1.9 for the Integrated Medication Manager and a mean of 8.7 minutes with standard deviation 1.6 for the standard EHR.

The listed condition is Electronic Health Records. The study is sponsored by the University of Utah. The registry lists phase as NA. It gives a start of 2007-10 and primary completion of 2011-03. Those registry dates place the work in the context of its record, but do not establish details of recruitment, simulation procedures, or analysis choices beyond what is explicitly described. The protocol’s original sample-size justification and its original analysis plan are not reported in the registry record. The registry lists 3 outcome measures and 3 analyses, but does not fully detail the outcome-measure and analysis mapping in the record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Analysis population for the primary outcome, as defined in the registry: 58 providers were enrolled

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: Half of the providers were assigned to use the new Integrated Medication Manager (intervention) during the simulation. The other half were assigned to use VA's CPRS (standard EHR). Providers were randomly assigned which system to use.

6. Primary Endpoint

The registry lists the primary outcome as Amount of Time to Complete Assessment and Plan, measured over 10 minutes.

The primary endpoint is the amount of time needed to complete an assessment and plan. It is a continuous measure reported in minutes, summarized for each arm by a mean and standard deviation. In this simulation, each provider had up to 10 minutes to review a patient case and write the assessment and plan. The endpoint therefore represents the duration of a defined EHR task under the study’s simulation conditions.

Time is used because the study concerned the design of an electronic medical record workflow. Comparing average task duration can describe whether task completion times differed between providers assigned to the new medication-management system and providers assigned to the standard EHR. The mean gives a central summary of observed completion times; the standard deviation describes how dispersed those times were among participants in each arm. Neither statistic, by itself, describes the quality or completeness of the assessment and plan. The registry lists identification of planned monitoring and follow-up encounters as a secondary outcome, which addresses a different aspect of the documented plan.

The endpoint should be interpreted within the task and setting actually described: experienced providers reviewing a simulated patient case, with a 10-minute maximum. The registry does not report additional details about the case, the timing procedure, or how incomplete tasks were handled. It also does not establish that task duration in simulation corresponds to time spent in routine clinical practice.

7. Results Posted in the Registry

• Integrated Medication Manager: mean 8.50 versus 8.70 on control; difference -0.20 (95% CI -1.10 to 0.70), standardized effect d = -0.11; z-test p = 0.664.

8. Statistical Interpretation

9. Statistical Methods Explained

What does the mean summarize?

For a continuous outcome such as task duration, the mean is the arithmetic average of participants’ measured times within an arm. The registry reports mean times of 8.5 minutes for the Integrated Medication Manager and 8.7 minutes for the standard EHR. The standard deviations are 1.9 and 1.6 minutes, respectively. A standard deviation describes the spread of individual observations around an arm’s mean. It is not a confidence interval for that mean, and it does not indicate how precisely the difference between arms has been estimated.

Comparing means is a natural summary when the question concerns average time. A mean difference expresses the contrast on the original measurement scale, so its units are minutes. The simple unadjusted re-calculation reports -0.2 minutes for the Integrated Medication Manager relative to the standard EHR. A negative value in this comparison means the observed intervention-arm mean was lower than the comparator-arm mean. The sign describes the direction of the numerical contrast; it is not, by itself, evidence that the difference is reliable or important.

What does the unadjusted comparison tell us?

The re-calculation uses the posted group summaries to estimate a difference without adjustment for other variables. It reports a 95% confidence interval from -1.102 to 0.702 minutes and p = 0.664. The confidence interval describes a range of values compatible with the data and the calculation’s assumptions. Because it includes zero, the summary-based comparison does not clearly distinguish a difference from no difference. The p-value measures how unusual a result at least as incompatible with a zero-difference model would be if that model and the test assumptions held. It is not the probability that the systems are equivalent, nor is it the probability that the null hypothesis is true.

This re-calculation is not the same as the analysis reported by the registry. It is an unadjusted comparison derived from the arm-level means, standard deviations, and participant counts. It does not reproduce a model that might account for other variables or data structure. The registry’s reported analysis is a mixed-effects linear model, with an estimate of -17.73, a 95% confidence interval from -35.24 to -0.23, and p = 0.047. The registry does not report enough detail to reconstruct that model or explain its difference from the unadjusted calculation.

Why might a mixed-effects model be used?

In general, a mixed-effects model can represent both fixed effects, such as a comparison between assigned systems, and random effects that account for grouping or repeated observations. Such models can be useful when observations are correlated rather than independent. That is a general description of the method, not a claim about the structure or specification used in this study. The registry does not report the random effects, fixed covariates, model diagnostics, or estimation details, so the model’s precise role in this analysis is not reported in the registry record.

When a model estimate differs from a calculation based only on group summaries, potential reasons can include adjustment, different analytic populations, different handling of repeated or grouped data, or a difference in the outcome scale. These are general possibilities, not explanations established for this study. The record does not identify which, if any, apply. The result must therefore retain its reported label and value without assigning an unstated unit or model explanation.

What does statistical significance mean in this comparison?

Statistical significance is a decision convention based on a specified analysis and threshold. It does not measure the size or practical importance of an effect. The unadjusted re-calculation has p = 0.664 and an interval including zero; the registry’s mixed-effects model has p = 0.047 and an interval that does not include zero. Those results lead to different statistical readings because they arise from analyses that do not agree. The registry does not report whether any multiplicity adjustment was made, nor does it state the significance threshold used for its own analysis.

The p-values should not be used to declare the two systems equivalent or to establish the clinical importance of a time difference. Equivalence requires a defined margin and an analysis designed to assess equivalence; neither is reported in the registry record. Interpretation also requires attention to the outcome’s meaning, uncertainty, and study setting. Here, the endpoint is time to complete a simulated assessment and plan, not a direct measure of patient benefit or plan quality.

How should the analyses be kept distinct?

The arm summaries describe the observed means and standard deviations. The unadjusted re-calculation gives a transparent comparison based on those summaries. The mixed-effects model is the analysis reported in the registry. Since the two analyses conflict and the discrepancy flag is true, the sponsor’s own analysis takes precedence as the trial’s reported result. That precedence does not erase the mismatch: readers should be able to see that the reported model estimate and the simple re-calculation differ, and that the record does not explain why.

10. Confidence Intervals and Precision

The unadjusted mean difference is -0.2 minutes, with a 95% confidence interval from -1.102 to 0.702 minutes. A confidence interval is a way to express uncertainty around an estimated difference. In general, a narrower interval indicates greater precision, while a wider interval leaves a broader range of values compatible with the data and the method used. Here, the interval spans values on both sides of zero, so the unadjusted summaries are compatible with a small difference in either direction as well as no difference.

The interval should not be read as a range containing the true effect with certainty. Its interpretation depends on the statistical procedure and its assumptions. It also does not resolve the mismatch with the registry’s mixed-effects model, which reports an estimate of -17.73 and a 95% confidence interval from -35.24 to -0.23. The registry does not report the unit for that model estimate or enough analytic detail to determine why its result differs from the unadjusted comparison.

The arm-level standard deviations, 1.9 minutes and 1.6 minutes, describe variation among provider task times. They are not measures of precision for the mean difference. Precision in the comparison depends on the data and analysis used, including the number of participants and the model assumptions. Although the registry reports 30 providers in one arm and 28 in the other, the discrepancy between the analyses limits a single, unified interpretation of precision. The confidence interval and p-value for each analysis should remain attached to that analysis rather than being combined.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations based on the observed effect, not the trial’s original design calculation. For this continuous outcome, the figures are participants per arm, not total events. The planning assumptions are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The effect to detect is specified as a stated fraction of the observed effect. The registry’s original sample-size assumptions and justification are not reported in the registry record.

The 80% power column describes the planned number of participants per arm for a design with an 80% chance of detecting the specified effect under the planning assumptions, if that effect and those assumptions were correct. The 90% power column gives the corresponding planning target for 90% power. These are design targets, not results from the participants enrolled in the study.

For planning for 100% of the observed effect, the figures are 1,219 participants per arm for 80% power and 1,631 for 90% power. For planning for 75% of the observed effect, they are 2,166 and 2,899 participants per arm, respectively. For planning for 50% of the observed effect, they are 4,873 and 6,523 participants per arm, respectively. The pattern illustrates a general feature of power calculations: smaller effects are harder to distinguish from random variation and therefore require more participants, all else being equal.

This is not observed power. It does not describe the probability that the completed trial detected an effect, and it does not establish whether the trial was adequately designed. The planning estimates use the observed effect as their reference, so they should be interpreted as hypothetical illustrations under the stated assumptions rather than as evidence about the original design.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, Δ is the difference in means to detect and σ is the common standard deviation.

Worked calculation for this trial

1. Observed difference Δ = -0.200; pooled standard deviation σ = 1.762.
2. n = 2 × (1.96 + 0.842)² × 1.762² / -0.200² = 1,218.0, rounded up to 1,219 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
delta <- -0.199999999999999                      # observed difference in means
sigma <- 1.76174629274479                     # pooled standard deviation
n_per_arm <- function(power, frac = 1) {
  ceiling(round(2 * (z_a + qnorm(power))^2 * sigma^2 / (delta * frac)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (t-based, so very slightly larger)
cat("power.t.test:", round(power.t.test(delta = delta, sd = sigma, power = 0.80)$n, 1), "\n")

# Simulation: exact two-sample t-test using simulated means and variances
set.seed(1787175)
n    <- n_per_arm(0.80)
reps <- 20000
m_diff <- rnorm(reps, delta, sigma * sqrt(2 / n))             # difference in sample means
s2     <- sigma^2 * rchisq(reps, 2 * (n - 1)) / (2 * (n - 1))   # pooled sample variance
tstat  <- m_diff / sqrt(s2 * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(tstat) > qt(0.975, 2 * (n - 1))), reps))

Output from running the code

100% of observed effect: n per arm = 1,219 (80% power), 1,631 (90% power)
 75% of observed effect: n per arm = 2,166 (80% power), 2,899 (90% power)
 50% of observed effect: n per arm = 4,873 (80% power), 6,523 (90% power)
power.t.test: 1219 
Simulated power at n = 1,219 per arm: 0.798 (20000 replicates)

The simulation draws many sets of group means and variances at the planned sample size and counts how often the two-sample t-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Analyses Reported by the Sponsor

The registry reports a mixed-effects linear model for the difference in time to complete the assessment and plan. Its estimate is -17.73, with a 95% confidence interval from -35.24 to -0.23 and p = 0.047. The record does not state the unit attached to this estimate in the reported analysis. The simple unadjusted re-calculation from the posted arm summaries instead gives a mean difference of -0.2 minutes, with a 95% confidence interval from -1.102 to 0.702 minutes and p = 0.664.

These are not matching estimates or inferences. The registry does not explain the discrepancy, and the available details are insufficient to determine whether it reflects adjustment, data handling, outcome scaling, or another analytic feature. Those are possible general sources of disagreement, not established explanations for this case. The model’s exact specification and analysis population are not reported in the registry record.

Because the discrepancy flag is true, the sponsor’s own analysis takes precedence as the reported trial analysis. That means the mixed-effects result should be identified as the sponsor-reported finding, not replaced by the unadjusted re-calculation. The re-calculation remains useful as a check of the posted group summaries, but it should be presented separately and not treated as a correction to the model. The registry’s failure to explain the difference limits interpretation of the magnitude and precision of the reported model estimate.

14. Multiplicity, Missing Data and Analysis Populations

The registry lists 3 outcome measures and 3 analyses. It identifies the primary outcome as time to complete an assessment and plan, and it lists identification of planned monitoring and follow-up encounters in the assessment and plan as a secondary outcome. The other outcome measures are not described in the record details. The registry does not provide a complete map connecting all listed outcomes to all analyses.

When a study tests multiple outcomes or conducts multiple analyses, the chance of obtaining at least one apparently notable result can increase. Statistical methods may account for this through a multiplicity plan, but the registry does not report whether an adjustment was made, which outcomes were included in any such plan, or what analysis hierarchy was prespecified. The statistical plan and the original handling of multiplicity are not reported in the registry record. The reported p-values should therefore be understood as the values attached to their respective analyses; no claim about multiplicity-adjusted inference is supported by the record.

The registry does not report whether all randomized providers contributed to the primary analysis, how missing measurements were handled, or whether any participants were excluded after assignment. It also does not describe an intention-to-treat or other analysis population. Those details matter because exclusion or missingness can affect how well an analysis preserves the randomized comparison. No assumption about missing data, imputation, or analysis population can be made from the information recorded.

The record describes random assignment to systems but does not report the sequence-generation method, allocation concealment, or whether assignment was stratified. Masking is listed as none. The registry also does not report the covariates or random-effects structure of the mixed-effects model. These omissions do not prove that such procedures were absent; they mean the relevant details are not reported in the registry record. This leaves uncertainty about the prespecified analytical framework and about the degree to which the reported model can be independently reconstructed.

15. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry reports zero serious adverse events among 30 providers assigned to the Integrated Medication Manager and zero among 28 assigned to the standard EHR. It also reports zero other adverse events in each arm. These entries describe event counts in the study record; they do not establish that no events occurred outside the scope of the registry’s reporting or observation procedures.

The registry does not list serious adverse event terms. Death counts are not reported in the registry record. The record does not describe how adverse events were defined, how they were collected, or the period over which they were monitored. Since this study involved providers completing EHR simulations, the event figures should be read in that study context, not as evidence about patient safety outcomes. The zero counts are a description of the posted event data and do not provide a broader safety assessment.

16. What the Result Does, and Does Not, Mean

The results describe a randomized comparison of two EHR systems during simulated provider work. The primary outcome was time to complete an assessment and plan, with a 10-minute maximum for reviewing the case and writing the plan. The observed arm means were 8.5 minutes for the Integrated Medication Manager and 8.7 minutes for the standard EHR. A simple unadjusted re-calculation gives a mean difference of -0.2 minutes, but its confidence interval includes zero and its p-value is 0.664.

The registry’s mixed-effects model reports a different result: -17.73, with a 95% confidence interval from -35.24 to -0.23 and p = 0.047. The registry does not state the unit for this model estimate or explain its disagreement with the unadjusted re-calculation. The sponsor’s own analysis takes precedence, while the discrepancy remains an important limitation on interpretation. Neither the difference in model results nor the re-calculation can be resolved by assuming unreported methods or units.

The outcome addresses task duration, not the correctness or completeness of the assessment and plan, medication safety, or patient health. The registry lists identification of planned monitoring and follow-up encounters as a secondary outcome, but its result is not reported in the registry record details. No conclusion about that outcome can be drawn from the time results.

The study’s setting also matters. Providers completed EHR simulations, and the registry does not establish whether performance in those simulations represents routine clinical workflows. The findings therefore describe the comparison as recorded for this simulation study. They do not establish how the systems would perform in other settings or for other tasks, and they do not demonstrate clinical benefit or harm.

17. Why This Trial Matters Statistically

This study illustrates why statistical interpretation should keep observed summaries, re-calculated comparisons, and the trial’s reported model analysis distinct. The group means are close: 8.5 minutes with the Integrated Medication Manager and 8.7 minutes with the standard EHR. A direct unadjusted comparison of those summaries yields an estimate and uncertainty interval that do not indicate a clear difference. The registry’s mixed-effects model, however, reports a different estimate and a confidence interval that excludes zero. Without the model details or an explanation of the mismatch, a reader cannot reconcile the two results.

That disagreement is statistically important even though the primary endpoint is straightforward. It shows that a p-value cannot be interpreted apart from the analysis that produced it, and that the reported estimate needs its scale, model, and analytic population to be clear. Reporting a model result alongside transparent arm summaries can help readers assess a trial, but unexplained inconsistency limits reproducibility and confidence in interpretation.

The study also highlights a distinction between statistical evidence and the broader meaning of a result. Task time is one measurable feature of an EHR workflow; it does not alone establish the quality of clinical documentation or the effects of a system on patient care. Random assignment supports a comparative design in the simulated setting, but the registry does not report enough detail about simulation procedures or generalizability to extend the result beyond that setting.

Finally, planning illustrations based on the observed effect should not be mistaken for the original sample-size rationale or for evidence of observed power. The registry’s original design assumptions are not reported. Keeping planning calculations separate from trial results helps readers avoid treating hypothetical design targets as study findings.

18. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry gives limited detail about the simulation itself. It states that experienced providers reviewed a patient case and wrote an assessment and plan within a maximum of 10 minutes, but does not report further information about case construction, task procedures, timing, or how completion was assessed. The extent to which the task represents routine EHR use is therefore not established by the record.

The most direct interpretive limitation is the discrepancy between the sponsor-reported mixed-effects model and the unadjusted re-calculation from the posted group summaries. The model estimate is -17.73, while the unadjusted estimate is -0.2 minutes. The registry does not state the unit of the model estimate or explain the difference. Model specification, covariates, the analytic population, and data handling are not reported in the registry record, so the model cannot be reconstructed from the recorded details.

The record does not report the original sample-size justification, missing-data approach, exclusions after randomization, or analysis population. It also does not report a multiplicity plan despite listing 3 outcome measures and 3 analyses. The full results for the listed secondary outcome are not included in the record details. These gaps limit assessment of prespecification and completeness of the reported findings.

The registry lists no masking. It does not provide further information about allocation concealment or the randomization procedure. The study concerns provider task time in a simulation and does not report patient outcomes in the primary endpoint. Its findings should therefore be understood within that narrow setting and outcome rather than generalized to clinical effectiveness or safety.

19. Related Tutorials

20. Related Statistical Calculators

21. Sources

• ClinicalTrials.gov: VA Integrated Medication Manager — NCT01787175.
• Registry record last updated 2016-06-23; status last verified 2016-05.
• Results first posted 2016-06-23.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

22. Record Summary

NCT01787175, titled VA Integrated Medication Manager, is a completed randomized parallel-group study sponsored by the University of Utah. The registry records 58 experienced providers assigned to use either the Integrated Medication Manager or the standard EHR in simulations; masking is listed as none. The primary outcome was mean time in minutes to complete an assessment and plan, with a maximum of 10 minutes for the task.

The arm means were 8.5 minutes for 30 providers assigned to the Integrated Medication Manager and 8.7 minutes for 28 assigned to the standard EHR. A simple unadjusted re-calculation gives a difference of -0.2 minutes, while the registry’s mixed-effects model reports -17.73, with p = 0.047. The registry does not explain this discrepancy or report the model estimate’s unit. The sponsor’s own analysis takes precedence, but the mismatch limits interpretation. The record also lists identification of planned monitoring and follow-up encounters as a secondary outcome; its result is not reported in the registry record.