Child Health Improvement Through Computer Automation (CHICA) Highlighting Study: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Child Health Improvement Through Computer Automation (CHICA) Highlighting Study (NCT01583101), a randomized na study. The primary outcome is The Number of Prompts That Were Responded to.

2. Clinical Question

3. Trial Design

The registry describes the study as randomized, single-masked, and parallel. It lists an enrollment of 2237 and records the study as completed. The study start is listed as 2012-04 and primary completion as 2012-10. The phase is listed as NA. The sponsor is Indiana University.

The two arms were receiving highlighted prompts and receiving non-highlighted prompts. In the first arm, prompts received by physicians were highlighted; in the other, prompts were not highlighted. The registry describes a decision support system that combines pediatric guidelines, a scannable paper interface, and an interface to electronic medical record systems. The study’s stated motivation was that physicians ignored some prompts and that system changes might increase response rates.

The outcome data list 1076 participants and 672 responses for the highlighted-prompt arm, and 1161 participants and 712 responses for the non-highlighted-prompt arm. The registry labels the primary measure “The Number of Prompts That Were Responded to,” gives its unit as participants, and describes it as whether a prompt was answered or discussed. How the participant-level counts relate to individual prompts, physicians, or encounters is not reported in the registry record. The identity of the person or people masked is also not reported in the registry record.

The registry record does not report the randomization procedure, allocation concealment, the setting or number of clinics, or further eligibility criteria. These details should not be inferred from the randomized, parallel design label.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: While physicians were the ones that received prompts, the prompts are the unit of analysis.

Overall Study

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as The Number of Prompts That Were Responded to, measured over 1 year.

The primary outcome concerns whether a prompt was answered or discussed. It is a binary measure: for the outcome as described, a prompt is classified as answered or not answered. The stated aim was to assess whether changing the visual presentation of prompts was associated with physicians’ response behavior in the CHICA system.

The posted results give a response count and a denominator for each group, along with a response proportion. In the highlighted-prompt group, 672 of 1076 were recorded as responses, with a proportion of 0.625. In the non-highlighted-prompt group, 712 of 1161 were recorded as responses, with a proportion of 0.613. These are the registry’s reported values; they should be interpreted in light of the record’s labeling of the outcome unit as participants while describing the outcome in terms of prompts.

A binary outcome is useful when the question is whether an event occurred, rather than how often or how long it occurred. Here it provides a direct way to compare the recorded response proportions between the assigned prompt-presentation groups. The registry gives a timeframe of 1 year. It does not explain how that timeframe was applied to individual participants or prompts, or how unanswered or unobserved prompts were handled.

7. Results Posted in the Registry

• Receiving Non-highlighted Prompts: 61.3% (712/1161) versus 62.5% (672/1076) on control; risk difference -1.1 percentage points (95% CI -5.2 to 2.9), risk ratio 0.98 (0.92–1.05); unadjusted two-proportion z-test p = 0.583.

8. Statistical Interpretation

9. Statistical Methods Explained

The outcome is binary as described: a prompt was answered or discussed, or it was not. The posted arm results give the number recorded as responses and the group denominator. A response proportion expresses the observed frequency of the outcome within each group. The reported proportions are 0.625 for highlighted prompts and 0.613 for non-highlighted prompts.

What does the risk difference describe?

The risk difference subtracts one group’s response proportion from the other’s. In the posted comparison, the non-highlighted group is compared with the highlighted group, and the estimate is -0.011. A negative estimate means the recorded response proportion was lower in the non-highlighted group under this comparison direction. The risk difference is an absolute contrast in proportions, rather than a ratio. The registry does not report an adjusted risk difference.

What does the confidence interval add?

General statistical explanation: a confidence interval describes the range of estimates compatible with the observed data under the method’s assumptions, while also reflecting sampling uncertainty. It is not a guarantee that the true value lies within the interval, and it does not include every source of uncertainty, such as measurement or design limitations. The reported interval for the risk difference is -0.051 to 0.029. Because it includes zero, the data are compatible with no difference as well as with differences in either direction under this interval method.

How should the p-value be read?

The unadjusted p-value is 0.583. General statistical explanation: a p-value measures how unusual data at least as incompatible with a specified null model would be if that model and the test assumptions were correct. It is not the probability that the null model is true, nor does it measure the size or importance of an effect. The registry does not report the test procedure or a prespecified decision threshold. The label “unadjusted” indicates that this posted p-value is not adjusted for multiplicity; the record does not explain whether other analyses were conducted.

Why are relative measures also shown?

The relative risk compares the response proportions as a ratio, and the odds ratio compares the odds of response. Their reported estimates are 0.982 and 0.953, respectively. The corresponding intervals are 0.920 to 1.048 for the relative risk and 0.804 to 1.131 for the odds ratio. General statistical explanation: a relative risk of 1 and an odds ratio of 1 represent no relative difference. Both reported intervals include that reference value. Relative measures can describe an association on a different scale than the risk difference, but neither removes uncertainty or establishes causation beyond what the design and analysis support.

What analysis details are absent?

The registry record reports no analyses in its analysis count and does not describe a statistical analysis plan. It does not report the test used, covariate adjustment, handling of repeated prompts or clustering, missing-data procedures, or an analysis population. The randomization label supports describing the study as randomized, but the record does not provide enough detail to reconstruct the analysis or assess all its assumptions. These missing details matter when judging how precisely the posted comparison represents the effect of prompt highlighting.

General statistical explanation: if observations are correlated—for example, when multiple observations come from the same unit—methods that treat every observation as independent can give misleading uncertainty estimates. Whether such dependence is relevant and how it was handled are not reported in the registry record. The same restraint applies to possible over-dispersion: it is a general concern when variability exceeds what a simple model assumes, but the posted record does not describe a model or diagnostic assessment from which to determine whether it occurred.

10. Confidence Intervals and Precision

The risk-difference estimate is -0.011, with a 95% confidence interval from -0.051 to 0.029. The interval spans zero, so the posted data do not pin down the direction of the difference. Values below zero and above zero both remain compatible with the interval. The interval is the clearest single summary of the uncertainty around the absolute comparison; the point estimate alone would hide that uncertainty.

The relative risk is 0.982, with an interval from 0.920 to 1.048. The odds ratio is 0.953, with an interval from 0.804 to 1.131. These intervals include 1, the no-relative-difference reference for each measure. The different scales should not be treated as interchangeable, but they lead to a similar broad reading: the estimates are uncertain and do not isolate a difference from the corresponding null value.

General statistical explanation: interval width reflects statistical precision under the analysis method, while the interval’s location indicates the range and direction of effects compatible with the data. Precision does not by itself address bias, data quality, or whether the outcome represents the question of interest. The registry does not report enough analysis detail to evaluate every assumption behind these intervals, including how the analysis treated the outcome’s unit and any dependence among observations.

The confidence intervals do not establish equivalence or rule out effects that might matter in a particular context. The registry record reports no threshold for a clinically or operationally important difference. The conclusions should therefore remain descriptive: the posted estimates are close to the no-difference reference, but the uncertainty is broad enough to include differences in either direction.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations based on the observed effect, not the original study’s design calculation. They are reported as participants per arm, not as total event counts. The planning assumptions are a two-sided significance level of 0.05, equal allocation between two arms, and a normal approximation. The effect to detect is set to the stated fraction of the observed effect. The original design assumptions are not reported in the registry record.

The 80% power column gives the number of participants per arm that the planning calculation estimates would be needed to have an 80% chance of detecting the specified effect under those assumptions, if that effect were the true one. The 90% power column gives the corresponding planning number for a 90% chance. These are hypothetical design calculations, not observed power for the completed trial.

Rows for 100%, 75%, and 50% of the observed effect show how the planning illustration changes when the target effect is reduced. Smaller effects are harder to distinguish from random variation, so the calculations require more participants per arm to detect them at the stated power. The figures depend on the assumptions and effect size used; they are not a recommendation or a guarantee that a future study would achieve its intended result.

The displayed curve should be read in the same planning context: it relates the assumed effect to the planned sample size. It does not show a new outcome from this trial or change the uncertainty in the observed comparison. The registry does not report the study’s own original sample-size justification, so the planning illustration should not be treated as evidence about why the trial enrolled its recorded population.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.625; treatment proportion p₁ = 0.613; average p̄ = 0.619.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.6868 = 1.3461.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.6868 = 0.5780.
4. n = (1.3461 + 0.5780)² / (-0.0113)² = 29,144.9, rounded up to 29,145 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.62453531598513                          # control proportion
p1 <- 0.613264427217916                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(1583101)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 29,145 (80% power), 39,017 (90% power)
 75% of observed effect: n per arm = 51,741 (80% power), 69,265 (90% power)
 50% of observed effect: n per arm = 116,249 (80% power), 155,623 (90% power)
power.prop.test: 29144.9 
Simulated power at n = 29,145 per arm: 0.805 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Multiplicity, Missing Data and Analysis Populations

The registry lists one primary outcome and no posted secondary outcomes. It reports an unadjusted p-value of 0.583 for the primary comparison. No multiplicity adjustment, testing hierarchy, or prespecified approach to multiple comparisons is reported in the registry record. Consequently, the record does not establish whether this was the only planned analysis or how the investigators intended to interpret results if several outcomes, subgroups, or analyses were examined.

The registry record does not report how missing outcome data were defined or handled. It does not state whether every randomized observation contributed to the displayed denominators, whether any responses were excluded, or how unavailable or incomplete prompt records were treated. The arm counts are reported, but the record does not provide a participant-flow account or an explicit analysis population such as intention-to-treat or per-protocol.

The outcome’s labeling also limits interpretation: its title refers to the number of prompts responded to, its unit is participants, and its description defines a binary answered/not-answered outcome. The record does not explain whether repeated prompts were aggregated, whether a participant could contribute multiple observations, or whether any clustering was accounted for. The appropriate analysis can depend on these details, but they are not reported in the registry record.

These omissions do not show that a particular procedure was or was not used; they mean it cannot be established from the registry record. The posted comparison can be described as an unadjusted result, but the record does not permit a full assessment of multiplicity, missingness, or analysis-population choices.

14. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry lists zero serious adverse events and zero other adverse events for the highlighted-prompt group, with 1076 at risk in each category. For the non-highlighted-prompt group, it likewise lists zero serious adverse events and zero other adverse events, with 1161 at risk in each category. Deaths are not reported in the registry record.

These entries describe what the record reports; they should not be read as proof that no adverse event occurred. The registry does not provide event definitions, monitoring methods, reporting completeness, or further safety context. The intervention is described as changing prompt highlighting, but no inference about safety beyond the posted adverse-event entries is warranted.

General statistical explanation: a zero count in a registry summary means no events were recorded in that summary, not that the underlying event risk is known to be zero. Interpretation depends on how events were sought, defined, and recorded. Those details are not reported in the registry record.

15. What the Result Does, and Does Not, Mean

The posted results show response proportions of 0.625 for highlighted prompts and 0.613 for non-highlighted prompts. The estimated risk difference for non-highlighted relative to highlighted prompts is -0.011, and its 95% confidence interval, -0.051 to 0.029, includes zero. The unadjusted p-value is 0.583. Taken together, these statistics do not establish a difference in the recorded response outcome between the groups.

This is not the same as proving that highlighting has no effect. The estimate has uncertainty, and the interval allows differences in either direction. The registry does not report an equivalence margin or a minimum important difference, so the result cannot establish that the approaches are equivalent or that any possible difference is unimportant.

The endpoint concerns whether a prompt was answered or discussed. It does not directly measure whether the advice was followed, whether a clinical decision was appropriate, or whether patient health changed. The registry record reports no secondary outcomes that would support those conclusions. It also does not provide enough information about the outcome unit, repeated prompts, missing data, or analysis population to resolve those aspects of interpretation.

The result applies to the comparison described in this completed randomized study of highlighted and non-highlighted prompts in the CHICA system. It should not be generalized to other systems, settings, or outcomes based on this record alone. The evidence is best read as an uncertain comparison of recorded prompt response behavior, rather than as a definitive claim about clinical effectiveness.

16. Why This Trial Matters Statistically

This study addresses a statistical question about behavior in a clinical decision support system: whether a change in prompt presentation is associated with a change in recorded responses. Such process outcomes can be measured as binary events, and comparing response proportions makes the outcome contrast easy to state. The analysis also illustrates why an effect estimate and its uncertainty are more informative together than either a point estimate or a p-value alone.

The estimated risk difference is -0.011, while its interval ranges from -0.051 to 0.029. The interval crosses the no-difference value, and the unadjusted p-value is 0.583. The statistical lesson is not simply that a threshold was or was not crossed: the data leave uncertainty about both direction and magnitude. The relative risk and odds ratio intervals likewise include their no-association reference values.

The record also highlights the importance of defining the analysis unit. The primary outcome is described in terms of prompts but is labeled with participants as its unit. When repeated observations or clustering might be present, clear reporting of how observations are organized and analyzed is essential for judging uncertainty. The registry does not report those details, so they remain an important limit on statistical interpretation.

Finally, planning illustrations based on an observed effect are distinct from the original design justification and from observed power. The registry makes no original sample-size rationale available. Keeping those concepts separate helps prevent retrospective planning calculations from being mistaken for evidence that the completed study was adequately or inadequately designed.

17. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record gives limited information about trial conduct and analysis. It identifies the study as randomized, single-masked, and parallel, but does not report the randomization method, allocation concealment, who was masked, or the setting and number of participating sites. It also does not give eligibility criteria beyond the broad description of physicians receiving prompts through the CHICA system.

The outcome is described as whether a prompt was answered or discussed, yet the record labels the measure as a number of prompts and gives its unit as participants. It does not clarify whether each participant contributed one outcome, how multiple prompts were handled, or whether the analysis accounted for repeated observations or clustering. These details affect the interpretation of both the event counts and the uncertainty estimates.

The record does not report missing-data procedures, exclusions, analysis populations, covariate adjustment, or the statistical test used. It provides no original sample-size justification or statistical analysis plan. No secondary outcomes are posted, and the record does not explain whether other analyses were planned or conducted. These gaps limit assessment of selective analysis, multiplicity, and robustness.

The study’s outcome is a process measure of prompt response, not a direct measure of clinical decisions or patient outcomes. The posted record also gives little detail about the context in which prompts appeared or whether other system changes differed between groups. Conclusions should therefore remain bounded to the recorded comparison and not be extended to unreported outcomes or settings.

18. Related Tutorials

19. Related Statistical Calculators

20. Sources

• ClinicalTrials.gov: Child Health Improvement Through Computer Automation (CHICA) Highlighting Study — NCT01583101.
• Registry record last updated 2017-05-15; status last verified 2017-04.
• Results first posted 2017-03-23.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

21. Record Summary

The registry record describes a completed randomized, single-masked, parallel study comparing highlighted with non-highlighted prompts in the CHICA system. The primary outcome concerns whether prompts were answered or discussed, with a listed timeframe of 1 year.

For highlighted prompts, the record reports 672 responses among 1076 participants and a proportion of 0.625. For non-highlighted prompts, it reports 712 responses among 1161 participants and a proportion of 0.613. The estimated risk difference for non-highlighted relative to highlighted prompts is -0.011, with a 95% confidence interval from -0.051 to 0.029 and an unadjusted p-value of 0.583.

The interval includes zero, so the posted results do not establish a difference in response proportions. The record does not report a statistical analysis plan, missing-data methods, analysis population, or details resolving the outcome’s participant-versus-prompt unit labeling.