This page provides an independent statistical analysis and educational interpretation of publicly reported results. ClinicalTrials.gov provides the official trial registry record. Numerical trial results on this page are restricted to the results posted in the ClinicalTrials.gov record; calculations added by Clinical Biostats are labelled as such.
1. Trial at a Glance
This page covers the registry results for Using Behavioral Economics to Reduce Low-Value Care (NCT04104256), a randomized na study. The primary outcome is Pre-Operative Testing Change.
| Feature | Registry value |
|---|---|
| ClinicalTrials.gov identifier | NCT04104256 |
| Phase | NA |
| Condition | Pre-Operative Testing for Cataract Surgery |
| Allocation | Randomized |
| Design model | Parallel |
| Masking | Double |
| Enrollment | 1045 |
| Arms in the primary outcome | n/a |
| Lead sponsor | University of California, Los Angeles |
| Results posted | Yes |
2. Clinical Question
Population
The study population includes UCLA Health physicians who completed a pre-operative visit for at least one patient undergoing cataract surgery at UCLA in the 12 months before and the 12 months after the study start date, as well as all patients seen during such visits.
Intervention
Physicians were assigned to one of three behavioral alerts. The alerts highlighted potential safety or patient harms, financial harms, or psychological harms associated with pre-operative testing. The descriptions indicate that an alert was encountered when a physician attempted to order a pre-operative test for a patient having a pre-operative visit.
Comparator
The control group received usual care. The registry describes the encounter as continuing as usual, without one of the three behavioral alerts.
Primary question
The trial asks whether adding these alerts to the electronic health record changes the percentage of patients undergoing pre-operative testing, including laboratory tests, electrocardiograms, and chest X-rays. The primary outcome is defined as a change from the pre-intervention baseline to the post-intervention period. The registry record does not report the observed change or a comparison of the alert groups with control.
3. Trial Design
This was a pragmatic, randomized, double-masked, parallel-group trial listed as completed. The registry lists enrollment of 1045 and gives the study dates as 2021-06-24 to 2023-04-11 for the start and primary completion. The phase is recorded as not applicable. The sponsor is the University of California, Los Angeles.
The intervention was implemented through UCLA Health’s electronic health record. Three alert strategies were assigned to physicians: one emphasizing potential safety harms, one emphasizing financial harms to patients, and one emphasizing potential psychological harms. A usual-care control group was also included. The arm descriptions place assignment at the physician level: an alert was intended for a pre-operative encounter when the physician attempted to order a test and had been assigned to the corresponding group. The record describes the design as randomized, but does not report the randomization procedure, allocation ratio, or method used to conceal allocation.
The trial is described as pragmatic and focused on test ordering in clinical practice rather than on an isolated laboratory outcome. Its primary outcome concerns pre-operative tests for cataract surgery, and its stated observation periods are pre-intervention baseline and post-intervention at 12 months. The primary population description covers both eligible physicians and patients seen during their relevant pre-operative visits. The registry record does not explain how those two levels were handled in the outcome analysis.
Although the record marks masking as double, it does not say which groups of people were masked or how masking was maintained for alerts that differed in their messages. It also does not report the protocol details, analysis plan, or participant flow needed to assess implementation and follow-up in greater depth.
4. Participant Flow and Accounting
The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.
Before assignment: Participants were enrolled into the study when they have a pre-operative encounter for cataract surgery AND the physician starts to order a pre-op lab. Eligible providers were assigned an arm of the study based on their 2019 pre-operative visits. Patients were enrolled into each arm based on the providers arm assignment. Healthcare Providers were not enrolled in the study
Overall Study
| Stage | Alert 1 | Alert 2 | Alert 3 | Control |
|---|---|---|---|---|
| STARTED | 269 | 267 | 272 | 237 |
| COMPLETED | 269 | 267 | 272 | 237 |
| NOT COMPLETED | 0 | 0 | 0 | 0 |
Analysis population for the primary outcome, as defined in the registry: The participants in this study are UCLA Health physicians who complete a pre-op visit for at least one patient undergoing cataract surgery at UCLA in the 12 months prior and 12 months after the study start date and all patients who are seen during such visits.
5. Baseline Characteristics
The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.
Population: Providers were not assessed for baseline
| Characteristic | Unit | Alert 1 | Alert 2 | Alert 3 | Control | Total |
|---|---|---|---|---|---|---|
| Participants analysed | 269 | 267 | 272 | 237 | 1045 | |
| Age, Continuous | years | 71.07 (SD 9.58) | 72.00 (SD 10.98) | 72.98 (SD 9.61) | 72.27 (SD 10.24) | 72.08 (SD 9.37) |
| Sex: Female, Male: Female | Participants | 168 | 140 | 161 | 126 | 595 |
| Sex: Female, Male: Male | Participants | 101 | 127 | 111 | 111 | 450 |
| Ethnicity (NIH/OMB): Hispanic or Latino | Participants | 28 | 34 | 38 | 26 | 126 |
| Ethnicity (NIH/OMB): Not Hispanic or Latino | Participants | 214 | 203 | 204 | 178 | 799 |
| Ethnicity (NIH/OMB): Unknown or Not Reported | Participants | 27 | 30 | 30 | 33 | 120 |
| Race/Ethnicity, Customized: EthnoRacial Category – Black or African American | Participants | 20 | 11 | 18 | 18 | 67 |
| Race/Ethnicity, Customized: EthnoRacial Category – White | Participants | 142 | 132 | 148 | 131 | 553 |
| Race/Ethnicity, Customized: EthnoRacial Category – Unknown or Not Reported | Participants | 36 | 40 | 29 | 30 | 135 |
| Race/Ethnicity, Customized: EthnoRacial Category – Asian | Participants | 33 | 45 | 31 | 24 | 133 |
| Race/Ethnicity, Customized: EthnoRacial Category – Hispanic or Latino | Participants | 26 | 32 | 37 | 26 | 121 |
| Race/Ethnicity, Customized: EthnoRacial Category – Middle Eastern or North African | Participants | 7 | 4 | 5 | 5 | 21 |
| Race/Ethnicity, Customized: EthnoRacial Category – Multiple Ethnoracial Categories | Participants | 5 | 3 | 3 | 2 | 13 |
| Race/Ethnicity, Customized: EthnoRacial Category – Native Hawaiian or Other Pacific Islander | Participants | 0 | 0 | 1 | 1 | 2 |
6. Primary Endpoint
The registry lists the primary outcome as Pre-Operative Testing Change, measured over Pre-intervention (Baseline), Post-Intervention (12 months).
| Element | Registry entry |
|---|---|
| Result type | Mean |
| Measure of spread | Standard Deviation |
| Unit | Percentage of patients with orders |
| Endpoint classification | Continuous Endpoint |
The primary endpoint is the change in the percentage of patients undergoing pre-operative testing, with the listed categories including laboratory tests, electrocardiograms, and chest X-rays. The registry describes the measure as a mean with standard deviation and gives its unit as the percentage of patients with orders. It names the observation periods as pre-intervention baseline and post-intervention at 12 months.
Using an ordering measure is relevant to the intervention’s stated purpose: the alerts appear when a physician attempts to place an order, and the trial evaluates whether testing patterns change. An outcome expressed as a percentage can describe the frequency of patients with orders while allowing a comparison across the stated periods. The registry record does not specify the exact numerator and denominator, whether the endpoint is calculated separately for each test category before aggregation, or how repeated visits or multiple orders are handled.
No baseline or post-intervention values, estimated change, uncertainty interval, or statistical test for this endpoint is reported in the registry record. Therefore, the endpoint’s definition can be described, but the size and precision of any observed change cannot be assessed from the record.
7. Why No Re-analysis Is Shown
8. Statistical Interpretation
The key interpretive point is that the registry record defines a primary outcome but does not report its results. It gives no baseline or post-intervention percentages, no estimated change, no comparison between any alert group and control, and no confidence interval or p-value for the primary endpoint. The record also lists zero analyses. Consequently, it does not establish the direction, magnitude, or statistical precision of a primary-outcome difference.
The stated endpoint is a change in the percentage of patients undergoing pre-operative testing, measured at baseline and after 12 months of intervention. Even when both values are available, interpretation depends on how the change is defined, which population contributes to each period, and whether the analysis compares each alert with usual care or evaluates a broader intervention effect. Those details are not reported in the registry record. The three alerts communicate different reasons to reconsider testing, so a pooled estimate, if one were calculated, would answer a different question from separate estimates for each alert. The record does not identify which approach was planned or performed.
The registry describes randomization and a parallel-group design, which are important features for interpreting treatment comparisons. However, the assignment unit is described as the physician, while the outcome population includes patients seen by those physicians. General statistical consideration: when patients are grouped under physicians, their outcomes may be correlated rather than independent. An analysis that accounts for physician-level assignment and clustering can affect both the estimated uncertainty and the interpretation of the comparison. The registry record does not report whether or how clustering was handled.
The enrollment figure is not a substitute for an analyzed denominator. The record does not provide group-specific primary-outcome denominators, losses to follow-up, exclusions, or missing-outcome counts. Nor does it explain how baseline and post-intervention observations were linked or summarized. These omissions prevent assessment of whether the analyzed population matches the population described in the eligibility statement.
The adverse-event listing contains counts for serious events, other events, and deaths by group, but those counts do not supply evidence about the primary endpoint. They also cannot establish whether an event was caused by an alert or by testing. As the registry provides no primary outcome estimates and no statistical analysis details, conclusions about the effect of the alerts on test ordering should remain limited to the trial’s stated question and design rather than inferred from the adverse-event information.
9. Statistical Methods Explained
What does a percentage endpoint represent?
General statistical explanation: a percentage describes the share of observations meeting a specified definition among the observations counted in its denominator. For this trial, the registry labels the primary outcome as the percentage of patients with orders and describes the change in the percentage undergoing pre-operative testing. To interpret such a measure, readers need to know exactly what counts as a patient with an order, which patients are eligible for the denominator, and whether the measure concerns any test or a defined category. Those operational details are not reported in the registry record. The percentage values and the method used to compare baseline with follow-up are also not reported.
The registry classifies the endpoint as continuous and specifies a mean and standard deviation. General statistical explanation: continuous outcomes are often summarized by a mean, which describes a central value, and a standard deviation, which describes variability among observations. A percentage calculated for a group can itself be treated as a continuous summary in some analyses, but the record does not clarify the level at which the mean and standard deviation were calculated. It is therefore not possible to determine whether these summaries refer to patient-level data, physician-level summaries, or another unit.
Why does assignment level matter?
The arm descriptions say physicians were assigned to the alert or control group, while patients seen during relevant visits are part of the study population. General statistical explanation: outcomes from patients treated by the same physician can share features, so they may be more alike than outcomes from patients treated by different physicians. This dependence is called clustering. If an analysis treats all patient observations as independent when they are clustered by physician, its uncertainty estimates can be too small. Analyses may account for clustering through methods appropriate to the design, but the registry record does not report whether such a method was used. It also does not give the number of physicians by group or explain how physician and patient observations contributed to the primary analysis.
Random assignment can help make groups comparable on average, but it does not by itself specify the statistical analysis. General statistical explanation: the assignment unit, outcome unit, timing of measurement, and comparison of interest need to align. In this trial, the record describes a physician-level assignment and a patient-focused testing outcome, with measurements at baseline and after intervention. It does not report whether the primary analysis compared post-intervention outcomes, changes over time, or a group-by-time contrast. These approaches answer related but not identical questions.
What do confidence intervals and p-values usually add?
General statistical explanation: a confidence interval expresses the precision of an estimated effect under the assumptions of the analysis. A narrow interval indicates greater precision than a wide interval, while an interval spanning values on both sides of no difference may be compatible with a range of interpretations. A p-value describes how unusual the observed data, or more extreme data, would be under a specified null hypothesis and model. Neither measure alone conveys clinical importance, and neither replaces the estimated effect itself. The registry record reports no primary-outcome confidence interval, p-value, or estimate, so none can be interpreted for this trial.
Multiplicity can matter when several intervention groups or outcomes are examined. General statistical explanation: making many comparisons can increase the chance that at least one appears noteworthy by chance, unless the analysis strategy addresses that issue. This trial includes three distinct alert groups and lists secondary outcomes concerning test changes, physician experience, cancellations, cost savings, and return on investment. The registry record does not report which comparisons were designated as primary, whether any adjustment for multiple testing was planned, or how secondary analyses were interpreted.
What is the role of planning illustrations?
The record describes a sample-size illustration based on a fraction of an observed effect, with a two-sided significance level of 0.05, equal allocation between two arms, and a normal approximation. It also distinguishes participant-per-arm quantities from total event quantities for time-to-event settings. These statements describe assumptions for a planning illustration, not the trial’s own design rationale or an analysis result. The registry record does not report the numerical planning outputs, the observed primary effect used as the basis, or the trial’s own sample-size justification. Such illustrative sample sizes should not be interpreted as observed power or as evidence that a comparison was adequately powered.
Overall, the available methods description supports a limited account of the intended endpoint and design, but it is not enough to reproduce or evaluate the primary analysis. The analysis model, handling of repeated observations, clustering, missing data, and adjustment for multiple comparisons are not reported in the registry record.
10. Multiplicity, Missing Data and Analysis Populations
The registry lists seven outcome measures and names secondary outcomes covering changes in pre-operative testing, categories of tests, physician experience, surgery cancellations, cost savings, and return on investment. It does not report a multiplicity strategy. In particular, the record does not say whether comparisons among the three alert groups and control were all intended to be confirmatory, whether some were exploratory, or whether statistical tests were adjusted for examining multiple outcomes or comparisons.
General statistical explanation: when many statistical questions are tested, the chance of at least one chance finding can rise. A prespecified hierarchy, a correction for multiple testing, or clear separation of confirmatory and exploratory analyses can help readers understand the evidence. Without a reported plan, the strength of any individual secondary result would be difficult to interpret. No secondary outcome estimates or tests are reported in the registry record.
Missing-data handling is also not reported. The record does not give the amount or pattern of missing primary-outcome information, reasons for missingness, exclusions, or methods for addressing incomplete observations. It does not state whether all eligible patients, only patients with complete data, or another analysis population contributed to the endpoint. Enrollment is listed, but the record does not provide group-specific analyzed denominators or a participant-flow account.
These gaps matter because missing observations and choices about analysis populations can affect both an estimate and its precision. General statistical explanation: complete-case analysis, imputation, and other approaches rely on different assumptions, so readers need the method and its rationale to judge robustness. The registry record does not identify any such approach. It also does not describe a statistical plan for physician-level assignment and patient-level outcomes, so the analysis population and handling of clustering remain unclear.
11. Safety Results
The registry reports serious adverse events as affected participants divided by participants at risk.
| Arm | Participants with serious adverse events | Participants at risk | Percentage |
|---|---|---|---|
| Alert 1 | 11 | 269 | 4.1% |
| Alert 2 | 4 | 267 | 1.5% |
| Alert 3 | 4 | 272 | 1.5% |
| Control | 12 | 237 | 5.1% |
Deaths
| Arm | Deaths (participants affected of at risk) |
|---|---|
| Alert 1 | 1 of 269 |
| Alert 2 | 3 of 267 |
| Alert 3 | 0 of 272 |
| Control | 2 of 237 |
Other (non-serious) adverse events
Participants with at least one non-serious event at or above the registry's 0% reporting threshold.
| Arm | Participants affected of at risk |
|---|---|
| Alert 1 | 9 of 269 |
| Alert 2 | 7 of 267 |
| Alert 3 | 12 of 272 |
| Control | 12 of 237 |
Most frequent serious adverse event terms
| Event term | Organ system | Alert 1 | Alert 2 | Alert 3 | Control |
|---|---|---|---|---|---|
| Inpatient Admission | General disorders | 11 of 269 | 4 of 267 | 4 of 272 | 12 of 237 |
Most frequent non-serious adverse event terms
| Event term | Organ system | Alert 1 | Alert 2 | Alert 3 | Control |
|---|---|---|---|---|---|
| Emergency Department visits that did not require admissions | General disorders | 9 of 269 | 7 of 267 | 12 of 272 | 12 of 237 |
Serious events with the highest arm-level rate: inpatient admission (5.1%).
The adverse-event listing reports serious adverse events, other adverse events, and deaths separately for the three alert groups and control. For Alert 1, the record lists 11 affected for serious events and 9 for other events, with 1 death. For Alert 2, it lists 4 affected for serious events and 7 for other events, with 3 deaths. For Alert 3, it lists 4 affected for serious events and 12 for other events, with 0 deaths. For control, it lists 12 affected for serious events and 12 for other events, with 2 deaths. The corresponding at-risk counts are 269, 267, 272, and 237 for Alert 1, Alert 2, Alert 3, and control, respectively.
The record identifies Inpatient Admission as the leading serious-event term and reports 5.1%. It does not explain the event-reporting window, how events were attributed to groups, whether events could be counted more than once, or whether the counts refer to people or event occurrences beyond the labels shown. No rate or comparative safety analysis is reported. The counts should therefore be read as registry-reported group summaries, not as evidence that an alert caused or prevented an event. They do not address the primary testing outcome.
12. What the Result Does, and Does Not, Mean
The record establishes the clinical and statistical question: whether behavioral alerts embedded in the electronic health record change pre-operative testing for cataract surgery. It also describes a randomized, double-masked, parallel design and identifies three alert messages and a usual-care group. Those details explain what the study set out to evaluate, but they do not answer the outcome question on their own.
The primary endpoint is defined as a change in the percentage of patients undergoing pre-operative tests, with baseline and post-intervention periods specified. Yet the registry record does not report the endpoint values, group comparisons, uncertainty intervals, or statistical tests. It therefore does not show how much testing changed, whether the groups differed, or how precise an estimate might be. No conclusion about the direction or size of an intervention effect can be drawn from the information recorded for the primary outcome.
Even if an outcome estimate were available, its interpretation would depend on its definition, denominator, analysis population, and treatment of physician-level assignment. The record does not provide these analytic details. It also does not report how missing data or multiple comparisons were handled. These are not merely technical extras: they help determine what population an estimate represents and how confidently it can be interpreted.
The adverse-event counts provide a separate description of reported safety events by group, but they do not substitute for the testing outcome or demonstrate causation. The listed planning assumptions likewise do not provide observed evidence about the alert groups. They describe a planning illustration rather than observed power.
In short, the registry record documents a completed trial and its intended comparison, but its posted information is insufficient to determine the result for the primary endpoint. That distinction separates a trial’s question and design from evidence about its outcome.
13. Why This Trial Matters Statistically
This trial is statistically notable for the relationship between its assignment structure and its outcome. The registry describes assignment of physicians to intervention groups, while the primary measure concerns testing among patients seen in pre-operative visits. This design feature makes the unit of assignment important when interpreting comparisons. General statistical principle: when outcomes are grouped within assigned clinicians, analyses may need to account for the resulting dependence between observations. The registry record does not say how that issue was addressed.
The intervention also has three distinct versions rather than a single alert. Each message emphasizes a different potential harm, and the control group receives usual care. Separate comparisons could help distinguish whether the messages differ in their association with ordering behavior; a combined comparison would address a broader question about alerts overall. The record does not specify which contrast was primary or how multiple comparisons were handled. That makes a clear analysis plan especially relevant to interpretation.
The endpoint uses a change across a baseline and post-intervention period, so both the time contrast and the group contrast matter. General statistical principle: a change over time within one group is not the same as a randomized comparison between groups. Other changes in practice over time could affect both periods, and a comparative analysis is needed to isolate differences associated with assignment under the design’s assumptions. The registry record does not report the analysis used or any estimates.
Finally, this record illustrates why a completed-trial status and an endpoint definition are not equivalent to posted outcome evidence. Without primary-outcome values, denominators, uncertainty measures, and analysis details, readers cannot judge the magnitude or precision of the trial’s result. The statistical value of the trial cannot be evaluated from design labels alone.
14. Limitations
- The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
- Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.
The largest limitation for interpreting this record is that it contains no reported primary outcome results. Baseline and post-intervention testing percentages, changes by group, comparisons with control, confidence intervals, and p-values are absent. The registry also lists zero analyses, so there is no statistical result to evaluate against the stated endpoint.
The endpoint is not fully operationalized in the available description. Although it is described as the percentage of patients with orders and as a change in pre-operative testing, the exact denominator, handling of repeat visits or orders, and method for combining testing categories are not stated. The record names a mean and standard deviation but does not identify the level of aggregation for those summaries.
The design description raises an additional interpretive issue: physicians are assigned to groups, while patients contribute to the testing outcome. The record does not state whether the analysis accounted for clustering, how many physicians contributed by group, or how patient observations were linked to physicians. It also does not specify the comparison model or how baseline measurements entered the analysis.
Missing-data procedures, analysis-population definitions, group-specific outcome denominators, exclusions, and follow-up completeness are not reported. Nor does the record explain the masking procedure despite recording double masking. The protocol, sample-size justification, allocation procedure, and multiplicity plan are also not reported in the registry record. These limitations constrain assessment of both the intended methods and any possible outcome; they do not establish that the trial lacked such procedures, only that they are not documented in this record.
15. Related Tutorials
16. Related Statistical Calculators
17. Sources
- ClinicalTrials.gov: Using Behavioral Economics to Reduce Low-Value Care — NCT04104256.
- Registry record last updated 2024-09-19; status last verified 2024-08.
- Results first posted 2024-09-19.
- Data retrieved from ClinicalTrials.gov on 2026-11-09.
- Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.
Continue through the Clinical Biostats statistical pathway
Use this trial's endpoint as a starting point for deeper study of effect measures, confidence intervals, and sample-size planning.
18. Record Summary
NCT04104256 is a completed pragmatic randomized, double-masked, parallel-group trial sponsored by the University of California, Los Angeles. The registry lists enrollment of 1045 and a primary completion date of 2023-04-11. Three electronic health record alerts emphasized potential safety, financial, or psychological harms of pre-operative testing; the comparator was usual care.
The primary outcome was defined as change in the percentage of patients undergoing pre-operative testing, measured from baseline to post-intervention at 12 months. The registry record does not report primary-outcome estimates, comparisons, uncertainty measures, or analysis methods. It lists adverse-event counts by group, but no comparative safety analysis. Missing-data methods, analysis populations, and multiplicity procedures are not reported.