Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment (NCT03530098), a randomized na study. The primary outcome is Paired Difference of Skeletal Age Estimate.

2. Clinical Question

3. Trial Design

The registry describes a prospective, multicenter, randomized controlled trial of an artificial-intelligence algorithm for skeletal-age assessment. It lists randomized allocation, a parallel-group model, and no masking. The study status is completed. The recorded start date is 2018-07-12 and the primary completion date is 2019-08-31. The listed phase is NA. Enrollment is recorded as 1903.

The control arm, titled Control (Without-AI), is described as current standard of care with no intervention. In the Experiment (With-AI) arm, participating radiologists received the BoneAgeModel output and were asked to incorporate it into their normal workflows. The brief summary describes de-identified hand radiographs being sent to the algorithm, with its output shown to the radiologist. It also states that the radiologist’s diagnosis was final and considered independent of the algorithm’s output. The record says radiologists involved in the study were trained to recognize the surfaced prediction as AI output.

The primary analysis population was narrower than the overall enrolled population: it included participants with ground-truth-labeled exam results and no bone deformity. The registry reports 739 participants for the primary outcome in the control arm and 792 in the AI arm. It does not report why other enrolled participants are not represented in those primary analysis counts, nor does it describe how missing or unusable observations were handled.

The primary completion timeframe is described as up to 10 minutes to acquire the scan and up to 2 days to complete diagnosis review. The record lists Time for Diagnosis as a secondary outcome, but the secondary outcome results are not reported in the registry record summarized by these results. Further details about the sites, randomization procedure, allocation concealment, or conduct of the trial are not reported in the registry record.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Analysis population for the primary outcome, as defined in the registry: Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity. Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity. Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.

6. Primary Endpoint

The registry lists the primary outcome as Paired Difference of Skeletal Age Estimate, measured over Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review.

The primary endpoint is the mean absolute difference between dictated final impressions and the consensus determination of a panel of radiologists following review. It is measured in months. The registry describes the dictated final impression as the baseline measure by a radiologist. The consensus determination was based on review by a panel; for the ground-truth label, the interpretations of 4 radiologists were averaged to determine a final label.

An absolute difference expresses the size of a discrepancy without retaining its direction. A smaller mean absolute difference therefore corresponds to less average distance from the stated reference determination in this metric. The endpoint is used to assess agreement with that reference, rather than to establish that the reference is error-free or to assess every aspect of diagnostic performance. It does not, by itself, indicate whether estimates tend to be higher or lower than the consensus value.

The registry reports group means of 5.95 months for the control arm and 5.36 months for the AI arm. Those values are arm-level means for the primary analysis population, not individual patient results. The comparison is reported separately as -0.59, with a 95% confidence interval from -1.14 to -0.04. Although the endpoint itself is an absolute difference, a contrast between arm means can be negative; that contrast is not an individual absolute difference.

The endpoint’s stated timeframe is up to 10 minutes for scan acquisition and up to 2 days for diagnosis review. These time limits describe the measurement period recorded for the outcome; they do not establish the amount of time each assessment took.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

What does a mean represent?

The endpoint is continuous: each assessment contributes a measure expressed in months, and the arm-level result is reported as a mean. A mean gives a summary of the values in an analysis group. It is useful for describing the average magnitude of the endpoint, but it does not describe every observation. Different distributions can have the same mean, so the mean alone does not show whether values are tightly grouped, widely dispersed, or influenced by unusually large observations.

Here, the endpoint is an absolute difference from a consensus determination. Absolute differences remove the sign of individual discrepancies, so the group mean summarizes their average magnitude. It does not preserve whether a radiologist’s estimate was above or below the consensus determination. Nor does a mean absolute difference alone show the proportion of assessments within a particular tolerance; such a threshold result is not reported in the registry record.

What does the confidence interval convey?

A confidence interval expresses uncertainty around an estimated quantity under the assumptions of the analysis. For the reported comparison, the registry gives a 95% confidence interval from -1.14 to -0.04. The interval is for the group contrast, not for the range of individual participant values. Its endpoints indicate the values compatible with the observed data under the statistical procedure; they do not assign a probability to a particular fixed value being true.

Because the interval is for a difference, values below zero indicate a contrast in one direction and values above zero would indicate the opposite direction, given the order in which groups are compared. The estimate is -0.59. The registry record does not explain the contrast ordering in a formal analysis specification, so the arm means are also important context when interpreting its direction. The reported interval does not include zero, while its upper endpoint is close to zero. The interval communicates both the direction and uncertainty of the estimated contrast.

How should the p-value be read?

The reported p-value is 0.04 for a two-sided t-test. In general, a p-value describes how unusual results at least as discrepant as those observed would be under a specified null model and its assumptions. It is not the probability that the null hypothesis is true, and it does not measure the size or practical importance of an effect. The record’s p-value should be understood in the context of the reported endpoint, analysis population, and comparison. The registry does not report a multiplicity adjustment or a prespecified testing hierarchy.

A two-sided test considers departures in either direction rather than only a difference in one prespecified direction. A t-test compares group means under assumptions about the data and the analysis. The registry names a two-sided t-test but does not report diagnostics, variance assumptions, covariate adjustment, or other details needed to independently assess those assumptions.

What does randomization contribute?

Random allocation is intended to make the assigned groups comparable on average, including characteristics that may not have been measured, so that differences in outcomes can be evaluated as a comparison of the assigned workflows. This is a general statistical rationale, not a claim about the success of allocation in this particular study. The record does not describe the randomization sequence, allocation concealment, or baseline characteristics, so those aspects cannot be assessed from the registry record.

The trial is listed as parallel and unmasked. A parallel design compares separate assigned groups over the study assessment, rather than having each participant receive both workflows. The record’s lack-of-masking designation means the study is not described as blinded; details of who knew the assignment and when are not reported. Because radiologists in the AI arm saw the algorithm output, knowledge of the workflow was part of the intervention itself. The registry does not report whether outcome review or consensus panel members were masked to assignment.

Why distinguish the registry label from the outcome scale?

The primary outcome is a continuous mean absolute difference in months. The registry labels the reported parameter “Odds Ratio (OR),” but an odds ratio is conventionally a ratio for odds and is not on the same scale as a difference in months. The reported estimate is -0.59, with limits of -1.14 and -0.04, which are signed values and are consistent with a difference-scale contrast, not a conventional odds ratio. The listed test is a t-test. The registry does not clarify whether the parameter label is a metadata error or whether some other reporting convention was used. Interpretation should therefore emphasize the stated endpoint, arm means, and signed estimate rather than treating the label as a meaningful odds ratio.

In general, over-dispersion occurs when observed variability is greater than a model assumes. It can make standard errors too small and confidence intervals too narrow if not addressed. The primary outcome is analyzed as a continuous measure with a t-test, and the registry record does not report an over-dispersion assessment or a model for counts. No conclusion about over-dispersion can be drawn from the reported summary statistics alone.

What can planning illustrations say?

Sample-size calculations based on an observed effect are planning illustrations, not observed properties of the completed trial and not “observed power.” The planning assumptions recorded for this study use a two-sided significance level of 0.05, equal allocation, and a normal approximation; the effect to detect is specified as a fraction of the observed effect. Such calculations can illustrate how a future study’s participant requirements depend on the effect size targeted and the assumptions chosen. They do not establish what the original trial planned, what effect a future study will obtain, or whether the completed trial had a particular probability of detecting its result. The original sample-size justification is not reported in the registry record.

10. Analyses Reported by the Sponsor

The registry reports a two-sided t-test, a comparison value of -0.59, a 95% confidence interval from -1.14 to -0.04, and p = 0.04. The arm means are 5.95 months for control and 5.36 months for AI-assisted assessment. The comparison is therefore read as a contrast on the continuous endpoint scale, although the parameter is labeled “Odds Ratio (OR)” in the record. That label is not appropriate to interpreting a negative difference in months, and the record does not explain it.

The registry does not identify a separate recalculated estimate or an independent analysis alongside the reported test. The comparison should therefore be described as the result reported in the registry, rather than as an independently reproduced estimate. No discrepancy is identified between the posted group summaries and the reported comparison in the registry record. The stated comparison and its confidence interval remain the relevant inferential results, with the parameter-label inconsistency noted as a reporting limitation.

The available summary statistics do not allow a full reconstruction of the sponsor’s analysis. Participant-level data, details of the analysis model, and a complete statistical analysis plan are not reported in the registry record. The reported p-value and interval should be interpreted as the registry’s analysis, without assuming unreported adjustment or additional modelling.

11. Multiplicity, Missing Data and Analysis Populations

The registry lists Time for Diagnosis as a secondary outcome and reports two outcome measures and two analyses overall. The secondary outcome results are not reported in the registry record summarized by these results. The record also does not state whether the primary outcome was designated as the sole confirmatory test, how secondary outcomes were ordered, or whether any adjustment for multiple testing was planned or performed.

Multiplicity arises when several outcomes or comparisons are tested. In general, conducting multiple tests can increase the chance of at least one apparently notable result even when no corresponding effect exists. Whether and how to address that concern depends on the trial’s prespecified aims and analysis plan. Because those details are not reported in the registry record, the primary p-value should not be treated as if a particular multiplicity procedure was used.

The stated primary analysis population includes participants with ground-truth-labeled exam results and no bone deformity. The registry gives arm-level analysis counts of 739 for control and 792 for AI. It does not report the reasons that enrolled participants are absent from these analysis counts, whether observations were missing, whether any participants were excluded after randomization, or how such cases were handled. A detailed missing-data method, including any imputation or sensitivity analyses, is not reported in the registry record.

The record does not specify whether the analysis followed an intention-to-treat principle, a per-protocol approach, or another defined population rule beyond the primary population description. It also does not provide a flow of participants from enrollment through analysis. This limits assessment of how the analyzed participants relate to randomized assignment and whether exclusions could affect the comparison. These omissions do not establish that data were missing or that the analysis was biased; they mean the registry record does not supply the information needed to evaluate those possibilities.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry reports zero serious adverse events in the control group, among 939 listed as at risk, and zero in the experiment group, among 964 listed as at risk. It also reports zero other adverse events in each group, with those same group-specific at-risk counts, and zero deaths in both groups. No serious adverse event terms are listed.

These figures mean that no events in the displayed categories were recorded for the listed groups in the registry’s adverse-event results. A zero count is not evidence that the underlying risk is exactly zero. Rare events may not occur in a study even when a risk exists, and the ability to detect safety patterns depends on study size, duration, event definitions, and completeness of reporting. Those details are not fully reported in the registry record.

The adverse-event at-risk counts differ from the primary outcome analysis counts. The registry does not explain the relationship between those populations, and they should not be treated as interchangeable. The serious and other adverse-event entries are separate from the primary endpoint of mean absolute difference in skeletal-age estimates. The safety figures do not establish that either workflow is free of risk, and they do not provide a basis for medical advice.

13. What the Result Does, and Does Not, Mean

The posted results describe a comparison of radiologists’ assessments in a randomized, parallel trial, with the AI output shown to radiologists in the experiment arm and no AI intervention in the control arm. For the defined primary analysis population, the mean absolute difference from the panel consensus determination was 5.95 months in control and 5.36 months in the AI arm. The registry’s reported group contrast was -0.59, with a 95% confidence interval from -1.14 to -0.04 and p = 0.04. This indicates a statistically detectable difference under the reported test and assumptions.

The endpoint measures average discrepancy from a reference consensus, not diagnostic accuracy in every sense. It does not show how individual results were distributed, whether errors were in one direction, or how many assessments fell within a clinically chosen tolerance. It also does not measure all consequences of using AI in a workflow. The registry lists Time for Diagnosis as a secondary outcome, but results for that outcome are not reported in the registry record.

The result does not establish that the algorithm’s output was correct, that the consensus determination is error-free, or that the observed comparison will be repeated in other settings. The registry describes a particular population that excludes participants with bone deformity and requires ground-truth-labeled exam results. Details about sites, participants’ characteristics, and subgroup performance are not reported in the registry record.

The comparison is not a substitute for clinical judgment or an individual assessment. No treatment recommendation follows from these statistical summaries. The record reports no serious adverse events, other adverse events, or deaths in the listed groups, but zero recorded events do not demonstrate an absence of all possible risk. The most defensible reading is limited to the measured endpoint, the analyzed population, and the uncertainty expressed by the reported confidence interval.

14. Why This Trial Matters Statistically

This trial raises a statistical question that is important when evaluating decision-support systems: whether making an algorithm’s output available to a clinician is associated with a change in the clinician’s final assessment relative to a reference determination. The tested workflow was not an algorithm acting alone. Radiologists received the AI output and incorporated it into their normal workflow, while the radiologist’s diagnosis remained final. That distinction matters when interpreting what the comparison evaluates.

The endpoint also makes the reference standard central to interpretation. A panel consensus provides a defined comparator for the dictated impressions, but the registry does not establish that consensus as error-free. The result concerns agreement with that panel determination, summarized as a mean absolute difference in months. It does not automatically answer broader questions about diagnostic quality, clinical decisions, or outcomes.

Statistically, the record is useful for illustrating the difference between a group mean, a contrast between group means, and an individual absolute difference. The arm means are nonnegative measures of discrepancy, while the reported contrast is signed. The registry’s odds-ratio label is inconsistent with the continuous endpoint and signed estimate, so interpreting the scale carefully is essential. The confidence interval and p-value give information about the reported group comparison, but they do not resolve the labeling or reporting gaps.

The record also shows why analysis populations and outcome reporting matter. The primary analysis is restricted to participants with ground-truth-labeled exams and no bone deformity, while overall enrollment and primary analysis counts differ. The reasons and handling are not reported. In addition, a secondary outcome is listed without results. These limitations do not invalidate the reported primary comparison, but they constrain how broadly it can be interpreted and how fully its analysis can be evaluated.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record does not provide participant-level data, baseline characteristics, site-level results, or subgroup analyses. It therefore does not allow assessment of whether the observed comparison varied across participants or study centers. The primary analysis population excludes participants with bone deformity and requires ground-truth-labeled exam results, so the result is specific to that recorded population. The record does not report how many enrolled participants did not enter the primary analysis or why.

Methods details are incomplete. The registry does not describe the randomization sequence, allocation concealment, statistical analysis plan, variance assumptions for the t-test, covariate adjustment, or diagnostic checks. Masking is listed as none, but the record does not specify which individuals were aware of assignment during each stage of assessment. It also does not state whether the consensus panel was masked to the assigned workflow.

The main outcome is agreement with a radiologist-panel consensus, rather than an independently established error-free reference. The summary does not provide the distribution of individual absolute differences, measures of spread for the arm means, directional errors, or performance at specific thresholds. This limits assessment of variation and practical meaning beyond the reported average.

The parameter is labeled as an odds ratio despite the continuous endpoint and negative reported estimate. The record does not explain this inconsistency. It reports a secondary outcome, Time for Diagnosis, but does not report its result in the available registry results. Multiplicity handling, missing-data methods, and a detailed analysis-population rule are also not reported in the registry record. These are limits on interpretation and independent verification, not evidence that any particular bias or analysis problem occurred.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment — NCT03530098.
• Registry record last updated 2021-06-09; status last verified 2021-05.
• Results first posted 2020-10-26.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT03530098 is a completed, randomized, parallel, unmasked trial titled “Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment.” The registry describes a prospective, multicenter comparison of radiologists’ skeletal-age assessments with or without the BoneAgeModel output. Enrollment is recorded as 1903.

The primary endpoint is the mean absolute difference, in months, between dictated final impressions and a radiologist-panel consensus determination. In the primary analysis population, the reported means were 5.95 months for control and 5.36 months for AI-assisted assessment. The registry reports a comparison of -0.59, a 95% confidence interval from -1.14 to -0.04, and p = 0.04 from a two-sided t-test. The result’s parameter label, “Odds Ratio (OR),” does not match the continuous endpoint scale.

The registry lists Time for Diagnosis as a secondary outcome, but its result is not reported in the registry record. Serious adverse events, other adverse events, and deaths are recorded as zero in both listed groups. Further details on missing data, multiplicity, and the full statistical plan are not reported in the registry record.