Virtual Reality vs Technical Video in Surgical Training: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Virtual Reality vs Technical Video in Surgical Training (NCT04404010), a randomized na study. The primary outcome is Objective Structured Assessment of Technical Skills (OSATS).

2. Clinical Question

The registry describes a study of teaching technical skills for reverse total shoulder arthroplasty. It frames immersive virtual reality as a training approach to compare with instructional video, a method described as commonly used in orthopaedic education. The primary endpoint is an OSATS score assessed during surgery up until immediately after surgery. The registry says a higher score indicates a better outcome.

The study also listed secondary objectives concerning the Global Ratings Scale, Transfer of Training, and Transfer Effectiveness Ratio. The description says that validating a virtual reality ratings scale through correlation with real-world performance was a secondary objective. The registry record does not report results for those secondary outcomes, so the primary score summaries do not answer those additional questions.

The central statistical question is descriptive as well as comparative: what mean OSATS scores were recorded in each training arm, and how precisely were those means estimated? The registry reports a mean and 95% confidence interval for each arm. It does not report a direct estimate of the between-arm contrast or an inferential test for that comparison.

3. Trial Design

The registry describes a randomized, blinded intervention-control trial with a parallel-group model and single masking. Participants were assigned to immersive virtual reality training or surgical video training. The record does not specify who was masked, so the masking label alone does not establish whether participants, assessors, or other study personnel were unaware of assignment.

The two training approaches covered completion of reverse shoulder arthroplasty. In the virtual reality arm, participants trained using an immersive simulator; in the video arm, participants watched a technical surgical instructional video. The study population was senior residents learning at the 2020 annual Canadian Shoulder and Elbow Society meeting. The registry lists surgical training as the condition.

The study is marked completed, and the registry records enrollment of 20. For the primary outcome, each arm's posted summary lists 9 participants. The record does not explain the relationship between the enrollment figure and the outcome-specific participant counts. Reasons for any difference, if applicable, are not reported in the registry record.

The primary outcome was OSATS, reported as a mean score with a 95% confidence interval. Its assessment timeframe is during surgery up until immediately after surgery. The record lists three secondary outcomes: Global Ratings Scale, Transfer of Training, and Transfer Effectiveness Ratio. It also describes scale validation through correlation with real-world performance as a secondary objective.

Although randomization is recorded, details of the randomization sequence, allocation concealment, and implementation are not reported in the registry record. The protocol's planned analysis, the handling of participants who did not contribute primary outcome data, and the analysis population are also not reported. Those omissions limit assessment of how the trial was conducted and how its posted summaries relate to all randomized participants.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: 1 participant was mistakenly consented (error in screening - participant was not a PGY4 or 5) and randomized to immersive virtual reality arm as part of study. After randomization, but prior to study testing we realized the participant was not eligible, therefore they were excluded prior to any further study procedures commenced. 1 additional participant was mistakenly consented (error in screening - participant was not a PGY4 or 5), therefore excluded PRIOR to randomization.

Overall Study

Reasons for not completing this period:

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Objective Structured Assessment of Technical Skills (OSATS), measured over During surgery up until immediately after surgery..

The primary endpoint is the Objective Structured Assessment of Technical Skills (OSATS), a score intended to assess overall technical ability during surgery. The registry describes a checklist with 27 items. An assessor marks whether each task was completed successfully: correctly completed items receive 1 point, while incorrectly completed items receive no point. The points are summed, with a possible score ranging from 0 to 27. A higher score indicates a better outcome according to the registry.

This type of structured checklist provides a consistent way to grade observed performance rather than relying only on a general impression. It makes task completion the basis of the score and allows performance to be expressed numerically. The registry identifies OSATS as a validated assessment tool, but it does not provide details of the validation evidence or specify assessor training and procedures.

The timeframe is during surgery up until immediately after surgery. This makes the endpoint a measure of assessed performance in the operative setting following the assigned training, rather than a measure of long-term retention. The registry reports the endpoint as a mean score and gives a 95% confidence interval for each arm.

The record does not report a separate endpoint definition for the timing of each checklist item, nor does it describe whether assessors were masked to training assignment. Those details matter when interpreting an observed performance score, particularly if assessment involves judgment. The posted summaries therefore describe the recorded OSATS scores, while leaving some measurement and assessment procedures unspecified.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The registry reports a continuous primary outcome: an OSATS score. For a continuous measure, a mean describes the arithmetic average of the recorded values in a group. Here, each arm has a posted mean, and the record gives a 95% confidence interval around each mean. The score has a defined range and higher scores are described as better, but the registry does not provide a participant-level distribution or a separate measure of spread such as a standard deviation.

What does a confidence interval communicate?

A confidence interval expresses uncertainty in an estimate using a range of values compatible with the data and the method used to calculate the interval. In repeated use of a specified procedure, intervals constructed in that way would cover the underlying quantity at the stated rate. A 95% confidence interval is not a statement that there is a 95% probability that a fixed true value lies inside this particular interval. It also is not a range containing 95% of individual scores.

The two intervals concern the mean in each arm separately. They are not a confidence interval for the difference between arms. A direct comparison would ordinarily report a contrast on the OSATS scale and its uncertainty, with an inferential procedure suited to the design and analysis plan. That contrast and its inferential result are not reported in the registry record. As a result, the arm-specific intervals cannot substitute for a direct comparison interval or test.

What can be inferred from the means?

The mean is useful for describing average performance, but it does not reveal whether scores are tightly grouped or widely dispersed. Different participant-level patterns can produce the same mean. Knowing the number contributing to a summary and the confidence interval helps convey how precisely the mean is estimated, but does not reconstruct the underlying score distribution.

In general, the choice of analysis for a continuous endpoint can depend on the score's distribution, the study design, and the prespecified analysis plan. The registry record does not describe the primary comparison method, any adjustment variables, assumptions checks, or handling of possible departures from a model. It is therefore not possible to attribute a particular statistical model or testing procedure to this trial.

What does the sample-size illustration mean?

The statistical planning illustration is based on the observed effect, with a two-sided significance level of 0.05, equal allocation between arms, and a normal approximation. It considers effects defined as stated fractions of the observed effect. Such calculations describe hypothetical planning needs under their assumptions; they are not observed power and do not show that the original trial was designed using those assumptions. The registry does not report the trial's own sample-size justification or planning assumptions.

Planning calculations based on an observed effect should be interpreted cautiously in general. An observed estimate can be uncertain, and a planning exercise that uses it is not independent evidence that the effect is stable. Any calculated planning requirement is an illustration rather than a finding about participants enrolled in this trial. The registry's own design rationale is not reported in the registry record.

Why is a direct comparison important?

Randomized assignment is intended to support comparison between groups, but the comparison needs to be estimated and reported. Group-specific means summarize each arm; they do not quantify the uncertainty of the arm-to-arm contrast. A contrast's uncertainty depends on the observed data and the comparison method. Without the direct estimate and its interval or test result, the strength and precision of the comparative evidence cannot be assessed from the posted outcome summaries alone.

Statistical significance and practical importance are also distinct concepts in general. A test addresses compatibility with a specified statistical null under its assumptions; it does not by itself establish whether a difference is educationally meaningful. The registry does not report a minimally important score difference or a direct test result, so neither question can be answered from these summaries.

Finally, general statistical interpretation depends on transparent reporting of data completeness, outcome assessment, and analysis choices. The registry record does not describe missing-data procedures or the analysis population, and the assessor-masking details are not reported. Those gaps do not alter the stated means, but they constrain how fully their relation to the randomized comparison can be evaluated.

10. Multiplicity, Missing Data and Analysis Populations

The registry lists three secondary outcomes: Global Ratings Scale, Transfer of Training, and Transfer Effectiveness Ratio. It also describes validation of a virtual reality ratings scale through correlation with real-world performance as a secondary objective. Numerical results for these outcomes, their analysis methods, and any multiplicity adjustment are not reported in the registry record.

Multiplicity arises in general when several outcomes or comparisons are examined, because considering multiple statistical tests can increase the chance of at least one apparently notable result. Whether and how that issue was addressed in this study is not reported in the registry record. No claim about adjusted or unadjusted secondary findings can be made from the posted information.

The registry gives enrollment of 20, while the primary outcome summaries list 9 participants in each arm. It does not explain the endpoint-specific counts or identify why they differ from enrollment. Missing outcome data, exclusions, withdrawal, and the timing or completeness of follow-up are not described, so the reason for the difference is not reported in the registry record.

The method for handling missing data is not reported in the registry record. It is also not reported whether the analysis followed an intention-to-treat principle, used another analysis population, or included only participants with observed scores. The record does not provide participant flow or a breakdown of missingness by reason. These omissions make it difficult to assess whether the posted endpoint summaries represent all randomized participants or a subset defined by outcome availability.

These are reporting limitations, not proof that data were missing or that a particular analysis was used. The available summaries can be read as the reported arm-specific OSATS results, while their relationship to the full randomized population and the interpretation of the secondary outcomes remain unclear.

11. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The safety record lists 9 participants at risk in each arm. For immersive virtual reality, it records 0 serious adverse events, 0 other adverse events, and 0 deaths. For surgical video, it also records 0 serious adverse events, 0 other adverse events, and 0 deaths. No serious adverse-event terms are listed.

These figures describe what the registry records for the listed groups. A zero count means no event of that category was recorded among the participants represented in the safety summary; it does not prove that an event is impossible or that the training approaches have no safety concerns. The record does not provide additional information about how adverse events were defined, collected, or monitored.

With a limited number of participants recorded in each group, absence of recorded events should not be interpreted as evidence that uncommon events cannot occur. The safety summaries are specific to this trial and its recorded observation. Duration of safety follow-up and details of any event-review procedures are not reported in the registry record.

These figures should be kept separate from the OSATS performance outcome. They do not provide evidence about comparative technical performance, and the performance summaries do not answer questions about adverse events. The registry's safety counts are descriptive, rather than a basis for broad claims about safety.

12. What the Result Does, and Does Not, Mean

The result describes observed OSATS scores after two different training approaches for reverse shoulder arthroplasty. The registry reports a mean score of 15.9 in the immersive virtual reality arm and 9.4 in the surgical video arm, with a 95% confidence interval for each mean. The posted values are descriptive findings for the endpoint and participants included in those summaries.

The result does not, by itself, establish a statistically significant difference. The registry record does not report a direct between-arm effect estimate, its confidence interval, or a p-value. Separate group means and their confidence intervals should not be mistaken for a formal test of the contrast. The summaries also do not establish whether any observed difference would be educationally meaningful; a threshold for meaningful change is not reported in the registry record.

Randomization is recorded, but the registry does not describe sequence generation or allocation concealment, and it does not identify who was masked. The record also does not explain the relationship between enrollment and the endpoint-specific group counts, or report the missing-data approach and analysis population. These details affect how confidently the observed summaries can be connected to the randomized comparison.

The question is specific to senior residents learning this procedure in the described setting and to assessment during surgery up until immediately afterward. The results do not establish long-term retention, performance in other operations, or effectiveness of other training formats. Nor do the primary summaries answer the secondary questions about global ratings, transfer of training, transfer effectiveness, or scale validation; those results are not reported in the registry record.

The registry therefore supports a descriptive account of the recorded OSATS means, not a complete inferential evaluation of the training comparison. The safety record separately reports zero serious and other adverse events and zero deaths in each listed group, but such counts should not be generalized beyond the recorded participants and observation.

13. Why This Trial Matters Statistically

This trial matters statistically because it places a structured, continuous technical-performance score at the center of a randomized comparison of two training modalities. The endpoint is tied to a defined checklist and operative assessment, rather than only to participants' impressions of training. Reporting a mean and confidence interval for each arm gives readers information about the observed scores and uncertainty around the arm-specific averages.

At the same time, the registry illustrates why complete comparative reporting is important. The central research question concerns the relative performance of the training approaches, but the posted primary results are arm-specific summaries. A direct contrast and its uncertainty would address the comparative question more directly. Without them, readers cannot assess the precision of the estimated arm-to-arm effect from the registry record.

The study also identifies secondary outcomes relevant to measurement and transfer: Global Ratings Scale, Transfer of Training, and Transfer Effectiveness Ratio. These outcomes could speak to whether a checklist score aligns with other assessments or captures transfer of learning, but numerical findings are not reported in the registry record. The stated validation objective therefore remains unanswered by the posted primary summaries.

The trial also highlights a distinction between enrollment and outcome-specific reporting. Enrollment is listed as 20, while each arm's primary summary lists 9 participants. Without participant flow and missing-data details, readers cannot determine how the endpoint summaries relate to the full randomized group.

For clinical researchers and biostatisticians, these are practical reporting issues: a continuous outcome needs a clear definition, a transparent account of analyzed participants, and an inferential result matched to the randomized comparison. This record provides the outcome definition and group summaries, while leaving important analysis and completeness details unreported.

14. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The primary limitation for interpretation is the absence of a reported direct comparison between arms. The registry provides each arm's mean OSATS score and 95% confidence interval, but no contrast estimate, confidence interval for the contrast, or p-value. This prevents a direct assessment of the comparative estimate and its precision from the registry record.

The endpoint summaries list 9 participants in each arm, while the record lists enrollment of 20. The registry does not explain this difference. Participant flow, reasons for non-contribution to the primary outcome, and missing-data handling are not reported in the registry record. The analysis population is likewise not specified.

Masking is labeled single, but the identity of the masked party is not reported. The record does not describe assessor masking or procedures for ensuring consistent scoring. Those details are particularly relevant to an endpoint based on assessment of observed technical performance.

The trial's own sample-size rationale, statistical analysis plan, randomization procedures, and allocation concealment are not reported in the registry record. The posted planning illustration is based on the observed effect and is not a substitute for the original design justification. No direct analysis results are posted for the three secondary outcomes, and multiplicity handling is not described.

The record describes the population and task as senior residents learning reverse shoulder arthroplasty at the 2020 annual Canadian Shoulder and Elbow Society meeting. The reported outcome timeframe extends through immediately after surgery, so these summaries do not establish longer-term retention. Results for other learner groups, settings, procedures, or training approaches are not reported in the registry record.

15. Related Tutorials

16. Related Statistical Calculators

17. Sources

• ClinicalTrials.gov: Virtual Reality vs Technical Video in Surgical Training — NCT04404010.
• Registry record last updated 2022-09-28; status last verified 2022-09.
• Results first posted 2021-09-05.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

18. Record Summary

The registry record describes a completed, randomized, parallel-group trial comparing immersive virtual reality training with technical surgical instructional video training for reverse shoulder arthroplasty. The population is senior residents learning at the 2020 annual Canadian Shoulder and Elbow Society meeting. The registry lists enrollment of 20 and single masking, without specifying who was masked.

The primary outcome is the Objective Structured Assessment of Technical Skills score, assessed during surgery up until immediately after surgery and reported as a mean with a 95% confidence interval. The posted summaries list 9 participants in each arm. Mean OSATS scores are 15.9 for immersive virtual reality and 9.4 for surgical video. The registry does not report a direct statistical comparison between arms.

Global Ratings Scale, Transfer of Training, and Transfer Effectiveness Ratio are listed as secondary outcomes, but their numerical results are not reported in the registry record. The record also does not report the missing-data approach, analysis population, or trial-specific sample-size justification. It records zero serious adverse events, zero other adverse events, and zero deaths in each listed arm.