Bone Age
Conditions
Keywords
Age Determination by Skeleton, Machine Learning, Deep Learning, Artificial Intelligence, Prospective, Clinical Validation
Brief summary
The purpose of this study is to understand the effects of using an Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make an estimation of the bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.
Detailed description
The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the radiographic images and health records. As part of this study, the radiologists will estimate skeletal age from radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age. This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard (gold standard) composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD). The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups. The investigators have recruited and analyzed data from a sample size of 1600 exams. Patients getting these exams will not undergo any research procedures that deviate from the current standard practices.
Interventions
DEVICEBoneAgeModel
BoneAgeModel is an Artificial Intelligence tool that takes in a hand radiograph and gender, and outputs the skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final.
Sponsors
Stanford University
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
DIAGNOSTIC
Masking
NONE
Intervention model description
A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the exam using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their skeletal age estimation. In all cases, the decision of the radiologist will be considered final.
Eligibility
Sex/Gender
ALL
Healthy volunteers
No
Inclusion criteria
Exams that meet the following inclusion criteria will be included: (1) exams read by radiologists who interpret pediatric skeletal age exams and verbally consent to participate (2) exams that contain a procedure code or study description indicative of a skeletal age exam. Exams containing more than one radiograph will not be included. Exams for which a trainee provides a preliminary interpretation will be excluded. No further
Exclusion criteria
will be applied on the basis of image quality metrics or manufacturers. No
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Paired Difference of Skeletal Age Estimate | Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review | Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review. |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| Time for Diagnosis | Up to approximately 4 minutes | Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not. |
Countries
United States
Participant flow
Participants by arm
| Arm | Count |
|---|---|
| Control (Without-AI) Diagnosis by radiologists made according to current standard of care methods. | 739 |
| Experiment (With-AI) Diagnosis by radiologists informed by BoneAgeModel AI algorithm incorporated into normal radiologist workflows and considered as a factor in the clinical decision making process. | 792 |
| Total | 1,531 |
Baseline characteristics
| Characteristic | Control (Without-AI) | Experiment (With-AI) | Total |
|---|---|---|---|
| Age, Continuous | 11.8 years STANDARD_DEVIATION 3.6 | 11.5 years STANDARD_DEVIATION 3.6 | 11.7 years STANDARD_DEVIATION 3.6 |
| Age, Customized 0-4 years | 12 Participants | 14 Participants | 26 Participants |
| Age, Customized >12-16 years | 331 Participants | 338 Participants | 669 Participants |
| Age, Customized >16-20 years | 75 Participants | 57 Participants | 132 Participants |
| Age, Customized >20 years | 5 Participants | 3 Participants | 8 Participants |
| Age, Customized >4-8 years | 110 Participants | 131 Participants | 241 Participants |
| Age, Customized >8-12 years | 206 Participants | 249 Participants | 455 Participants |
| Clinical histories Congenital/syndrome | 9 Participants | 12 Participants | 21 Participants |
| Clinical histories Endocrine | 391 Participants | 430 Participants | 821 Participants |
| Clinical histories Medical | 16 Participants | 17 Participants | 33 Participants |
| Clinical histories More than one category | 20 Participants | 17 Participants | 37 Participants |
| Clinical histories Not available | 217 Participants | 241 Participants | 458 Participants |
| Clinical histories Orthopedic | 82 Participants | 63 Participants | 145 Participants |
| Clinical histories Other | 4 Participants | 12 Participants | 16 Participants |
| Race and Ethnicity Not Collected | — | — | 0 Participants |
| Region of Enrollment United States | 739 participants | 792 participants | 1531 participants |
| Sex: Female, Male Female | 338 Participants | 359 Participants | 697 Participants |
| Sex: Female, Male Male | 401 Participants | 433 Participants | 834 Participants |
| Skeletal age final impression (categorical) 0-4 years | 21 Participants | 17 Participants | 38 Participants |
| Skeletal age final impression (categorical) >12-16 years | 344 Participants | 362 Participants | 706 Participants |
| Skeletal age final impression (categorical) >16-20 years | 73 Participants | 59 Participants | 132 Participants |
| Skeletal age final impression (categorical) >20 years | 0 Participants | 0 Participants | 0 Participants |
| Skeletal age final impression (categorical) >4-8 years | 110 Participants | 131 Participants | 241 Participants |
| Skeletal age final impression (categorical) >8-12 years | 191 Participants | 223 Participants | 414 Participants |
| Skeletal age final impression (mean) | 11.6 years STANDARD_DEVIATION 3.6 | 11.4 years STANDARD_DEVIATION 3.5 | 11.5 years STANDARD_DEVIATION 3.5 |
Adverse events
| Event type | EG000 affected / at risk | EG001 affected / at risk |
|---|---|---|
| deaths Total, all-cause mortality | 0 / 939 | 0 / 964 |
| other Total, other adverse events | 0 / 939 | 0 / 964 |
| serious Total, serious adverse events | 0 / 939 | 0 / 964 |
Outcome results
Primary
Paired Difference of Skeletal Age Estimate
Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review.
Time frame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review
Population: Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity.~Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.
| Arm | Measure | Value (MEAN) |
|---|---|---|
| Control (Without-AI) | Paired Difference of Skeletal Age Estimate | 5.95 months |
| Experiment (With-AI) | Paired Difference of Skeletal Age Estimate | 5.36 months |
p-value: 0.0495% CI: [-1.14, -0.04]t-test, 2 sided
Secondary
Time for Diagnosis
Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not.
Time frame: Up to approximately 4 minutes
Population: Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity.~Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.
| Arm | Measure | Value (MEDIAN) |
|---|---|---|
| Control (Without-AI) | Time for Diagnosis | 142 seconds |
| Experiment (With-AI) | Time for Diagnosis | 102 seconds |
p-value: 0.001Wilcoxon rank-sum test