Bone AgeNCT03530098

Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment

A study testing BoneAgeModel in people with Bone Age, run by Stanford University at 6 sites in United States.

Completed

Finished. The study has ended and is not enrolling. Results may be posted on the official record.

Official record

Search all trials

Trial summary

Who is it for?: Any age
Conditions the study is for: Bone Age
What is being tested?: BoneAgeModel
Type of treatment: Medical device
Could I receive a placebo?: No, everyone receives an active treatment
How many people: 1,903 participants
Where: 6 sitesUnited States
Study dates: 12 Jul 2018to 31 Aug 2019
Phase: Interventional
Are there results?: Yes, posted in the registrySummary in the doctors' view

Key requirements

Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.

Ages Any age
All sexes
No healthy volunteers
Child, Adult, Older adult

Exams that meet the following inclusion criteria will be included: (1) exams read by radiologists who interpret pediatric skeletal age exams and verbally consent to participate (2) exams that contain a procedure code or study description indicative of a skeletal age exam.

Exams containing more than one radiograph will not be included. Exams for which a trainee provides a preliminary interpretation will be excluded. No further exclusion criteria will be applied on the basis of image quality metrics or manufacturers. No exclusion criteria will be applied on the basis of patient chronological age.

Trial locations and contacts

6 sites in 1 country.

City, postcode or hospital

6 sites in 1 country

United States6 sites

Boston Children's Hospital
Boston, Massachusetts, 02115
Cincinnati Children's Hospital Medical Center
Cincinnati, Ohio, 45229
Yale New Haven Hospital
New Haven, Connecticut, 06519
New York University
New York, New York, 10016
Children's Hospital of Philadelphia
Philadelphia, Pennsylvania, 19104
Stanford University
Stanford, California, 94305

Other trials in United States

About this study

Official title: Prospective, Multi-Center, Randomized Controlled Trial for Skeletal Age Assessment AI Model

The purpose of this study is to understand the effects of using an Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make an estimation of the bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.

Read the detailed description

The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the radiographic images and health records. As part of this study, the radiologists will estimate skeletal age from radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age.

This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard ("gold standard") composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD). The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups.

The investigators have recruited and analyzed data from a sample size of 1600 exams. Patients getting these exams will not undergo any research procedures that deviate from the current standard practices.

Conditions: Bone Age
Being tested: BoneAgeModel
Keywords: Age Determination by Skeleton, Machine Learning, Deep Learning, Artificial Intelligence, Prospective, Clinical Validation

What participants receive

What participants receive. In a randomised study each person is assigned to one group by chance.

BoneAgeModelDevice
BoneAgeModel is an Artificial Intelligence tool that takes in a hand radiograph and gender, and outputs the skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final.

Study groups in NCT03530098
Study group	Role	What they receive
Control (Without-AI)	No intervention	This is the control arm where no intervention is provided; represents current standard of care.
Experiment (With-AI)	Experimental	This is the experiment arm where the intervention, "BoneAgeModel", is provided. The participating radiologists in this arm will receive the output of the Artificial Intelligence algorithm. They will be asked to incorporate this new information with their normal workflows to make a diagnosis. The radiologists' diagnosis will be considered final. Receives: Device: BoneAgeModel

What is being measured

The primary outcome is the main question the study is designed to answer.

Paired Difference of Skeletal Age Estimate
Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review.
Measured: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review

Secondary outcomes (1)

Time for Diagnosis
Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not.
Measured: Up to approximately 4 minutes

Timeline

From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.

12 Jul 201831 Aug 2019

First posted21 May 2018
Study start12 Jul 2018
Primary completion31 Aug 2019
Study completion31 Aug 2019

Resources

Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
Posted resultsParticipant flow, outcome measures and adverse events reported by the sponsor.
Other trials for Bone AgeStudies of the same condition, open and under way.
More trials run by Stanford UniversityThe sponsor's other registered studies.
What the trial phases meanHow a treatment moves from first tests in people to approval.

More details

For doctors and researchers: how the study is designed, who runs it, and its registry IDs.

Study design

Study type: Interventional
Phase: Phase N/A
Primary purpose: Diagnosis
Allocation: Randomised
Intervention model: Parallel groups
Model description: A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the exam using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their skeletal age estimation. In all cases, the decision of the radiologist will be considered final.
Masking: None (open label)
Enrolment: 1,903 participants (actual)

IDs, dates and publications

ClinicalTrials.gov ID: NCT03530098
Sponsor's protocol ID: IRB #44764
Status last verified: May 2021
First submitted: 7 May 2018
Last updated: 9 Jun 2021
Results posted: Yes, on ClinicalTrials.gov
FDA-regulated drug: No
FDA-regulated device: Yes
Individual data sharing: Yes: Individual participant data that underlie the results reported in this article after deidentification (text, tables, figures and appendices).

Want to know more?

The sponsor has not listed a contact. The official record may have more details.

Talk to your doctor first and take this page with you. Only the study team can confirm whether you can join.

Questions to ask the study team

What is the study trying to find out?
What would I need to do, and how often would I visit?
Could I receive a placebo instead of the treatment?
What are the possible risks and benefits for me?
Will it cost me anything? Are travel or time paid for?
Can I keep seeing my own doctor during the study?
What happens when the study ends?

See locations

RecruitingPhase 3
A Study of MACI in Patients Aged 17 to 65 Years With Symptomatic Chondral or Osteochondral Defects of the Ankle
Chondral Defect, Articular Cartilage Defect
RecruitingPhase 1/2
Pacritinib, a Kinase Inhibitor of CSF1R, IRAK1, JAK2, and FLT3, in Adults and Pediatric Participants 12 Years of Age or Older With Myelodysplastic Syndromes or Myelodysplastic/Myeloproliferative Neoplasms
Myelodysplastic Syndromes
RecruitingPhase 2
Testing New Therapies for Patients With Acute Myeloid Leukemia Who Are Newly Diagnosed and Have Not Yet Started Treatment (A MyeloMATCH Treatment Trial)
Acute Myeloid Leukemia, Acute Myeloid Leukemia Arising From Previous Myelodysplastic/Myeloproliferative Neoplasm
Active, not recruitingPhase 2
Azacitidine and Gemtuzumab Ozogamicin in Treating Older Patients With Previously Untreated Acute Myeloid Leukemia
Acute Myeloid Leukemia, Adult Acute Megakaryoblastic Leukemia
Enrolling by invitationObservational study
Multidimensional Determinants of Functional Health In Women Aged 45-60
Healthy, Perimenopausal Depression

All Bone Age trials

Study at a glance

From the registry record, last updated by the sponsor on 9 Jun 2021. Recruitment status is as the sponsor reported it: confirm with the study team.

Official title: Prospective, Multi-Center, Randomized Controlled Trial for Skeletal Age Assessment AI Model
ClinicalTrials.gov ID: NCT03530098
Acronym / sponsor ID: IRB #44764
Conditions: Bone Age
Keywords: Age Determination by Skeleton, Machine Learning, Deep Learning, Artificial Intelligence, Prospective, Clinical Validation
Study type: Interventional
Recruitment status: Completed
Enrolment: 1,903 participants (actual)
Lead sponsor: Stanford UniversityUniversity, hospital or other
Study start: 12 Jul 2018 (actual)
Primary completion: 31 Aug 2019 (actual)
Study completion: 31 Aug 2019 (actual)
First posted: 21 May 2018 (actual)
Results: Posted 26 Oct 2020

Clinical summary

The research question as the sponsor registered it. This describes what the study plans to test, not what it found.

Detailed description

Study design

Primary purpose: Diagnosis
Allocation: Randomised
Intervention model: Parallel groups
Masking: None (open label)
Number of arms: 2
Model description: A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the exam using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their skeletal age estimation. In all cases, the decision of the radiologist will be considered final.

Arm	Type	Receives
Control (Without-AI)	No intervention	–This is the control arm where no intervention is provided; represents current standard of care.
Experiment (With-AI)	Experimental	Device: BoneAgeModelThis is the experiment arm where the intervention, "BoneAgeModel", is provided. The participating radiologists in this arm will receive the output of the Artificial Intelligence algorithm. They will be asked to incorporate this new information with their normal workflows to make a diagnosis. The radiologists' diagnosis will be considered final.

Who may qualify

Sex: All
Healthy volunteers: Not accepted
Age groups: Child, Adult, Older adult

Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.

Interventions

Intervention	Type	Arms
BoneAgeModelBoneAgeModel is an Artificial Intelligence tool that takes in a hand radiograph and gender, and outputs the skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final.	Device	Experiment (With-AI)

Primary outcome

Paired Difference of Skeletal Age Estimate
Time frame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review
Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review.

1 secondary outcome

Time for Diagnosis
Time frame: Up to approximately 4 minutes

Results and safety

Results posted to the registry on 26 Oct 2020, as reported by the sponsor. Figures are copied from the record, not re-analysed. Full results on ClinicalTrials.gov.

Participant flow

Group	Started	Completed
Control (Without-AI)	939	939
Experiment (With-AI)	964	964

Primary outcome: Paired Difference of Skeletal Age Estimate

Time frame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review. Mean (95% Confidence Interval). Unit: months

Measure	Control (Without-AI) n = 739	Experiment (With-AI) n = 792
Value	5.95 (5.53 to 6.37)	5.36 (5.01 to 5.71)

Control (Without-AI) vs Experiment (With-AI): Odds Ratio (OR): -0.59 (95% CI -1.14 to -0.04), p = 0.04, t-test, 2 sided

Population analysed: Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity. Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.

1 secondary outcome is reported in the full results.

Adverse events

Time frame: Day of study visit (average approximately 1 hour). Other adverse events reported at 0% frequency or more

Group	Serious adverse events	Other adverse events	Deaths
Control (Without-AI)	0 of 939 (0%)	0 of 939 (0%)	0 of 939
Experiment (With-AI)	0 of 964 (0%)	0 of 964 (0%)	0 of 964

Loading the posted results from ClinicalTrials.gov…

Research links

Official record on ClinicalTrials.govLast updated by the sponsor 9 Jun 2021.
Posted resultsParticipant flow, baseline, all outcome measures and adverse events.
Individual participant dataSharing planned: Yes. Individual participant data that underlie the results reported in this article after deidentification (text, tables, figures and appendices).

Related trials for Bone Age

RecruitingPhase 3
A Study of MACI in Patients Aged 17 to 65 Years With Symptomatic Chondral or Osteochondral Defects of the Ankle
Chondral Defect, Articular Cartilage Defect
RecruitingPhase 1/2
Pacritinib, a Kinase Inhibitor of CSF1R, IRAK1, JAK2, and FLT3, in Adults and Pediatric Participants 12 Years of Age or Older With Myelodysplastic Syndromes or Myelodysplastic/Myeloproliferative Neoplasms
Myelodysplastic Syndromes
RecruitingPhase 2
Testing New Therapies for Patients With Acute Myeloid Leukemia Who Are Newly Diagnosed and Have Not Yet Started Treatment (A MyeloMATCH Treatment Trial)
Acute Myeloid Leukemia, Acute Myeloid Leukemia Arising From Previous Myelodysplastic/Myeloproliferative Neoplasm
Active, not recruitingPhase 2
Azacitidine and Gemtuzumab Ozogamicin in Treating Older Patients With Previously Untreated Acute Myeloid Leukemia
Acute Myeloid Leukemia, Adult Acute Megakaryoblastic Leukemia
Enrolling by invitationObservational study
Multidimensional Determinants of Functional Health In Women Aged 45-60
Healthy, Perimenopausal Depression

Referral and contact

AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.

Curtis Langlotz, M.D. Ph.D. study chair

Stanford University

David Eng, B.S. study director

Stanford University

Nishith Khandwala, B.S. study director

Stanford University

6 sites in 1 country: see every site and its status.

Questions about this trial

What is this clinical trial studying?

A study testing BoneAgeModel in people with Bone Age, run by Stanford University at 6 sites in United States.

The full registered description is in the About this study section and the official record.

Is this clinical trial currently recruiting?

The record says: finished, as of the sponsor's last update on 9 Jun 2021. Status can change between updates, so confirm with the study contact before making plans.

Where is the study taking place?

The record lists 6 sites in United States. Whether a site is taking part right now should be confirmed with that site. See every site.

What are the possible benefits and risks?

They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.

Are results available for this study?

Yes. Results are posted in the registry (first posted 26 Oct 2020). A summary is in the For doctors view, and the full results are on ClinicalTrials.gov.

How can a doctor refer a patient or contact the research team?

The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.

What does the primary outcome measure?

Paired Difference of Skeletal Age Estimate (time frame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review). It is what the study was designed to measure, not a finding.

Where can I read the original study information?

On the ClinicalTrials.gov record for NCT03530098.

Does a listing mean the treatment is approved or proven to work?

No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.

How often is this information updated?

AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 9 Jun 2021.