AzTrials
AzTrials

Artificial Intelligence (AI) in DiagnosisNCT07179861

Comparing Artificial Intelligence and Physicians: A Vignette-Based Study in Pediatric Clinical Decision-Making

An observational study of Artificial Intelligence (AI) in Diagnosis and Decision Support Systems, Clinical, run by Haseki Training and Research Hospital at 1 site in Turkey (Türkiye).

Completed

Finished. The study has ended and is not enrolling. Results may be posted on the official record.

Official record
Search all trials

Trial summary

This study evaluates how well anonymized artificial-intelligence (AI) tools perform on standardized pediatric case vignettes and whether showing AI suggestions can improve clinicians' answers. About 30 board-certified/eligible pediatric specialists at a single hospital complete a one-time session. Read more about this study

Who is it for?
Ages 28 to 40 yearsHealthy volunteers can take part
What is being tested?
AI Suggestions (Anonymized 5-tool panel) and Confidence Rating Task (1-10 Likert)
How many people
30 participants
Where
1 siteTurkey (Türkiye)
Study dates
27 Aug 2025to 11 Sep 2025
Phase
Observational
Are there results?
None posted in the registry

Key requirements

Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.

  • Ages 28 to 40 years
  • All sexes
  • Healthy volunteers accepted
  • Adult

Study population: Board-certified/eligible general pediatrics specialists recruited from SBÜ Sultangazi Haseki Training and Research Hospital network.

Who may be able to join

  • Board-certified or board-eligible pediatric specialist (general pediatrics) (in the first 10 years of expertise)
  • Actively practicing at the participating institution/network at the time of enrollment.
  • Able and willing to complete all vignette items individually in a single session and to follow study instructions for the assigned cohort (direct answers or confidence rating + viewing anonymized AI suggestions).
  • Fluent in Turkish and able to use a computer interface.
  • Provides written informed consent.

Who cannot join

  • Pediatric subspecialist practice as primary role (e.g., cardiology, infectious diseases, neurology, neonatology, etc.), to maintain a homogeneous general pediatrics cohort.
  • Prior access to or participation in creating the study vignettes, answer keys, or scoring rubrics; direct involvement with the study team.
  • Inability to complete the session without external help or use of non-protocol resources (internet/AI tools) during answering (outside of anonymized AI suggestions shown by the system in Group 2).
  • Failure to complete ≥90% of items or major protocol deviation (e.g., discussion with others during the task).
  • Any condition judged by investigators to interfere with valid participation (e.g., severe time constraints, inability to provide consent).

Trial locations and contacts

1 site in 1 country.

Turkey (Türkiye)1 site
  • SBÜ Sultangazi Haseki Training and Research Hospital

    Istanbul, Sultangazi, 34010

Other trials in Turkey (Türkiye)

About this study

Official title: A Prospective, Cross-Sectional, Vignette-Based Observational Study Comparing Clinical Decision-Making Performance of Pediatriciansand AI Models

This study evaluates how well anonymized artificial-intelligence (AI) tools perform on standardized pediatric case vignettes and whether showing AI suggestions can improve clinicians' answers. About 30 board-certified/eligible pediatric specialists at a single hospital complete a one-time session. Participants are randomized to two groups. Group A (n≈15): physicians answer each vignette once. Group B (n≈15): physicians answer and rate confidence (1-10), then review anonymized suggestions from five different AI tools (tool names not shown) and may keep or change their answer; changes and confidence are recorded.

Primary focus: measure AI performance (diagnostic accuracy, medication-dosing accuracy, interpretation accuracy) overall and by difficulty tier, and record AI response time. Secondary focus: quantify how AI suggestions affect human performance (change in accuracy, direction of change, confidence shift, and time). No patients or biospecimens are involved; risks are minimal (time and possible discomfort with performance review). Findings may inform safe, evidence-based ways to use AI alongside clinicians in pediatrics.

Being tested
AI Suggestions (Anonymized 5-tool panel), Confidence Rating Task (1-10 Likert)

What participants receive

What participants receive. In a randomised study each person is assigned to one group by chance.

  • AI Suggestions (Anonymized 5-tool panel)Other

    What: Display of AI-generated suggestions for each vignette, aggregated from five large language model tools (names not shown to participants).

    When/Who: Shown only in Group 2, after the physician's initial answer and confidence score.

    Purpose: Measure AI performance (primary) and quantify the effect of AI suggestions on physicians' answers (secondary).

    Applies to: Group 2.

  • Confidence Rating Task (1-10 Likert)Other

    What: Self-rated confidence for the initial answer on a 1-10 scale. When/Who: Group 2 before viewing AI suggestions. Purpose: Quantify confidence changes pre- vs post-AI and relate confidence to correctness.

    Applies to: Group 2.

Study groups in NCT07179861
Study groupRoleWhat they receive
Group/Cohort 1: Direct Answer (Physician-only)Board-certified/eligible pediatric specialists answer each standardized vignette once, without confidence scoring and without viewing AI suggestions. Outcomes captured: Diagnostic accuracy, dosing accuracy, interpretation accuracy, completion time.
Group/Cohort 2: Confidence + AI SuggestionsPediatric specialists first answer and rate confidence on a 1-10 scale; then they view anonymized suggestions from five distinct AI tools (tool names not shown) and may keep or revise their answer. All changes and confidence shifts are recorded. Outcomes captured: Pre- vs post-AI accuracy (and direction of change), dosing and interpretation accuracy changes, confidence shift, completion time.
Receives: Other: AI Suggestions (Anonymized 5-tool panel); Other: Confidence Rating Task (1-10 Likert)

What is being measured

The primary outcome is the main question the study is designed to answer.

  1. AI Interpretation Accuracy (%)

    Proportion of correct laboratory/imaging interpretations or appropriate next-test selections, per AI tool and pooled; stratified by difficulty tier. Unit: percent (0-100).

    Measured: Day 1

  2. AI Diagnostic Accuracy (%)

    Proportion of vignettes with a correct primary diagnosis produced by each anonymized AI tool and pooled across tools. Correctness is defined against a pre-specified reference answer key; results are also stratified by pre-defined difficulty tiers (easy/moderate/difficult/very difficult). Unit of measure: percent (0-100).

    Measured: Day 1

  3. AI Medication-Dosing Accuracy (%)

    Proportion of dose recommendations meeting pediatric standards (weight- or BSA-based ranges, route, frequency) per reference rubric, per AI tool and pooled; stratified by difficulty tier. Unit: percent (0-100).

    Measured: Day 1

Secondary outcomes (5)
  1. Change in Physician Diagnostic Accuracy (percentage points) (Group 2 only)

    Post-AI accuracy minus pre-AI accuracy per participant on the same case set; also categorized as beneficial (incorrect→correct), harmful (correct→incorrect), or no change. Accuracy is the proportion of cases with a correct final diagnosis according to a prespecified answer key.

    Measured: Day 1: Baseline (pre-AI) and immediate Post-AI within the same session (0-15 min after baseline).

  2. Confidence Shift (Δ on a 1-10 scale) (Group 2 only)

    Post-AI self-rated confidence minus pre-AI confidence; association with correctness is examined. Unit: scale points (-9 to +9).

    Measured: Day 1: Baseline (pre-AI) and immediate Post-AI within the same session (0-15 min after baseline).

  3. Answer-Change Frequency (%) (Group 2 only)

    Proportion of vignettes for which physicians revised their initial answer after AI suggestions; reported overall and by difficulty tier. Unit: percent (0-100).

    Measured: Day 1

  4. AI Response Time (seconds per vignette)

    Time from vignette display to final AI output, reported per tool and pooled; also by difficulty tier. Unit: seconds.

    Measured: Day 1

  5. Net Benefit Index of AI Exposure (percentage points) (Group 2 only)

    Beneficial change rate (incorrect→correct) minus harmful change rate (correct→incorrect) for diagnostic items; sensitivity analyses for dosing/interpretation. Unit: percentage points.

    Measured: Day 1

Timeline

From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.

  1. First posted18 Sep 2025
  2. Study start27 Aug 2025
  3. Primary completion10 Sep 2025
  4. Study completion11 Sep 2025

Resources

More details

For doctors and researchers: how the study is designed, who runs it, and its registry IDs.

Study design
Study type
Observational
Observational model
Cohort
Time perspective
Cross-sectional
Enrolment
30 participants (actual)
IDs, dates and publications
ClinicalTrials.gov ID
NCT07179861
Sponsor's protocol ID
140-2025
Status last verified
Sep 2025
First submitted
11 Sep 2025
Last updated
23 Sep 2025
Results posted
No
Data monitoring committee
No
FDA-regulated drug
No
FDA-regulated device
No
Individual data sharing
No: IPD Description: No IPD will be shared. The dataset comprises detailed, item-level responses from a small, single-center cohort of pediatric specialists. Despite de-identification, the risk of re-identification is non-trivial given granular performance metrics and professional identifiers. The informed consent did not include permission to share raw individual responses outside the study team. Plan to Share Supporting Materials: Yes Supporting Documents: Study Protocol, Statistical Analysis Plan, scoring rubrics, redacted vignette templates, and analysis code. Time Frame: Available within 6 months after the primary manuscript is published and for at least 36 months thereafter. Access Criteria and URL: Materials will be provided upon reasonable request to the Principal Investigator (email: drberkerokay@gmail.com). A data use agreement will be required; use is limited to non-commercial research and aggregate reporting.

Publications

  • Su H, Sun Y, Li R, Zhang A, Yang Y, Xiao F, Duan Z, Chen J, Hu Q, Yang T, Xu B, Zhang Q, Zhao J, Li Y, Li H. Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis. J Med Internet Res. 2025 Jun 9;27:e72062. doi: 10.2196/72062.

    PubMed 40489764
  • Bicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, Spaedy O, Skelton A, Edupuganti N, Dzubinski L, Tate H, Dyess G, Lindeman B, Lehmann LS. ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Med Educ. 2024 Nov 6;10:e63430. doi: 10.2196/63430.

    PubMed 39504445
  • Cross JL, Choma MA, Onofrey JA. Bias in medical AI: Implications for clinical decision-making. PLOS Digit Health. 2024 Nov 7;3(11):e0000651. doi: 10.1371/journal.pdig.0000651. eCollection 2024 Nov.

    PubMed 39509461
  • Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ; SPIRIT-AI and CONSORT-AI Working Group; SPIRIT-AI and CONSORT-AI Steering Group; SPIRIT-AI and CONSORT-AI Consensus Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020 Sep;26(9):1351-1363. doi: 10.1038/s41591-020-1037-7. Epub 2020 Sep 9.

    PubMed 32908284
  • Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P; DECIDE-AI expert group. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022 May 18;377:e070904. doi: 10.1136/bmj-2022-070904.

    PubMed 35584845

Questions about this trial

What is this clinical trial studying?

An observational study of Artificial Intelligence (AI) in Diagnosis and Decision Support Systems, Clinical, run by Haseki Training and Research Hospital at 1 site in Turkey (Türkiye).

The full registered description is in the About this study section and the official record.

Who may be eligible to take part?

The sponsor lists 5 inclusion and 5 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.

Is this clinical trial currently recruiting?

The record says: finished, as of the sponsor's last update on 23 Sep 2025. Status can change between updates, so confirm with the study contact before making plans.

Where is the study taking place?

The record lists 1 site in Turkey (Türkiye). Whether a site is taking part right now should be confirmed with that site. See every site.

What are the possible benefits and risks?

They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.

Are results available for this study?

The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.

How can a doctor refer a patient or contact the research team?

The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.

What does the primary outcome measure?

AI Interpretation Accuracy (%) (time frame: Day 1). The study lists 3 primary outcomes in all. It is what the study was designed to measure, not a finding.

Where can I read the original study information?

On the ClinicalTrials.gov record for NCT07179861, which also lists 5 linked publications.

Does a listing mean the treatment is approved or proven to work?

No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.

How often is this information updated?

AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 23 Sep 2025.

Free account

Keep searching with a free account

You’ve used your 3 free searches for this session. Log in or create a free AzTrials account to keep searching clinical trials and save the ones you want to follow.

Create a free accountLog in