DiagnosisNCT06774612
The Impact of Large Language Models on Diagnostic Reasoning Among LLM-Trained Medical Doctors
A study testing ChatGPT-4o in people with Diagnosis, run by Lahore University of Management Sciences at 1 site in Pakistan.
Trial summary
This study aims to evaluate whether large language model-trained medical doctors demonstrate enhanced diagnostic reasoning performance when utilizing ChatGPT-4o alongside conventional resources compared to using conventional resources alone. Read more about this study
- Who is it for?
- Any ageHealthy volunteers can take part
- Conditions the study is for
- Diagnosis
- What is being tested?
- ChatGPT-4o
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 60 participants
- Where
- 1 sitePakistan
- Study dates
- 10 Jan 2025to 17 May 2025
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- Healthy volunteers accepted
- Child, Adult, Older adult
Who may be able to join
- Full or Provisionally Registered Medical Practitioners with the Pakistan Medical and Dental Council (PMDC).
- Completed Bachelor of Medicine, Bachelor of Surgery (MBBS) Exam. The equivalent degree of MBBS in US and Canada is called Doctor of Medicine (MD).
- Participants must have completed a structured training program on the use of ChatGPT (or a comparable large language model), totaling at least 10 hours of instruction. The program must include hands-on practice related to LLM's aspects, specifically prompt engineering and content evaluation.
Who cannot join
- Any other Registered Medical Practitioners (Full or Provisional) with PMDC (e.g., Professionals with Bachelor of Dental Surgery or BDS).
Trial locations and contacts
1 site in 1 country.
Pakistan1 site
Lahore University of Management Sciences
Lahore, Punjab Province, 54792
About this study
Official title: Diagnostic Reasoning With and Without AI Support: A Randomized Controlled Trial of LLM-Trained Medical Doctors
This study aims to evaluate whether large language model-trained medical doctors demonstrate enhanced diagnostic reasoning performance when utilizing ChatGPT-4o alongside conventional resources compared to using conventional resources alone.
Read the detailed description
Diagnostic errors are a major source of preventable patient harm. Recent advances in Large Language Models (LLM), particularly ChatGPT-4o, have shown promise in enhancing medical decision-making. However, little is known about their impact on medical doctors' (e.g., physicians' and surgeons') diagnostic reasoning.
Diagnostic accuracy relies on complex clinical reasoning and careful evaluation of patient data. While AI assistance could potentially reduce errors and improve efficiency, ChatGPT-4o lacks medical validation and could introduce new risks through incorrect information generation (also known as hallucinations). To mitigate these risks, doctors need adequate training in understanding ChatGPT-4o's capabilities, limitations, and proper usage. Given these uncertainties and the importance of proper AI training, systematic evaluation is essential before clinical implementation.
This randomized study will assess whether ChatGPT-4o access improves LLM-trained medical doctors' diagnostic performance compared to conventional resources (e.g., textbooks, online medical databases) alone. All participating doctors will have completed at least a 10-hour training program covering ChatGPT-4o usage, prompt engineering techniques, and output evaluation strategies. Participants will provide differential diagnoses with supporting evidence and recommended next steps for clinical cases, with responses evaluated by blinded reviewers.
- Conditions
- Diagnosis
- Being tested
- ChatGPT-4o
- Keywords
- clinical reasoning, large language models, computer-assisted diagnosis
- Medical subject headings
- Disease
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
ChatGPT-4oOther
OpenAI's ChatGPT-4o large language model with chat interface.
| Study group | Role | What they receive |
|---|---|---|
| ChatGPT-4o | Active comparator | Group will be given access to ChatGPT-4o. Receives: Other: ChatGPT-4o |
| Conventional resources | No intervention | Group will not be given access to ChatGPT-4o but will be encouraged to use any resources they wish besides large language models (PubMed, Google without AI Overviews, etc). |
What is being measured
The primary outcome is the main question the study is designed to answer.
Diagnostic reasoning
The primary outcome will be the percent correct for each case (range: 0 to 100). For each case, participants will be asked for three top diagnoses, findings from the case that support that diagnosis, and findings from the case that oppose that diagnosis. For each plausible diagnosis, participants will receive 1 point. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.
Measured: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
Secondary outcomes (1)
Time Spent on Diagnosis
We will compare how much time (in seconds) participants spend per case between the two study arms.
Measured: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted14 Jan 2025
- Study start10 Jan 2025
- Primary completion17 May 2025
- Study completion17 May 2025
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Other trials for DiagnosisStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
- Masking
- None (open label)
- Enrolment
- 60 participants (actual)
Sponsor and investigators
- Lead sponsor
- Lahore University of Management Sciences
- Sponsor type
- University, hospital or other
- Collaborators
- King Edward Medical University
- Responsible party
- Principal investigator, Ihsan Ayyub Qazi, PhD, Lahore University of Management Sciences
- Investigators
- Ihsan Ayyub Qazi, PhD, Principal investigator, Lahore University of Management SciencesMuhammad Asadullah Khawaja, MBBS, Principal investigator, King Edward Medical UniversityAyesha Ali, PhD, Principal investigator, Lahore University of Management Sciences
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06774612
- Sponsor's protocol ID
- IRB-0342
- Status last verified
- Jul 2025
- First submitted
- 4 Jan 2025
- Last updated
- 17 Jul 2025
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Study at a glance
From the registry record, last updated by the sponsor on 17 Jul 2025. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Diagnostic Reasoning With and Without AI Support: A Randomized Controlled Trial of LLM-Trained Medical Doctors
- ClinicalTrials.gov ID
- NCT06774612
- Acronym / sponsor ID
- IRB-0342
- Conditions
- Diagnosis
- Keywords
- clinical reasoning, large language models, computer-assisted diagnosis
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 60 participants (actual)
- Lead sponsor
- Lahore University of Management SciencesUniversity, hospital or other
- Collaborators
- King Edward Medical University
- Study start
- 10 Jan 2025 (actual)
- Primary completion
- 17 May 2025 (actual)
- Study completion
- 17 May 2025 (actual)
- First posted
- 14 Jan 2025 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This study aims to evaluate whether large language model-trained medical doctors demonstrate enhanced diagnostic reasoning performance when utilizing ChatGPT-4o alongside conventional resources compared to using conventional resources alone.
Detailed description
Diagnostic errors are a major source of preventable patient harm. Recent advances in Large Language Models (LLM), particularly ChatGPT-4o, have shown promise in enhancing medical decision-making. However, little is known about their impact on medical doctors' (e.g., physicians' and surgeons') diagnostic reasoning.
Diagnostic accuracy relies on complex clinical reasoning and careful evaluation of patient data. While AI assistance could potentially reduce errors and improve efficiency, ChatGPT-4o lacks medical validation and could introduce new risks through incorrect information generation (also known as hallucinations). To mitigate these risks, doctors need adequate training in understanding ChatGPT-4o's capabilities, limitations, and proper usage. Given these uncertainties and the importance of proper AI training, systematic evaluation is essential before clinical implementation.
This randomized study will assess whether ChatGPT-4o access improves LLM-trained medical doctors' diagnostic performance compared to conventional resources (e.g., textbooks, online medical databases) alone. All participating doctors will have completed at least a 10-hour training program covering ChatGPT-4o usage, prompt engineering techniques, and output evaluation strategies. Participants will provide differential diagnoses with supporting evidence and recommended next steps for clinical cases, with responses evaluated by blinded reviewers.
Study design
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- None (open label)
- Number of arms
- 2
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
| Arm | Type | Receives |
|---|---|---|
| ChatGPT-4o | Active comparator | Other: ChatGPT-4oGroup will be given access to ChatGPT-4o. |
| Conventional resources | No intervention | –Group will not be given access to ChatGPT-4o but will be encouraged to use any resources they wish besides large language models (PubMed, Google without AI Overviews, etc). |
Who may qualify
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (3 listed)
- Full or Provisionally Registered Medical Practitioners with the Pakistan Medical and Dental Council (PMDC).
- Completed Bachelor of Medicine, Bachelor of Surgery (MBBS) Exam. The equivalent degree of MBBS in US and Canada is called Doctor of Medicine (MD).
- Participants must have completed a structured training program on the use of ChatGPT (or a comparable large language model), totaling at least 10 hours of instruction. The program must include hands-on practice related to LLM's aspects, specifically prompt engineering and content evaluation.
Key exclusion criteria (1 listed)
- Any other Registered Medical Practitioners (Full or Provisional) with PMDC (e.g., Professionals with Bachelor of Dental Surgery or BDS).
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| ChatGPT-4oOpenAI's ChatGPT-4o large language model with chat interface. | Other | ChatGPT-4o |
Primary outcome
Diagnostic reasoning
Time frame: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
The primary outcome will be the percent correct for each case (range: 0 to 100). For each case, participants will be asked for three top diagnoses, findings from the case that support that diagnosis, and findings from the case that oppose that diagnosis. For each plausible diagnosis, participants will receive 1 point. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.
1 secondary outcome
Time Spent on Diagnosis
Time frame: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 17 Jul 2025.
- Individual participant dataSharing planned: No
Related trials for Diagnosis
RecruitingPhase 1
Robot-Assisted Prostate Biopsy With a Novel Ultrasound Probe
Prostate Cancer (Diagnosis)
Recruiting
68Ga-XMFH01 PET/Fluorescence Imaging in Prostate Cancer
Prostate Cancer (Diagnosis), ACP3
Recruiting
STEpwise Research Program to Promote INGeniouS ONline Supportive Solutions in the Relief of Cancer-related Fatigue
Cancer Diagnosis, Moderate or Severe Score of Cancer-related Fatigue (CRF)
Active, not recruitingObservational study
A Multicenter Study in Bronchoscopy Combining Stimulated Raman Histology With Artificial Intelligence for Rapid Lung Cancer Detection - The ON-SITE Study
Lung Biopsy, Lung Cancer
Enrolling by invitationObservational study
Persona SoluTion PPS Femur PMCF
Rhematoid Arthritis, Osteo Arthritis of the Knee
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Ihsan Ayyub Qazi, PhD principal investigator
Lahore University of Management Sciences
Muhammad Asadullah Khawaja, MBBS principal investigator
King Edward Medical University
Ayesha Ali, PhD principal investigator
Lahore University of Management Sciences
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing ChatGPT-4o in people with Diagnosis, run by Lahore University of Management Sciences at 1 site in Pakistan.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 3 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 17 Jul 2025. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Pakistan. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Diagnostic reasoning (time frame: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 17 Jul 2025; AzTrials last refreshed its copy on 7 Nov 2026.