Chasing sleep physicians: ChatGPT-4o on the interpretation of polysomnographic results

Christopher Seifen; Tilman Huppertz; Haralampos Gouveris; Katharina Bahr-Hamm; Johannes Pordzik; Jonas Eckrich; Harry Smith; Tom Kelsey; Andrew Blaikie; Christoph Matthias; Sebastian Kuhn; Christoph Raphael Buhr

doi:10.1007/s00405-024-08985-3

Chasing sleep physicians: ChatGPT-4o on the interpretation of polysomnographic results

Eur Arch Otorhinolaryngol. 2024 Oct 20. doi: 10.1007/s00405-024-08985-3. Online ahead of print.

Affiliations

¹ Sleep Medicine Center & Department of Otolaryngology, Head and Neck Surgery, University Medical Center Mainz, Mainz, Germany.
² School of Computer Science, University of St Andrews, St Andrews, UK.
³ School of Medicine, University of St Andrews, St Andrews, UK.
⁴ Institute for Digital Medicine, University Hospital of Giessen and Marburg, Philipps-University Marburg, Marburg, Germany.
⁵ Sleep Medicine Center & Department of Otolaryngology, Head and Neck Surgery, University Medical Center Mainz, Mainz, Germany. buhrchri@uni-mainz.de.
⁶ School of Medicine, University of St Andrews, St Andrews, UK. buhrchri@uni-mainz.de.

^# Contributed equally.

PMID: 39427271
DOI: 10.1007/s00405-024-08985-3

Abstract

Background: From a healthcare professional's perspective, the use of ChatGPT (Open AI), a large language model (LLM), offers huge potential as a practical and economic digital assistant. However, ChatGPT has not yet been evaluated for the interpretation of polysomnographic results in patients with suspected obstructive sleep apnea (OSA).

Aims/objectives: To evaluate the agreement of polysomnographic result interpretation between ChatGPT-4o and a board-certified sleep physician and to shed light into the role of ChatGPT-4o in the field of medical decision-making in sleep medicine.

Material and methods: For this proof-of-concept study, 40 comprehensive patient profiles were designed, which represent a broad and typical spectrum of cases, ensuring a balanced distribution of demographics and clinical characteristics. After various prompts were tested, one prompt was used for initial diagnosis of OSA and a further for patients with positive airway pressure (PAP) therapy intolerance. Each polysomnographic result was independently evaluated by ChatGPT-4o and a board-certified sleep physician. Diagnosis and therapy suggestions were analyzed for agreement.

Results: ChatGPT-4o and the sleep physician showed 97% (29/30) concordance in the diagnosis of the simple cases. For the same cases the two assessment instances unveiled 100% (30/30) concordance regarding therapy suggestions. For cases with intolerance of treatment with positive airway pressure (PAP) ChatGPT-4o and the sleep physician revealed 70% (7/10) concordance in the diagnosis and 44% (22/50) concordance for therapy suggestions.

Conclusion and significance: Precise prompting improves the output of ChatGPT-4o and provides sleep physician-like polysomnographic result interpretation. Although ChatGPT shows some shortcomings in offering treatment advice, our results provide evidence for AI assisted automation and economization of polysomnographic interpretation by LLMs. Further research should explore data protection issues and demonstrate reproducibility with real patient data on a larger scale.

Keywords: Artificial intelligence; ChatGPT; ChatGPT-4o; Digital health; Large language models; OSA; Obstructive sleep apnea; PSG; Polysomnography; Sleep medicine.