Risk Stratification of Potential Drug Interactions Involving Common Over-the-Counter Medications and Herbal Supplements by a Large Language Model

Abstract

Background: As polypharmacy, the use of over-the-counter (OTC) drugs, and herbal supplements become increasingly prevalent, the potential for adverse drug-drug interactions (DDIs) poses significant challenges to patient safety and healthcare outcomes.

Objectives: This study evaluates the capacity of Generative Pre-trained Transformer (GPT) models to accurately assess DDIs involving prescription drugs (Rx) with OTC medications and herbal supplements.

Methods: Leveraging a popular subscription-based tool (Lexicomp®), we compared the risk ratings assigned by these models to 43 Rx-OTC and 30 Rx-herbal supplement pairs.

Results: Our findings reveal that all models generally underperform, with accuracies below 50% and poor agreement with Lexicomp standards as measured by Cohen's kappa. Notably, GPT-4 and GPT-4o demonstrated a modest improvement in identifying higher-risk interactions compared to GPT-3.5.

Conclusion: These results highlight the challenges and limitations of using off-the-shelf Large Language Models (LLMs) for guidance in DDI assessment.

Keywords: Drug-Drug Interactions; Generative Pre-trained Transformer; Large Language Models.