An evaluation of existing text de-identification tools for use with patient progress notes from Australian general practice

Carol El-Hayek; Siamak Barzegar; Noel Faux; Kim Doyle; Priyanka Pillai; Simon J Mutch; Alaina Vaisey; Roger Ward; Lena Sanci; Adam G Dunn; Margaret E Hellard; Jane S Hocking; Karin Verspoor; Douglas Ir Boyle

doi:10.1016/j.ijmedinf.2023.105021

An evaluation of existing text de-identification tools for use with patient progress notes from Australian general practice

Int J Med Inform. 2023 May:173:105021. doi: 10.1016/j.ijmedinf.2023.105021. Epub 2023 Feb 11.

Authors

Carol El-Hayek¹, Siamak Barzegar², Noel Faux³, Kim Doyle⁴, Priyanka Pillai⁵, Simon J Mutch⁴, Alaina Vaisey⁶, Roger Ward⁷, Lena Sanci⁷, Adam G Dunn⁸, Margaret E Hellard⁹, Jane S Hocking⁶, Karin Verspoor¹⁰, Douglas Ir Boyle⁷

Affiliations

¹ Burnet Institute, Melbourne, Australia; Melbourne School of Population and Global Health, University of Melbourne, Australia; School of Public Health and Preventive Medicine, Monash University, Australia. Electronic address: carol.el-hayek@burnet.edu.au.
² School of Computing and Information Systems, University of Melbourne, Australia.
³ Melbourne Data Analytics Platform, University of Melbourne, Australia; Florey Institute of Neuroscience and Mental Health, University of Melbourne, Australia.
⁴ Melbourne Data Analytics Platform, University of Melbourne, Australia.
⁵ Melbourne Data Analytics Platform, University of Melbourne, Australia; The Peter Doherty Institute for Infection and Immunity, Melbourne, Australia.
⁶ Melbourne School of Population and Global Health, University of Melbourne, Australia.
⁷ Department of General Practice and Primary Care, University of Melbourne, Australia.
⁸ School of Medical Sciences, University of Sydney, Australia.
⁹ Burnet Institute, Melbourne, Australia; Melbourne School of Population and Global Health, University of Melbourne, Australia; School of Public Health and Preventive Medicine, Monash University, Australia; The Peter Doherty Institute for Infection and Immunity, Melbourne, Australia.
¹⁰ School of Computing and Information Systems, University of Melbourne, Australia; School of Computing Technologies, RMIT University, Melbourne, Australia.

PMID: 36870249
DOI: 10.1016/j.ijmedinf.2023.105021

Abstract

Introduction: Digitized patient progress notes from general practice represent a significant resource for clinical and public health research but cannot feasibly and ethically be used for these purposes without automated de-identification. Internationally, several open-source natural language processing tools have been developed, however, given wide variations in clinical documentation practices, these cannot be utilized without appropriate review. We evaluated the performance of four de-identification tools and assessed their suitability for customization to Australian general practice progress notes.

Methods: Four tools were selected: three rule-based (HMS Scrubber, MIT De-id, Philter) and one machine learning (MIST). 300 patient progress notes from three general practice clinics were manually annotated with personally identifying information. We conducted a pairwise comparison between the manual annotations and patient identifiers automatically detected by each tool, measuring recall (sensitivity), precision (positive predictive value), f1-score (harmonic mean of precision and recall), and f2-score (weighs recall 2x higher than precision). Error analysis was also conducted to better understand each tool's structure and performance.

Results: Manual annotation detected 701 identifiers in seven categories. The rule-based tools detected identifiers in six categories and MIST in three. Philter achieved the highest aggregate recall (67%) and the highest recall for NAME (87%). HMS Scrubber achieved the highest recall for DATE (94%) and all tools performed poorly on LOCATION. MIST achieved the highest precision for NAME and DATE while also achieving similar recall to the rule-based tools for DATE and highest recall for LOCATION. Philter had the lowest aggregate precision (37%), however preliminary adjustments of its rules and dictionaries showed a substantial reduction in false positives.

Conclusion: Existing off-the-shelf solutions for automated de-identification of clinical text are not immediately suitable for our context without modification. Philter is the most promising candidate due to its high recall and flexibility however will require extensive revising of its pattern matching rules and dictionaries.

Keywords: De-identification; Electronic medical records; General practice; Patient privacy.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Australia
Confidentiality
Data Anonymization
Electronic Health Records*
General Practice*
Humans
Natural Language Processing