SPICED-ACS: Study of the potential impact of a computer-generated ECG diagnostic algorithmic certainty index in STEMI diagnosis: Towards transparent AI

Abstract

Background: Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known.

Purpose: To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy.

Methodology: Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence.

Results: A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75% of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95% CI 1.022-1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95% CI 0.923-1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95% CI 0.898-1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95% CI 0.391-0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation.

Conclusions: Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.

Keywords: Algorithm; Automated decision-making; Automation bias; Electrocardiogram; Ischaemic heart disease; Myocardial infarction.