Training to Increase Rater Reliability When Assessing the Quality of Ethics Consultation Records with the Ethics Consultation Quality Assessment Tool (ECQAT)

Abstract

The Ethics Consultation Quality Assessment Tool (ECQAT) establishes standards by which the quality of ethics consultation records (ECRs) can be assessed. These standards relate to the ethics question, consultation-specific information, ethical analysis, and recommendations and/or conclusions, and result in a score associated with one of four levels of ethics consultation quality. For the ECQAT to be useful in assessing and improving the quality of healthcare ethics consultations, individuals who rate the quality of ECRs need to be able to reliably use the tool. We developed a short course to train ethics consultants in using the ECQAT, and evaluated whether the participants (1) achieved an acceptable level of calibration in matching expert-established quality scores for a set of ethics consultations, and (2) were satisfied with the course. We recruited 28 ethics consultants to participate in a virtual, six-session course. At each session participants and faculty reviewed, rated, and discussed one to two ECRs. The participants' calibration in matching expert-established quality scores improved with repeated exposure at all levels of ethics consultation quality. Participants were generally more accurate when assessing consultation quality at the dichotomous level of "acceptable" (scores of three or four) versus "unacceptable" (scores of one or two) than they were with more a specific score. Participants had higher rates of accuracy with the extreme ratings of "strong" (level four) or "poor" (level one). Although participants were highly satisfied with the course, only a minority of participants achieved the prespecified acceptable level of calibration (that is, 80 percent or greater accuracy between their score and expert-established scores). These results suggest that ECQAT training may require more sessions or need modification in the protocol to achieve higher reliability in scoring. Such trainings are an important next step in ensuring that the ECQAT is a tool that can be used to promote improvement in ethics consultation quality.