Background: Studies have provided little evidence that critical appraisal skills improve with focused courses. However, outcome measures in these studies have been questionable. The goal of this study was to develop a feasible, reliable, and valid assessment of critical appraisal skills.
Methods: Forty-four surgery residents read three articles and then responded to short answer questions and provided 7-point ratings regarding various methodological aspects of each article. Reliability and validity of the examination were assessed.
Results: The mean score was 52.4% (SD 8.6%). Internal consistency of the 55-question examination was 0.77. Interrater reliability of clinician markers was 0.91. Mean score for residents with more intensive critical appraisal training was significantly higher than for those with little or no training (56.6% versus 49.3%, t(35) = 2.31, P = 0.02), suggesting construct validity.
Conclusions: This examination has promising psychometric properties, and may be useful in evaluating critical appraisal curricula.