A Retrospective External Validation of the Cleveland Clinic Malignancy Probability Prediction Model for Indeterminate Pulmonary Nodules

Michal M Reid; Jack J Amja; Irene T Riestra Guiance; Rupesh R Andani; Robert A Vierkant; Amit Goyal; Janani S Reisenauer

doi:10.1016/j.mayocpiqo.2024.05.005

A Retrospective External Validation of the Cleveland Clinic Malignancy Probability Prediction Model for Indeterminate Pulmonary Nodules

Mayo Clin Proc Innov Qual Outcomes. 2024 Jul 3;8(4):375-383. doi: 10.1016/j.mayocpiqo.2024.05.005. eCollection 2024 Aug.

Authors

Michal M Reid^{1

2}, Jack J Amja^{3

4}, Irene T Riestra Guiance¹, Rupesh R Andani¹, Robert A Vierkant⁵, Amit Goyal³, Janani S Reisenauer^{1

6}

Affiliations

¹ Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, MN.
² Division of Pulmonary, Critical Care, and Sleep Medicine, University of Kansas Medical Center, Kansas City, KS.
³ Division of Pulmonary and Critical Care Medicine, Loyola University Medical Center, Maywood, IL.
⁴ Division of Pulmonary, Critical Care, and Sleep Medicine, Hartford Healthcare Medical Group, Hartford, CT.
⁵ Division of Clinical Trials and Biostatistics, Mayo Clinic, Rochester, MN.
⁶ Division of Thoracic Surgery, Mayo Clinic, Rochester, MN.

Abstract

Objective: To perform a retrospective, multicenter, external validation of the Cleveland Clinic malignancy probability prediction model for incidental pulmonary nodules.

Patients and methods: From July 1, 2022, to May 31, 2023, we identified 296 patients who underwent tissue acquisition at Mayo Clinic (MC) (n=198) and Loyola University Medical Center (n=98) with histopathology indicating malignant (n=195) or benign (n=101). Data was collected at initial radiographic identification (point 1) and at the time of intervention (point 2). Point 3 represented the most recent data. The areas under the receiver operating characteristics were calculated for each model per time point. Calibration was evaluated by comparing the predicted and observed rates of malignancy.

Results: The areas under the receiver operating characteristics at time points 1, 2, and 3 for the MC model were 0.67 (95% CI, 0.61-0.74), 0.67 (95% CI, 0.58-0.77), and 0.70 (95% CI, 0.63-0.76), respectively. The Cleveland Clinic model (CCM) was 0.68 (95% CI, 0.61-0.74), 0.75 (95% CI, 0.65-0.84), and 0.72 (95% CI, 0.66-0.78), respectively. The mean ± SD estimated probability for malignant pulmonary nodules (PNs) at time points 1, 2, and 3 for the CCM was 64.2±25.9, 65.8±24.0, and 64.7±24.4, which resembled the overall proportion of malignant PNs (66%). The mean estimated probability of malignancy for the MC model at each time point was 38.3±27.4, 36.2±24.4, and 42.1±27.3, substantially lower than the observed proportion of malignancies.

Conclusion: The CCM found discrimination similar to its internal validation and good calibration. The CCM can be used to augment clinical and shared decision-making when evaluating high-risk PNs.