Comprehensive multimodal deep learning survival prediction enabled by a transformer architecture: A multicenter study in glioblastoma

Ahmed Gomaa; Yixing Huang; Amr Hagag; Charlotte Schmitter; Daniel Höfler; Thomas Weissmann; Katharina Breininger; Manuel Schmidt; Jenny Stritzelberger; Daniel Delev; Roland Coras; Arnd Dörfler; Oliver Schnell; Benjamin Frey; Udo S Gaipl; Sabine Semrau; Christoph Bert; Peter Hau; Rainer Fietkau; Florian Putz

doi:10.1093/noajnl/vdae122

Comprehensive multimodal deep learning survival prediction enabled by a transformer architecture: A multicenter study in glioblastoma

Neurooncol Adv. 2024 Jul 11;6(1):vdae122. doi: 10.1093/noajnl/vdae122. eCollection 2024 Jan-Dec.

Authors

Ahmed Gomaa^{1

2

3}, Yixing Huang^{1

2

3}, Amr Hagag^{1

2}, Charlotte Schmitter^{1

2}, Daniel Höfler^{1

2}, Thomas Weissmann^{1

2}, Katharina Breininger⁴, Manuel Schmidt^{2

5

3}, Jenny Stritzelberger^{2

6}, Daniel Delev^{2

7}, Roland Coras^{2

8}, Arnd Dörfler^{2

5}, Oliver Schnell^{2

7}, Benjamin Frey^{1

2}, Udo S Gaipl^{1

2

9}, Sabine Semrau^{1

2}, Christoph Bert^{1

2}, Peter Hau³, Rainer Fietkau^{1

2}, Florian Putz^{1

2

3}

Affiliations

¹ Department of Radiation Oncology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
² Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), Erlangen, Germany.
³ The Bavarian Cancer Research Center (BZKF), Erlangen, Germany.
⁴ Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
⁵ Institute of Neuroradiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
⁶ Department of Neurology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
⁷ Department of Neurosurgery, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
⁸ Institute for Neuropathology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
⁹ FAU Profile Center Immunomedicine (FAU I-MED), Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Erlangen, Germany.

Abstract

Background: This research aims to improve glioblastoma survival prediction by integrating MR images, clinical, and molecular-pathologic data in a transformer-based deep learning model, addressing data heterogeneity and performance generalizability.

Methods: We propose and evaluate a transformer-based nonlinear and nonproportional survival prediction model. The model employs self-supervised learning techniques to effectively encode the high-dimensional MRI input for integration with nonimaging data using cross-attention. To demonstrate model generalizability, the model is assessed with the time-dependent concordance index (Cdt) in 2 training setups using 3 independent public test sets: UPenn-GBM, UCSF-PDGM, and Rio Hortega University Hospital (RHUH)-GBM, each comprising 378, 366, and 36 cases, respectively.

Results: The proposed transformer model achieved a promising performance for imaging as well as nonimaging data, effectively integrating both modalities for enhanced performance (UCSF-PDGM test-set, imaging Cdt 0.578, multimodal Cdt 0.672) while outperforming state-of-the-art late-fusion 3D-CNN-based models. Consistent performance was observed across the 3 independent multicenter test sets with Cdt values of 0.707 (UPenn-GBM, internal test set), 0.672 (UCSF-PDGM, first external test set), and 0.618 (RHUH-GBM, second external test set). The model achieved significant discrimination between patients with favorable and unfavorable survival for all 3 datasets (log-rank P 1.9 × 10^-8, 9.7 × 10^-3, and 1.2 × 10^-2). Comparable results were obtained in the second setup using UCSF-PDGM for training/internal testing and UPenn-GBM and RHUH-GBM for external testing (Cdt 0.670, 0.638, and 0.621).

Conclusions: The proposed transformer-based survival prediction model integrates complementary information from diverse input modalities, contributing to improved glioblastoma survival prediction compared to state-of-the-art methods. Consistent performance was observed across institutions supporting model generalizability.

Keywords: MRI; deep learning; glioblastoma; multimodal data; prognosis; survival prediction.