High-grade glioma (HGG) is an aggressive brain tumor. Sex is an important factor that differentially affects survival outcomes in HGG. We used an end-to-end deep learning approach on hematoxylin and eosin (H&E) scans to (i) identify sex-specific histopathological attributes of the tumor microenvironment (TME), and (ii) create sex-specific risk profiles to prognosticate overall survival. Surgically resected H&E-stained tissue slides were analyzed in a two-stage approach using ResNet18 deep learning models, first, to segment the viable tumor regions and second, to build sex-specific prognostic models for prediction of overall survival. Our mResNet-Cox model yielded C-index (0.696, 0.736, 0.731, and 0.729) for the female cohort and C-index (0.729, 0.738, 0.724, and 0.696) for the male cohort across training and three independent validation cohorts, respectively. End-to-end deep learning approaches using routine H&E-stained slides, trained separately on male and female patients with HGG, may allow for identifying sex-specific histopathological attributes of the TME associated with survival and, ultimately, build patient-centric prognostic risk assessment models.