Patients with chronic epilepsy frequently display cognitive comorbidity and might have widespread network abnormalities outside the epileptic zone, which might affect a variety of cognitive functions and global intelligence. We aimed to study the role of white matter connectivity in cognitive comorbidity. Thirty-nine patients with nonsymptomatic localization-related epilepsy and varying degrees of cognitive impairment and 23 age-matched healthy controls were included. Whole brain white matter networks were constructed from fiber tractography. Weighted graph theoretical analysis was performed to study white matter network abnormalities associated with epilepsy and cognition. Patients with severe cognitive impairment showed lower clustering (a measure of brain network segregation) and higher path length (a measure of brain network integration) compared with the healthy controls and patients with little or no cognitive impairment, whereas whole brain white matter volume did not differ. Correlation analyses revealed that IQ and cognitive impairment were strongly associated with clustering and path lengths. This study revealed impaired white matter connectivity, associated with cognitive comorbidity in patients with chronic epilepsy. As whole brain white matter volumes were preserved in the patient group, our results suggest an important role for the network topology rather than volumetric changes, in epilepsy with cognitive decline.