The objective of this study is to investigate the relationships among frontotemporal fiber tract compromise and task-switching performance in healthy controls and patients with temporal lobe epilepsy (TLE). We performed diffusion tensor imaging (DTI) on 30 controls and 32 patients with TLE (15 left TLE). Fractional anisotropy (FA) was calculated for four fiber tracts [uncinate fasciculus (UncF), arcuate fasciculus (ArcF), dorsal cingulum (CING), and inferior fronto-occipital fasciculus (IFOF)]. Participants completed the Trail Making Test-B (TMT-B) and Verbal Fluency Category Switching (VFCS) test. Multivariate analyses of variances (MANOVAs) were performed to investigate group differences in fiber FA and set-shifting performances. Canonical correlations were used to examine the overall patterns of structural-cognitive relationships and were followed by within-group bivariate correlations. We found a significant canonical correlation between fiber FA and task-switching performance. In controls, TMT-B correlated with left IFOF, whereas VFCS correlated with FA of left ArcF and left UncF. These correlations were not significant in patients with TLE. We report significant correlations between frontotemporal fiber tract integrity and set-shifting performance in healthy controls that appear to be absent or attenuated in patients with TLE. These findings suggest a breakdown of typical structure-function relationships in TLE that may reflect aberrant developmental or degenerative processes.