Attention deficits may present dysfunctions in any one or two components of attention (alerting, orienting, and executive control (EC)). However, these various forms of attention deficits generally have abnormal microstructure integrity of inferior fronto-occipital fasciculus (IFOF). In this work, we aim to deeply explore: (1) associations between microstructure integrities of IFOF (including frontal, parietal, temporal, occipital, and insular segments) and attention by means of structural equation models and multiple regression analyses; (2) genetic/environmental effects on IFOF, attention, and their correlations using bivariate genetic analysis. EC function was attributed to the fractional anisotropy (FA) of left (correlation was driven by genetic and environmental factors) and right IFOF (correlation was driven by environmental factors), especially to left frontal part and right occipital part (correlation was driven by genetic factors). Alerting was associated with FA in parietal and insular parts of left IFOF. No significant correlation was found between orienting and IFOF. This study revealed the advantages of lobar-segmental analysis in structure-function correlation study and provided the anatomical basis for kinds of attention deficits. The common genetic/environmental factors implicated in the certain correlations suggested the common physiological mechanisms for two traits, which should promote the discovery of single-nucleotide polymorphisms affecting IFOF and attention.