Previous diffusion tensor imaging (DTI) studies have shown that various factors can affect white matter (WM) tract diffusivity. The aim of the present study was to investigate the effects of childhood adversity (CA), age and gender on WM diffusivity in tracts that are thought to be involved in emotional regulation in individuals with major depressive disorder (MDD) and healthy controls (HC). DTI was obtained from 46 subjects with MDD and 46 HC subjects. Data were pre-processed and deterministic tractography was applied in the cingulum, uncinate fasciculus (UF), fornix, superior longitudinal fasciculus (SLF) and fronto-occipital fasciculus (FOF). In subjects with a history of CA, fractional anisotropy (FA) was greater in the rostral cingulum (RC) and dorsal cingulum, whereas radial diffusivity (RD) was smaller in the RC when compared with subjects with no history of CA. In the UF, FOF and parahippocampal cingulum, FA was greater in the left hemisphere in the subjects with CA when compared with those without CA. Age affected FA, longitudinal diffusivity and RD in the UF, fornix, FOF and SLF, reflecting axonal and myelin degeneration with increasing age. Depression or gender did not have any effects on the diffusivity measures. Due to the cross-sectional nature of the study, a recall bias for CA and possible effects of medical treatment on diffusivity measures could have played a role. CA and age could increase the likelihood to develop WM microstructural anomalies in the brain affective network. Moreover, subjects with CA could be more vulnerable to FA changes.