Biological causes underpinning the well known gender dimorphisms in human behavior, cognition, and emotion have received increased attention in recent years. The advent of diffusion-weighted magnetic resonance imaging has permitted the investigation of the white matter microstructure in unprecedented detail. Here, we aimed to study the potential influences of biological sex, gender identity, sex hormones, and sexual orientation on white matter microstructure by investigating transsexuals and healthy controls using diffusion tensor imaging (DTI). Twenty-three female-to-male (FtM) and 21 male-to-female (MtF) transsexuals, as well as 23 female (FC) and 22 male (MC) controls underwent DTI at 3 tesla. Fractional anisotropy, axial, radial, and mean diffusivity were calculated using tract-based spatial statistics (TBSS) and fiber tractography. Results showed widespread significant differences in mean diffusivity between groups in almost all white matter tracts. FCs had highest mean diffusivities, followed by FtM transsexuals with lower values, MtF transsexuals with further reduced values, and MCs with lowest values. Investigating axial and radial diffusivities showed that a transition in axial diffusivity accounted for mean diffusivity results. No significant differences in fractional anisotropy maps were found between groups. Plasma testosterone levels were strongly correlated with mean, axial, and radial diffusivities. However, controlling for individual estradiol, testosterone, or progesterone plasma levels or for subjects' sexual orientation did not change group differences. Our data harmonize with the hypothesis that fiber tract development is influenced by the hormonal environment during late prenatal and early postnatal brain development.