In the past decennia, our understanding of the sexual differentiation of the mammalian brain has dramatically changed. The simple model according to which testosterone masculinizes the brain of males away from a default female form, was replaced with a complex scenario, according to which sex effects on the brain of both females and males are exerted by genetic, hormonal, and environmental factors. These factors act via multiple partly independent mechanisms that may vary according to internal and external factors. These observations led to the "mosaic" hypothesis-the expectation of high variability in the degree of "maleness"/"femaleness" of different features within a single brain. Here, we briefly review animal data that form the basis of current understanding of sexual differentiation; present, in this context, the results of co-analyses of human brain measures obtained by magnetic resonance imaging or postmortem; discuss criticisms and controversies of the mosaic hypothesis and implications for research; and conclude that co-analysis of several (preferably, many) features and going back from the group level to that of the individual would advance our understanding of the relations between sex and the brain in health and disease.