Exposure to stress during the early postnatal period (i.e., early life stress, ES) can impact brain physiology and modify individual variability in adult social behavior. Arginine vasopressin (AVP) and oxytocin (OXT) are two centrally released neuropeptides that are involved in shaping essential social behaviors, like aggression, social recognition, and social motivation. AVP and OXT modulate activity in brain regions important for the establishment of social behavior, and may be particularly sensitive to ES. In this review, we discuss whether ES alters the characteristics of the AVP- and OXT- systems in rodents, and whether these changes are associated with later alterations in aggression, social recognition, and social motivation. We have integrated causal studies indicating that (1) ES affects AVP/OXT, and (2) that changing AVP/OXT in affected regions alters social behavior. Although there is encouraging evidence that ES causes AVP- and OXT-system changes, and that these may mediate social behavior, a comprehensive understanding of the exact nature of AVP- and OXT changes and whether they are causal in establishing these behavioral disturbances needs further investigation. As there are indications that ES alters AVP- and OXT characteristics in humans as well, and that these may interact with adult predisposition to psychopathology with social dysfunction, future rodent studies may lay ground for a better understanding of such changes in humans. Ultimately, this may assist in developing therapeutic strategies to target ES effects on social behavior.