Extra‐pair paternity (EPP) has been suggested to improve the genetic quality of offspring, but evidence has been equivocal. Benefits of EPP may be only available to specific individuals or under certain conditions. Red‐winged fairy‐wrens have extremely high levels of EPP, suggesting fitness benefits might be large and available to most individuals. Furthermore, extreme philopatry commonly leads to incestuous social pairings, so inbreeding avoidance may be an important selection pressure. Here, we quantified the fitness benefits of EPP under varying conditions and across life‐stages. Extra‐pair offspring (EPO) did not appear to have higher fitness than within‐pair offspring (WPO), neither in poor years nor in the absence of helpers‐at‐the‐nest. However, EPP was beneficial for closely related social pairs, because inbred WPO suffered an overall 75% reduction in fitness. Inbreeding depression was nonlinear and reduced nestling body condition, first year survival and reproductive success. Our comprehensive study indicates that EPP should be favored for the 17% of females paired incestuously, but cannot explain the widespread infidelity in this species. Furthermore, our finding that fitness benefits of EPP only become apparent for a small part of the population could potentially explain the apparent absence of fitness differences in population wide comparisons of EPO and WPO.