Perceiving other people's actions triggers activity in premotor and parietal areas, brain areas also involved in executing and sensing our own actions. Paralleling this phenomenon, observing emotional states (including pain) in others is associated with activity in the same brain areas as activated when experiencing similar emotions directly. This emotion perception associated activity has been shown to be affected by the perceived fairness of the actor, and in-group membership more generally. Here, we examine whether action observation associated brain activity is also affected by the perceived social fairness of the actors. Perceived fairness was manipulated using an alternating iterated Prisoner's Dilemma game between the participant and two confederates, one of whom played fairly and the other unfairly. During fMRI scanning the participants watched movies of the confederates performing object-directed hand actions, and then performed hand actions themselves. Mass-univariate analysis showed that observing the actions triggered robust activation in regions associated with action execution, but failed to identify a strong modulation of this activation based on perceived fairness. Multivariate pattern analysis, however, identified clusters potentially carrying information about the perceived fairness of the actor in the middle temporal gyrus, left postcentral gyrus, right inferior parietal lobule, right middle cingulate cortex, right angular gyrus, and right superioroccipital gyrus. Despite being identified by a whole-brain searchlight analysis (and so without anatomical restriction), these clusters fall into areas frequently associated with action observation. We conclude that brain activity during action observation may be modulated by perceived fairness, but such modulation is subtle; robust activity is associated with observing the actions of both fair and unfair individuals.