There is now converging evidence that others' actions are represented in the motor system. However, social cognition requires us to represent not only the actions but also the interactions of others. To do so, it is imperative that the motor system can represent multiple observed actions. The current fMRI study investigated whether this is possible by measuring brain activity from 29 participants while they observed 2 right hands performing sign language gestures. Three key results were obtained. First, brain activity in the premotor and parietal motor cortex was stronger when 2 hands performed 2 different gestures than when 1 hand performed a single gesture. Second, both individual observed gestures could be decoded from brain activity in the same 2 regions. Third, observing 2 different gestures compared with 2 identical gestures activated brain areas related to motor conflict, and this activity was correlated with parietal motor activity. Together, these results show that the motor system is able to represent multiple observed actions, and as such reveal a potential mechanism by which third-party social encounters could be processed in the brain.