Early endosome-to-trans-Golgi network (TGN) transport is organized by the retromer complex. Consisting of cargo-selective and membrane-bound subcomplexes, retromer coordinates sorting with membrane deformation and carrier formation. Here, we describe four mammalian retromers whose membrane-bound subcomplexes contain specific combinations of the sorting nexins (SNX), SNX1, SNX2, SNX5, and SNX6. We establish that retromer requires a dynamic spatial organization of the endosomal network, which is regulated through association of SNX5/SNX6 with the p150(glued) component of dynactin, an activator of the minus-end directed microtubule motor dynein; an association further defined through genetic studies in C. elegans. Finally, we also establish that the spatial organization of the retromer pathway is mediated through the association of SNX1 with the proposed TGN-localized tether Rab6-interacting protein-1. These interactions describe fundamental steps in retromer-mediated transport and establish that the spatial organization of the retromer network is a critical element required for efficient retromer-mediated sorting.