Assemblages of large herbivores may compete for food or facilitate one another. However, small vertebrate herbivore species co-occurring with large herbivores may be affected by large herbivore grazing through changes in plant species composition, nutrient content and vegetation structure. These changes can be either positive or negative for the smaller herbivores, but this may depend on the species of small herbivores. We experimentally tested the impact of cattle grazing on habitat choice of European rabbits (Oryctolagus cuniculus) and common voles (Microtus arvalis). We excluded cattle for 7 years and measured changes in vegetation parameters, and the response of rabbits and voles. Rabbits were facilitated by cattle, whereas voles strongly preferred vegetation without cattle. The facilitation effect was stronger at low rabbit densities. Vegetation biomass and nitrogen concentration were not affected by cattle grazing, but vegetation height increased significantly where cattle were excluded. Plant species composition also changed following cattle exclusion; however, the main food plants of rabbits and voles remained abundant in each grazing treatment. We conclude that the response of both rabbits and voles predominantly reflect the differences in vegetation height in the presence and absence of cattle, but in a contrasting fashion. The difference in response between rabbits and voles may result from reduced perceived predation risk, which is lowest in high vegetation for voles, but in short vegetation for rabbits, which depend on their burrows for safety. The use of large herbivores in grassland conservation management can thus have a contrasting effect on different species of small herbivores.