Microbial communities are at the very basis of life on earth, catalyzing biogeochemical reactions driving global nutrient cycles. However, unlike for plants and animals, microbial diversity is not on the biodiversity–conservation agenda. The latter, however, would imply that microbial diversity is not under any threat by anthropogenic disturbance or climate change. This maybe a misconception caused by the rudimentary knowledge we have concerning microbial diversity and its role in ecosystem functioning. This perspective paper identifies major areas with knowledge gaps within the field of environmental microbiology that preclude a comprehension of microbial ecosystems on the level we have for plants and animals. Opportunities and challenges are pointed out to open the microbial black box and to go from descriptive to predictive microbial ecology.