Soil-borne pathogens cause great crop losses in agriculture. Because of their resilience in the soil, these pathogens persist in a population reservoir, causing future outbreaks of crop diseases. Management focus is usually on the most common pathogens occurring, but it is likely that a mixed population of pathogens together affect crops. Next generation sequencing of DNA from environmental samples can provide information on the presence of potential pathogens. The aim of this study was to obtain insight into the factors that drive the composition of potential plant pathogen populations in agricultural soils. To this end, the alpha and beta diversity of fungal OTUs that were assigned as potential plant pathogens for 42 agricultural soils were assessed. The presented study is the first inventory of the pool of pathogens and its correlating factors. The results of this inventory indicate that the composition of pathogens in soil is driven by pH, soil type, crop history, litter saprotrophic fungi and spatial patterns. The major driving factors differed between potential root- and shoot-infecting fungi, suggesting interactions among environmental factors and pathogen traits like reproduction, survival and dispersal. This information is important to understand risks for disease outbreaks and to recommend management strategies to prevent such outbreaks.