For proper maturation of the neocortex and acquisition of specific functions and skills, exposure to sensory stimuli is vital during critical periods of development when synaptic connectivity is highly malleable. To preserve reliable cortical processing, it is essential that these critical periods end after which learning becomes more conditional and active interaction with the environment becomes more important. How these age-dependent forms of plasticity are regulated has been studied extensively in the primary visual cortex. This has revealed that inhibitory innervation plays a crucial role and that a temporary decrease in inhibition is essential for plasticity to take place. Here, we discuss how different interneuron subsets regulate plasticity during different stages of cortical maturation. We propose a theory in which different interneuron subsets select the sources of neuronal input that undergo plasticity.