Less is Better: A cognitively inspired unsupervised model for language segmentation

Jinbiao Yang, Stefan Frank, A. van den Bosch

Research output: Chapter in book/volumeContribution to conference proceedingsScientificpeer-review


Language users process utterances by segmenting them into many cognitive units, which vary in their sizes and linguistic levels. Although we can do such unitization/segmentation easily, its cognitive mechanism is still not clear. This paper proposes an unsupervised model, Less-is-Better (LiB), to simulate the human cognitive process with respect to language unitization/segmentation. LiB follows the principle of least effort and aims to build a lexicon which minimizes the number of unit tokens (alleviating the effort of analysis) and number of unit types (alleviating the effort of storage) at the same time on any given corpus. LiB’s workflow is inspired by empirical cognitive phenomena. The design makes the mechanism of LiB cognitively plausible and the computational requirement light-weight. The lexicon generated by LiB performs the best among different types of lexicons (e.g. ground-truth words) both from an information-theoretical view and a cognitive view, which suggests that the LiB lexicon may be a plausible proxy of the mental lexicon.
Original languageEnglish
Title of host publicationProceedings of the Workshop on the Cognitive Aspects of the Lexicon
PublisherAssociation for Computational Linguistics (ACL)
Publication statusPublished - 01 Dec 2020


Dive into the research topics of 'Less is Better: A cognitively inspired unsupervised model for language segmentation'. Together they form a unique fingerprint.

Cite this