ucto: a rule-based tokeniser

Maarten van Gompel (Ontwikkelaar), Ko van der Sloot (Ontwikkelaar)

Onderzoeksoutput: Niet-tekstuele vormSoftwareWetenschappelijk


Ucto tokenizes text files: it separates words from punctuation, and splits
sentences. This is one of the first tasks for almost any Natural Language
Processing application. Ucto offers several other basic preprocessing steps
such as changing case that you can all use to make your text suited for further
processing such as indexing, part-of-speech tagging, or machine translation.

Ucto comes with tokenisation rules for several languages (packaged separately)
and can be easily extended to suit other languages. It has been incorporated
for tokenizing Dutch text in Frog (https://languagemachines.github.io/frog),
our Dutch morpho-syntactic processor.

The software is intended to be used from the command-line by researchers in
Natural Language Processing or related areas, as well as software developers.
Originele taal-2Engels
StatusGepubliceerd - 08 okt. 2020


Duik in de onderzoeksthema's van 'ucto: a rule-based tokeniser'. Samen vormen ze een unieke vingerafdruk.
  • python-ucto

    van Gompel, M., 09 okt. 2020

    Onderzoeksoutput: Niet-tekstuele vormSoftwareWetenschappelijk

    Open Access

Citeer dit