Batavia asked for advice: Pretrained language models for Named Entity Recognition in historical texts

TY - CHAP

T1 - Batavia asked for advice

T2 - Pretrained language models for Named Entity Recognition in historical texts

AU - Arnoult, Sophie

AU - Petram, Lodewijk

AU - Vossen, Piek

N1 - Funding Information: project, and supported by NWO project number CP-WP6-19-005. We thank the members of the VOC use case group for interesting discussions and the anonymous reviewers for their pertinent comments. Publisher Copyright: © LaTeCHCLfL,EMNLP 2021 - Proceedings.All rights reserved.

PY - 2021/11

Y1 - 2021/11

N2 - Pretrained language models like BERT have advanced the state of the art for many NLP tasks. For resource-rich languages, one has the choice between a number of language-specific models, while multilingual models are also worth considering. These models are well known for their crosslingual performance, but have also shown competitive in-language performance on some tasks. We consider monolingual and multilingual models from the perspective of historical texts, and in particular for texts enriched with editorial notes: how do language models deal with the historical and editorial content in these texts? We present a new Named Entity Recognition dataset for Dutch based on 17th and 18th century United East India Company (VOC) reports extended with modern editorial notes. Our experiments with multilingual and Dutch pretrained language models confirm the crosslingual abilities of multilingual models while showing that all language models can leverage mixed-variant data. In particular, language models successfully incorporate notes for the prediction of entities in historical texts. We also find that multilingual models outperform monolingual models on our data, but that this superiority is linked to the task at hand: multilingual models lose their advantage when confronted with more semantical tasks.

AB - Pretrained language models like BERT have advanced the state of the art for many NLP tasks. For resource-rich languages, one has the choice between a number of language-specific models, while multilingual models are also worth considering. These models are well known for their crosslingual performance, but have also shown competitive in-language performance on some tasks. We consider monolingual and multilingual models from the perspective of historical texts, and in particular for texts enriched with editorial notes: how do language models deal with the historical and editorial content in these texts? We present a new Named Entity Recognition dataset for Dutch based on 17th and 18th century United East India Company (VOC) reports extended with modern editorial notes. Our experiments with multilingual and Dutch pretrained language models confirm the crosslingual abilities of multilingual models while showing that all language models can leverage mixed-variant data. In particular, language models successfully incorporate notes for the prediction of entities in historical texts. We also find that multilingual models outperform monolingual models on our data, but that this superiority is linked to the task at hand: multilingual models lose their advantage when confronted with more semantical tasks.

KW - computational linguistics

KW - history

KW - pretrained language models

KW - named entity recognition

KW - VOC

UR - http://www.scopus.com/inward/record.url?scp=85138470608&partnerID=8YFLogxK

M3 - Contribution to conference proceedings

SP - 21

EP - 30

BT - Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature

A2 - Dagaetano-Ortlieb, Stefania

A2 - Kazantseva, Anna

A2 - Reiter, Nils

A2 - Szpakowicz, Stan

PB - Association for Computational Linguistics (ACL)

ER -