Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora

F.B. Karsdorp; Enrique Manjavacas; Lauren Fonteyn

Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora

F.B. Karsdorp, Enrique Manjavacas, Lauren Fonteyn

Research output: Chapter in book/volume › Contribution to conference proceedings › Scientific › peer-review

Abstract

The question how we can reliably estimate the lexical diversity of a particular text (collection) has often been asked by linguists and literary scholars alike. This short paper introduces a way of operationalizing functional diversity measurements by means of token-based embeddings, and argues that functional diversity is not only a practically advantageous, but also a theoretically relevant addition to the Computational Humanities Research toolkit. By means of an experiment on the historical ARCHER corpus, we show that lexical diversity at the level of functional groups is less sensitive to orthographic variation, and provides insight into an important and often disregarded dimension of vocabulary diversity
in textual data.

Original language	English
Title of host publication	Proceedings of the Computational Humanities Research Conference 2022
Editors	Folgert Karsdorp, Alie Lassche, Kristoffer Nielbo
Publisher	CEUR Workshop Proceedings
Pages	114-126
Volume	3290
Publication status	Published - Nov 2022

Access to Document

https://ceur-ws.org/Vol-3290/short_paper2780.pdfLicence: CC BY

Cite this

Karsdorp, F. B., Manjavacas, E., & Fonteyn, L. (2022). Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora. In F. Karsdorp, A. Lassche, & K. Nielbo (Eds.), Proceedings of the Computational Humanities Research Conference 2022 (Vol. 3290, pp. 114-126). CEUR Workshop Proceedings. https://ceur-ws.org/Vol-3290/short_paper2780.pdf

@inbook{ef0b51f61e224a5fbe51c192a3b4c024,

title = "Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora",

abstract = "The question how we can reliably estimate the lexical diversity of a particular text (collection) has often been asked by linguists and literary scholars alike. This short paper introduces a way of operationalizing functional diversity measurements by means of token-based embeddings, and argues that functional diversity is not only a practically advantageous, but also a theoretically relevant addition to the Computational Humanities Research toolkit. By means of an experiment on the historical ARCHER corpus, we show that lexical diversity at the level of functional groups is less sensitive to orthographic variation, and provides insight into an important and often disregarded dimension of vocabulary diversityin textual data.",

author = "F.B. Karsdorp and Enrique Manjavacas and Lauren Fonteyn",

year = "2022",

month = nov,

language = "English",

volume = "3290",

pages = "114--126",

editor = "Folgert Karsdorp and Alie Lassche and Kristoffer Nielbo",

booktitle = "Proceedings of the Computational Humanities Research Conference 2022",

publisher = "CEUR Workshop Proceedings",

}

Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora. / Karsdorp, F.B.; Manjavacas, Enrique; Fonteyn, Lauren.
Proceedings of the Computational Humanities Research Conference 2022. ed. / Folgert Karsdorp; Alie Lassche; Kristoffer Nielbo. Vol. 3290 CEUR Workshop Proceedings, 2022. p. 114-126.

Research output: Chapter in book/volume › Contribution to conference proceedings › Scientific › peer-review

TY - CHAP

T1 - Introducing Functional Diversity

T2 - A Novel Approach to Lexical Diversity in (Historical) Corpora

AU - Karsdorp, F.B.

AU - Manjavacas, Enrique

AU - Fonteyn, Lauren

PY - 2022/11

Y1 - 2022/11

N2 - The question how we can reliably estimate the lexical diversity of a particular text (collection) has often been asked by linguists and literary scholars alike. This short paper introduces a way of operationalizing functional diversity measurements by means of token-based embeddings, and argues that functional diversity is not only a practically advantageous, but also a theoretically relevant addition to the Computational Humanities Research toolkit. By means of an experiment on the historical ARCHER corpus, we show that lexical diversity at the level of functional groups is less sensitive to orthographic variation, and provides insight into an important and often disregarded dimension of vocabulary diversityin textual data.

AB - The question how we can reliably estimate the lexical diversity of a particular text (collection) has often been asked by linguists and literary scholars alike. This short paper introduces a way of operationalizing functional diversity measurements by means of token-based embeddings, and argues that functional diversity is not only a practically advantageous, but also a theoretically relevant addition to the Computational Humanities Research toolkit. By means of an experiment on the historical ARCHER corpus, we show that lexical diversity at the level of functional groups is less sensitive to orthographic variation, and provides insight into an important and often disregarded dimension of vocabulary diversityin textual data.

M3 - Contribution to conference proceedings

VL - 3290

SP - 114

EP - 126

BT - Proceedings of the Computational Humanities Research Conference 2022

A2 - Karsdorp, Folgert

A2 - Lassche, Alie

A2 - Nielbo, Kristoffer

PB - CEUR Workshop Proceedings

ER -

Introducing Functional Diversity: A Novel Approach to Lexical Diversity in (Historical) Corpora

Abstract

Access to Document

Fingerprint

Cite this