Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022

Ali Hürriyetoğlu; Osman Mutlu; Fırat Duruşan; Onur Uca; Alaeddin Selçuk Gürel; Benjamin Radford; Yaoyao Dai; Hansi Hettiarachchi; Niklas Stoehr; Tadashi Nomoto; Milena Slavcheva; Francielle Vargas; Aaqib Javid; Fatih Beyhan; Erdem Yörük

Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022

Ali Hürriyetoğlu, Osman Mutlu, Fırat Duruşan, Onur Uca, Alaeddin Selçuk Gürel, Benjamin Radford, Yaoyao Dai, Hansi Hettiarachchi, Niklas Stoehr, Tadashi Nomoto, Milena Slavcheva, Francielle Vargas, Aaqib Javid, Fatih Beyhan, Erdem Yörük

Research output: Contribution to conference › Paper › Scientific › peer-review

13 Citations (Scopus)

Abstract

We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection. This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification. The training data from CASE 2021 in English, Portuguese and Spanish were utilized. Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting. The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well. We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting. The winning approaches are mainly ensembling models and merging data in multiple languages. The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages. Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.

Original language	English
Pages	223-228
Number of pages	6
Publication status	Published - 2022

Access to Document

https://aclanthology.org/2022.case-1.31.pdf

Cite this

@conference{04f145c43aad4dec989d15b36405c5fe,

title = "Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022",

abstract = "We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection. This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification. The training data from CASE 2021 in English, Portuguese and Spanish were utilized. Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting. The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well. We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting. The winning approaches are mainly ensembling models and merging data in multiple languages. The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages. Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.",

author = "Ali H{\"u}rriyetoğlu and Osman Mutlu and Fırat Duru{\c s}an and Onur Uca and G{\"u}rel, {Alaeddin Sel{\c c}uk} and Benjamin Radford and Yaoyao Dai and Hansi Hettiarachchi and Niklas Stoehr and Tadashi Nomoto and Milena Slavcheva and Francielle Vargas and Aaqib Javid and Fatih Beyhan and Erdem Y{\"o}r{\"u}k",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.",

year = "2022",

language = "English",

pages = "223--228",

}

TY - CONF

T1 - Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022

AU - Hürriyetoğlu, Ali

AU - Mutlu, Osman

AU - Duruşan, Fırat

AU - Uca, Onur

AU - Gürel, Alaeddin Selçuk

AU - Radford, Benjamin

AU - Dai, Yaoyao

AU - Hettiarachchi, Hansi

AU - Stoehr, Niklas

AU - Nomoto, Tadashi

AU - Slavcheva, Milena

AU - Vargas, Francielle

AU - Javid, Aaqib

AU - Beyhan, Fatih

AU - Yörük, Erdem

PY - 2022

Y1 - 2022

N2 - We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection. This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification. The training data from CASE 2021 in English, Portuguese and Spanish were utilized. Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting. The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well. We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting. The winning approaches are mainly ensembling models and merging data in multiple languages. The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages. Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.

AB - We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection. This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification. The training data from CASE 2021 in English, Portuguese and Spanish were utilized. Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting. The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well. We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting. The winning approaches are mainly ensembling models and merging data in multiple languages. The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages. Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.

UR - http://www.scopus.com/inward/record.url?scp=85143403080&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85143403080

SP - 223

EP - 228

ER -

Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022

Abstract

Access to Document

Other files and links

Fingerprint

Cite this