Automatic summarization of domain-specific forum threads: Collecting reference data

TY - CHAP

T1 - Automatic summarization of domain-specific forum threads

T2 - Collecting reference data

AU - Verberne, Suzan

AU - van den Bosch, Antal

AU - Wubben, Sander

AU - Krahmer, Emiel

PY - 2017/3/7

Y1 - 2017/3/7

N2 - We create and analyze two sets of reference summaries fordiscussion threads on a patient support forum: expert summariesand crowdsourced, non-expert summaries. Ideally,reference summaries for discussion forum threads are createdby expert members of the forum community. Whenthere are few or no expert members available, crowdsourcingthe reference summaries is an alternative. In this paperwe investigate whether domain-specific forum data requiresthe hiring of domain experts for creating referencesummaries. We analyze the inter-rater agreement for bothdatasets and we train summarization models using the twotypes of reference summaries. The inter-rater agreement incrowdsourced reference summaries is low, close to random,while domain experts achieve a considerably higher, fair,agreement. The trained models however are similar to eachother. We conclude that it is possible to train an extractivesummarization model on crowdsourced data that is similarto an expert model, even if the inter-rater agreement for thecrowdsourced data is low.

AB - We create and analyze two sets of reference summaries fordiscussion threads on a patient support forum: expert summariesand crowdsourced, non-expert summaries. Ideally,reference summaries for discussion forum threads are createdby expert members of the forum community. Whenthere are few or no expert members available, crowdsourcingthe reference summaries is an alternative. In this paperwe investigate whether domain-specific forum data requiresthe hiring of domain experts for creating referencesummaries. We analyze the inter-rater agreement for bothdatasets and we train summarization models using the twotypes of reference summaries. The inter-rater agreement incrowdsourced reference summaries is low, close to random,while domain experts achieve a considerably higher, fair,agreement. The trained models however are similar to eachother. We conclude that it is possible to train an extractivesummarization model on crowdsourced data that is similarto an expert model, even if the inter-rater agreement for thecrowdsourced data is low.

UR - http://dl.acm.org/authorize.cfm?key=N22263

M3 - Contribution to conference proceedings

SP - 253

EP - 256

BT - Proceedings of the 2017 Conference on Human Information Interaction and Retrieval (CHIIR-2017)

PB - Association for Computing Machinery (ACM)

ER -