Creating a reference data set for the summarization of discussion forum threads

Suzan Verberne, Emiel Krahmer, Iris Hendrickx, Sander Wubben, Antal van Den Bosch

Research output: Contribution to journal/periodicalArticleScientificpeer-review


In this paper we address extractive summarization of long threads in online discussion fora. We present an elaborate user evaluation study to determine human preferences in forum summarization and to create a reference data set. We showed long threads to ten different raters and asked them to create a summary by selecting the posts that they considered to be the most important for the thread. We study the agreement between human raters on the summarization task, and we show how multiple reference summaries can be combined to develop a successful model for automatic summarization. We found that although the inter-rater agreement for the summarization task was slight to fair, the automatic summarizer obtained reasonable results in terms of precision, recall, and ROUGE. Moreover, when human raters were asked to choose between the summary created by another human and the summary created by our model in a blind side-by-side comparison, they judged the model's summary equal to or better than the human summary in over half of the cases. This shows that even for a summarization task with low inter-rater agreement, a model can be trained that generates sensible summaries. In addition, we investigated the potential for personalized summarization. However, the results for the three raters involved in this experiment were inconclusive. We release the reference summaries as a publicly available dataset.
Original languageEnglish
Pages (from-to)1-23
Number of pages23
JournalLanguage Resources and Evaluation
Publication statusPublished - 21 Apr 2017


  • Data collection
  • Discussion forums
  • Evaluation
  • Inter-rater agreement
  • Summarization
  • User study


Dive into the research topics of 'Creating a reference data set for the summarization of discussion forum threads'. Together they form a unique fingerprint.

Cite this