Mijail Kabadjov
ABSTRACT
In this paper we present the Online Forum Summarisation (OnForumS) pilot task at MultiLing'15. OnForumS is a pioneering attempt at encompassing automatic summarisation, argumentation mining and sentiment analysis into one shared task and at bringing crowdsourcing to the evaluation of systems for automatic summarisation and argument structure parsing. It covered two languages, English and Italian. Four research groups, each submitting two runs, participated in the task and these complemented with two baseline system runs were evaluated via crowdsourcing. Performance results are presented and briefly discussed. Being the first of its kind, we believe OnForumS'15 was a successful campaign and hope it will establish itself as a valuable exercise in advancing the state-of-the-art in this new emerging area. Current plans are to organise it again jointly with MultiLing in 2017 and to include more languages.
- A. Aker, F. Celli, A. Funk, E. Kurtic, M. Hepple, and R. Gaizauskas. Sheffield-Trento System for Sentiment and Argument Structure Enhanced Comment-to-Article Linking in the Online News Domain. http://multiling.iit.demokritos.gr/file/download/1577, 2015. {Online; accessed 06-August-2015}.Google Scholar
- A. Balahur, E. van der Goot, R. Steinberger, and A. Montoyo, editors. Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Baltimore (MD), USA, 2014.Google ScholarCross Ref
- S. Bhatia and P. Mitra. Adopting inference networks for online thread retrieval. In Proceedings of AAAI, pages 1300--1305, 2010.Google ScholarCross Ref
- B. Boguraev and C. Kennedy. Salience-based content characterisation of text documents. In I. Mani, editor, Proceedings of the Workshop on Intelligent and Scalable Text Summarization at the Annual Joint Meeting of the ACL/EACL, Madrid, 1997.Google Scholar
- F. Boltuzic and J. Šnajder. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49--58, Baltimore (MD), USA, 2014.Google ScholarCross Ref
- C. Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'09), volume 1, pages 286âĂŞ--295, 2009. Google ScholarDigital Library
- G. Erkan and D. Radev. LexRank: Graph-based centrality as salience in text summarization. Journal of Artificial Intelligence Research (JAIR), 2004. Google ScholarDigital Library
- D. Ghosh, S. Muresan, N. Wacholder, M. Aakhus, and M. Mitsui. Analyzing argumentative discourse units in online interactions. In Proceedings of the First Workshop on Argumentation Mining, pages 39--48, Baltimore (MD), USA, 2014.Google ScholarCross Ref
- G. Giannakopoulos and V. Karkaletsis. Summary evaluation: Together we stand npower-ed. In Computational Linguistics and Intelligent Text Processing, pages 436--450. Springer, 2013. Google ScholarDigital Library
- P. Krejzl, J. Steinberger, T. Hercig, and T. Brychcín. UWB Participation in the Multiling's OnForumS Task. http://multiling.iit.demokritos.gr/file/download/1578, 2015. {Online; accessed 06-August-2015}.Google Scholar
- A. Louis and A. Nenkova. Automatically assessing machine summary content without a gold-standard. Computational Linguistics, 39(2):267--300, 2013. Google ScholarDigital Library
- H. Luhn. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159--165, 1958. Google ScholarDigital Library
- M.-F. Moens. Argumentation mining: Where are we now, where do we want to be and how do we get there? In P. Majumder, M. Mitra, M. Agrawal, and P. Mehta, editors, Proceedings of the 5th 2013 Forum on Information Retrieval Evaluation (FIRE'13), ACM, New York, NY, USA, 2013. Google ScholarDigital Library
- R. M. Palau and M.-F. Moens. Study on the structure of argumentation in case law. In Proceedings of the Conference on Legal Knowledge and Information Systems, pages 11--20, 2008. Google ScholarDigital Library
- R. M. Palau and M.-F. Moens. Argumentation mining. Artificial Intelligence and Law, 19(1):1--22, 2011. Google ScholarDigital Library
- B. Pang and L. Lee. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1--2):1--135, 2008. Google ScholarDigital Library
- J. Park and C. Cardie. Identifying appropriate support for propositions in on-line user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29--38, Baltimore (MD), USA, 2014.Google ScholarCross Ref
- R. J. Passonneau and B. Carpenter. The benefits of a model of annotation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 187ââĂŞ--195, Sofia, Bulgaria, August 2013.Google Scholar
- R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast -- but is it good?: Evaluating nonexpert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'08), pages 254âĂŞ--263, 2008. Google ScholarDigital Library
- I. Soboroff. Test collection diagnosis and treatment. In Proceedings of the Third International Workshop on Evaluating Information Access (EVIA), pages 34--41, Tokyo, Japan, June 2010.Google Scholar
- C. Stab and I. Gurevych. Annotating argument components and relations in persuasive essays. In Proceedings of COLING, pages 1501--1510, 2014.Google Scholar
- H. Tanev and A. Balahur. Tackling the OnForumS Challenge. http://multiling.iit.demokritos.gr/file/download/1576, 2015. {Online; accessed 06-August-2015}.Google Scholar
- P. Turney and M. Littman. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21, 2003. Google ScholarDigital Library
- M. Walker, J. F. Tree, P. Anand, R. Abbott, and J. King. A corpus for research on deliberation and debate. In Proceedings of LREC, 2012.Google Scholar
- S. Wan, L. Li, T. Huang, Z. Gao, L. Mao, and F. Huang. CIST System Report for SIGdial MultiLing 2015. http://multiling.iit.demokritos.gr/file/download/1575, 2015. {Online; accessed 06-August-2015}.Google Scholar
- H. Wang, C. Wang, C. Zhai, and J. Han. Learning online discussion structures by conditional random fields. In Proceedings of ACM SIGIR, pages 435--444, 2011. Google ScholarDigital Library
Comments