Skip to main content
SpringerLink
Log in

Argument discovery via crowdsourcing

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

The amount of controversial issues being discussed on the Web has been growing dramatically. In articles, blogs, and wikis, people express their points of view in the form of arguments, i.e., claims that are supported by evidence. Discovery of arguments has a large potential for informing decision-making. However, argument discovery is hindered by the sheer amount of available Web data and its unstructured, free-text representation. The former calls for automatic text-mining approaches, whereas the latter implies a need for manual processing to extract the structure of arguments. In this paper, we propose a crowdsourcing-based approach to build a corpus of arguments, an argumentation base, thereby mediating the trade-off of automatic text-mining and manual processing in argument discovery. We develop an end-to-end process that minimizes the crowd cost while maximizing the quality of crowd answers by: (1) ranking argumentative texts, (2) pro-actively eliciting user input to extract arguments from these texts, and (3) aggregating heterogeneous crowd answers. Our experiments with real-world datasets highlight that our method discovers virtually all arguments in documents when processing only 25% of the text with more than 80% precision, using only 50% of the budget consumed by a baseline algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. http://www.whyichoose.org/vaccinesafety.html.

References

  1. McAfee, A., Brynjolfsson, E.: Big data: the management revolution. Harv. Bus. Rev. 90, 60–66 (2012)

    Google Scholar 

  2. Yang, Z., Li, Y., Cai, J., Nyberg, E.: Quads: question answering for decision support. In: SIGIR, pp. 375–384 (2014)

  3. Yuan, T., Moore, D., Grierson, A.: A human-computer debating system and its dialogue strategies. Int. J. Intell. Syst. 22, 133–156 (2007)

    Article  Google Scholar 

  4. Dzindolet, M.T., Peterson, S.A., Pomranky, R.A., Pierce, L.G., Beck, H.P.: The role of trust in automation reliance. In: IJMMS pp. 697–718 (2003)

  5. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: CSCW, pp. 241–250 (2000)

  6. Ibm watson. http://researcher.ibm.com/researcher/view_group.php?id=5443

  7. Moens, M.F., Boiy, E., Palau, R.M., Reed, C.: Automatic detection of arguments in legal texts. In: ICAIL, pp. 225–230 (2007)

  8. Palau, R., Moens, M.: Argumentation mining: the detection, classification and structure of arguments in text. In: ICAIL, pp. 98–107 (2009)

  9. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast: Evaluating non-expert annotations for natural language tasks. In: EMNLP, pp. 254–263 (2008)

  10. Freeley, A.J., Steinberg, D.L.: Argumentation and Debate. Springer, Berlin (2013)

    Google Scholar 

  11. Aharoni, E., Polnarov, A., Lavee, T., Hershcovich, D.: A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In: ACL, pp. 64–68 (2014)

  12. Besnard, P., Hunter, A.: Elements of Argumentation, vol. 47. MIT press, Cambridge (2008)

    Book  Google Scholar 

  13. Sycara, K.: Persuasive argumentation in negotiation. In: Theory and Decision, pp. 203–242 (1990)

  14. Werder, A.: Argumentation rationality of management decisions. In: Organization Science, pp. 672–690 (1999)

  15. Mochales, R., Ieven, A.: Creating an argumentation corpus: do theories apply to real arguments?: a case study on the legal argumentation of the echr. In: ICAIL, pp. 21–30 (2009)

  16. Nguyen, H.V., Litman, D.J.: Extracting argument and domain words for identifying argument components in texts. In: ArgMining, pp. 22–28 (2015)

  17. Wolf, F., Gibson, E.: Coherence in Natural Language: Data Structures and Applications. MIT Press, Cambridge (2006)

    Google Scholar 

  18. Feng, V.W., Hirst, G.: A linear-time bottom-up discourse parser with constraints and post-editing. In: ACL, pp. 511–521 (2014)

  19. Cabrio, E., Villata, S.: Natural language arguments: a combined approach. ECAI 242, 205–210 (2012)

    MATH  Google Scholar 

  20. Dagan, I., Dolan, B., Magnini, B., Roth, D.: Recognizing textual entailment: Rational, evaluation and approaches. Nat. Lang. Engi. 15(4), i–xvii (2009)

  21. Horton, J.J., Chilton, L.B.: The labor economics of paid crowdsourcing. In: EC, pp. 209–218 (2010)

  22. Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., Singer, Y.: Online passive-aggressive algorithms. In: JMLR, 551–585 (2006)

  23. Settles, B.: Active learning literature survey. In: Technical Report. UW-Madison (2010)

  24. Prechelt, L.: Early stopping-but when? In: Neural Networks: Tricks of the Trade, pp. 55–69. Springer (1998)

  25. Li, X., Dong, X.L., Lyons, K., Meng, W., Srivastava, D.: Truth finding on the deep web: is the problem solved? In: VLDB, pp. 97–108 (2013)

  26. Tan, P.N., Steinbach, M., Kumar, V., et al.: Introduction to data mining, vol. 1. Pearson Education India, London (2006)

    Google Scholar 

  27. Drosou, M., Pitoura, E.: Disc diversity: result diversification based on dissimilarity and coverage. In: VLDB, pp. 13–24 (2012)

  28. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., Horvitz, E.: Human computation tasks with global constraints. In: CHI, pp. 217–226 (2012)

  29. Eickhoff, C., de Vries, A.: How crowdsourcable is your task. In: CSDM, pp. 11–14 (2011)

  30. Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: CHI, pp. 453–456 (2008)

  31. Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: CSCW, pp. 1003–1012 (2012)

  32. von Ahn, L.: Human computation. In: DAC, pp. 418 –419 (2009)

  33. Mikheev, A.: Tagging sentence boundaries. In: NAACL, pp. 264–271 (2000)

  34. Banko, M., Cafarella, M.J., Soderland, S., Broadhead, M., Etzioni, O.: Open information extraction for the web. IJCAI 7, 2670–2676 (2007)

    Google Scholar 

  35. Ernst, P., Meng, C., Siu, A., Weikum, G.: Knowlife: a knowledge graph for health and life sciences. In: ICDE, pp. 1254–1257 (2014)

  36. Hung, N.Q.V., Tam, N.T., Tran, L.N., Aberer, K.: An evaluation of aggregation techniques in crowdsourcing. In: WISE, pp. 1–15 (2013)

  37. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using EM. J. R. Stat. Soc., 20–28 (1979)

  38. Kschischang, F.R., Frey, B.J., Loeliger, H.A.: Factor graphs and the sum-product algorithm. TIT 47, 498–519 (2001)

    MathSciNet  MATH  Google Scholar 

  39. Gao, J., Li, Q., Zhao, B., Fan, W., Han, J.: Truth discovery and crowdsourcing aggregation: a unified perspective. VLDB 8, 2048–2049 (2015)

    Google Scholar 

  40. Zhang, C., Ré, C.: Towards high-throughput gibbs sampling at scale: a study across storage managers. In: SIGMOD, pp. 397–408 (2013)

  41. Oyama, S., Baba, Y., Sakurai, Y., Kashima, H.: Accurate integration of crowdsourced labels using workers’ confidence scores. In: IJCAI, pp. 2554–2560 (2013)

  42. Krippendorff, K.: Content Analysis: An Introduction to its Methodology. Sage, Thousand Oaks (2012)

    Google Scholar 

  43. Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: CIKM, pp. 1941–1944 (2011)

  44. Argdiscovery. http://argdiscovery.github.io/

  45. Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: HCOMP, pp. 64–67 (2010)

  46. Zhou, D., Liu, Q., Platt, J.C., Meek, C.: Aggregating ordinal labels from crowds by minimax conditional entropy. In: ICML, pp. 262–270 (2014)

  47. Welinder, P., Perona, P.: Online crowdsourcing: rating annotators and obtaining cost-effective labels. In: CVPRW, pp. 25–32 (2010)

  48. Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: UIST, pp. 313–322 (2010)

  49. Marcus, A., Wu, E., Karger, D., Madden, S., Miller, R.: Human-powered sorts and joins. In: VLDB, pp. 13–24 (2011)

  50. Amsterdamer, Y., Grossman, Y., Milo, T., Senellart, P.: Crowd mining. In: SIGMOD, pp. 241–252 (2013)

  51. Lee, K., Caverlee, J., Webb, S.: The social honeypot project: protecting online communities from spammers. In: WWW, pp. 1139–1140 (2010)

  52. Parameswaran, A., Sarma, A.D., Garcia-Molina, H., Polyzotis, N., Widom, J.: Human-assisted graph search: it’s okay to ask questions. In: VLDB, pp. 267–278 (2011)

  53. Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: WWW, pp. 469–478 (2012)

  54. Liu, Q., Peng, J., Ihler, A.T.: Variational inference for crowdsourcing. In: NIPS, pp. 692–700 (2012)

  55. Wick, M., McCallum, A., Miklau, G.: Scalable probabilistic databases with factor graphs and mcmc. In: VLDB, pp. 794–804 (2010)

  56. Mozafari, B., Sarkar, P., Franklin, M., Jordan, M., Madden, S.: Scaling up crowd-sourcing to very large datasets: a case for active learning. In: VLDB, pp. 125–136 (2014)

  57. Azaria, A., Aumann, Y., Kraus, S.: Automated agents for reward determination for human work in crowdsourcing. In: AAMAS, pp. 1–22 (2013)

  58. Faradani, S., Hartmann, B., Ipeirotis, P.G.: What’s the right price? Pricing tasks for finishing on time. In: HCOMP, pp. 26–31 (2011)

  59. Jeffery, S.R., Franklin, M.J., Halevy, A.Y.: Pay-as-you-go user feedback for dataspace systems. In: SIGMOD, pp. 847–860 (2008)

  60. Nguyen, Q.V.H., Nguyen, T.T., Miklós, Z., Aberer, K., Gal, A., Weidlich, M.: Pay-as-you-go reconciliation in schema matching networks. In: ICDE, pp. 220–231 (2014)

  61. Dong, X.L., Naumann, F.: Data fusion: resolving data conflicts for integration. In: VLDB, pp. 1654–1655 (2009)

  62. Galland, A., Abiteboul, S., Marian, A., Senellart, P.: Corroborating information from disagreeing views. In: WSDM, pp. 131–140 (2010)

  63. Wu, Y., Agarwal, P.K., Li, C., Yang, J., Yu, C.: Toward computational fact-checking. In: VLDB, pp. 589–600 (2014)

  64. Wang, D., Kaplan, L., Le, H., Abdelzaher, T.: On truth discovery in social sensing: a maximum likelihood estimation approach. In: IPSN, pp. 233–244 (2012)

  65. Pasternack, J., Roth, D.: Latent credibility analysis. In: WWW, pp. 1009–1020 (2013)

  66. Pochampally, R., Das Sarma, A., Dong, X.L., Meliou, A., Srivastava, D.: Fusing data with correlations. In: SIGMOD, pp. 433–444 (2014)

  67. Zhao, B., Rubinstein, B.I., Gemmell, J., Han, J.: A bayesian approach to discovering truth from conflicting sources for data integration. In: VLDB, pp. 550–561 (2012)

  68. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A.K., Webber, B.L.: The penn discourse treebank 2.0. In: LREC (2008)

  69. Mullen, T., Malouf, R.: A preliminary investigation into sentiment analysis of informal political discourse. In: AAAI, pp. 159–162 (2006)

  70. Anand, P., Walker, M., Abbott, R., Tree, J.E.F., Bowmani, R., Minor, M.: Cats rule and dogs drool!: classifying stance in online debate. In: ACL-WASSA, pp. 1–9 (2011)

  71. Hearst, M.A., Dumais, S.T., Osman, E., Platt, J., Scholkopf, B.: Support vector machines. In: EXPERT pp. 18–28 (1998)

  72. McCallum, A., Nigam, K., et al.: A comparison of event models for naive bayes text classification. AAAI-TEXTCAT 752, 41–48 (1998)

    Google Scholar 

  73. Dumais, S.T.: Latent semantic analysis. In: ARIST pp. 188–230 (2004)

  74. Bex, F., Prakken, H., Reed, C., Walton, D.: Towards a formal account of reasoning about evidence: argumentation schemes and generalisations. In: AIL pp. 125–165 (2003)

  75. Walton, D., Reed, C.: Argumentation schemes and defeasible inferences. In: ECAI-CMNA, pp. 11–20 (2002)

  76. Mitray, M., Singhalz, A., Buckleyyy, C.: Automatic text summarization by paragraph extraction. Compare 22215, 26 (1997)

    Google Scholar 

  77. Nenkova, A., McKeown, K.: A survey of text summarization techniques. In: Mining Text Data, pp. 43–76 (2012)

  78. Deshpande, O., Lamba, D.S., Tourn, M., Das, S., Subramaniam, S., Rajaraman, A., Harinarayan, V., Doan, A.: Building, maintaining, and using knowledge bases: a report from the trenches. In: SIGMOD, pp. 1209–1220 (2013)

  79. Freebase. http://www.freebase.com

  80. Yago. http://www.mpi-inf.mpg.de/yago

  81. Dbpedia. http://www.dbpedia.org

  82. DeRose, P., Shen, W., Chen, F., Doan, A., Ramakrishnan, R.: Building structured web community portals: a top-down, compositional, and incremental approach. In: VLDB, pp. 399–410 (2007)

  83. Hung, N.Q.V., Thang, D.C., Weidlich, M., Aberer, K.: Minimizing efforts in validating crowd answers. In: SIGMOD, pp. 999–1014 (2015)

  84. Brants, T., Chen, F., Tsochantaridis, I.: Topic-based document segmentation with probabilistic latent semantic analysis. In: CIKM, pp. 211–218 (2002)

  85. Hearst, M.A.: Texttiling: segmenting text into multi-paragraph subtopic passages. Comput. Linguist. 23, 33–64 (1997)

    Google Scholar 

  86. Dung, P.M.: On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artif. Intell. 77, 321–357 (1995)

  87. Nguyen, H.Q.V., Luong, X.H., Miklós, Z., Quan, T.T., Aberer, K.: Collaborative schema matching reconciliation. In: CoopIS, pp. 222–240 (2013)

  88. Li, Y., Li, Q., Gao, J., Su, L., Zhao, B., Fan, W., Han, J.: On the discovery of evolving truth. In: KDD, pp. 675–684 (2015)

  89. Chiang, Y.H., Doan, A., Naughton, J.F.: Modeling entity evolution for temporal record matching. In: SIGMOD, pp. 1175–1186 (2014)

  90. Xu, L., Yan, P., Chang, T.: Best first strategy for feature selection. In: ICPR, pp. 706–708 (1988)

Download references

Author information

Authors and Affiliations

  1. The University of Queensland, Brisbane, Australia

    Quoc Viet Hung Nguyen, Hongzhi Yin & Xiaofang Zhou

  2. École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

    Chi Thang Duong, Thanh Tam Nguyen & Karl Aberer

  3. Humboldt-Universität zu Berlin, Berlin, Germany

    Matthias Weidlich

  4. Macau University of Science and Technology, Macau, China

    Xiaofang Zhou

Authors
  1. Quoc Viet Hung Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chi Thang Duong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thanh Tam Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matthias Weidlich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Karl Aberer
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongzhi Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiaofang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quoc Viet Hung Nguyen.

Appendix: Study on textual features

Appendix: Study on textual features

Setup To identify textual features that hint at arguments, we conducted a preliminary study. For this study, we collected 500 documents by querying common search engines such as Bing with 5 keywords from the domains of health, society and economics. The retrieved documents have been segmented into around 4000 paragraphs. Then, five experts assessed for each paragraph whether it contains an argument. For the 193 paragraphs that contain arguments, we then calculated the values for a set of features that are commonly used in argumentation mining and NLP [8]. For this dataset, we selected the features that turned out to be good predictors for paragraphs that contain arguments. Following a best-first selection strategy [90], we ended up with the following features.

Lexical features Lexical features deal with the words or vocabulary of a language. The following lexical information of a paragraph turned out to be useful to classify its argumentative nature.

Thematic words the most frequent words (ignoring common stop-words, such as connectives and articles) in a document and thus paragraph are considered to be thematic words. Selecting a small number of thematic words that are particularly relevant to the keyword, this feature is defined as the frequency counts of thematic words. The intuition of this feature is that a paragraph containing the most important keywords should express the argumentative point of view of the writer.

Example 9

Consider the following article regarding vaccineFootnote 1, there are various words that appear in high frequency in the documents such as vaccine, shots, disease, autism, health. This means these words discuss the theme of the documents. Among the paragraphs in this document, the following paragraph contains many thematic words, which shows that it may contains an argument:

(S1) Yes. (S2) Vaccines are safe. (S3) In fact, experts including American Academy of Pediatrics, the Institute of Medicine, and the World Health Organization agree that vaccines are even safer than vitamins. (S4) Millions of children and adults are vaccinated every year safely. (S5) Thousands of people take part in clinical trials to test a vaccine before it is licensed by the Food and Drug Administration (FDA). (S6) After it’s licensed, the Vaccine Adverse Events Reporting System (VAERS) helps track any health effect that happens hours, days, weeks, or even months later. (S7) Anyone can report a possible side-effect so that it can be studied. (S8) This monitoring helps ensure vaccines are safe. (S9) To learn more about vaccine safety from the Centers for Disease Control and Prevention, visit the CDC vaccine safety page.

Segment \(S_2\) is likely to be a claim as the segment is the second sentence in the paragraph and the keyword vaccine is the subject. Segment \(S_3\) may be an evidence as it contains evidence-related word (fact) and names (American Academy of Pediatrics, Institute of Medicine, World Health Organization). Segment \(S_4\) and \(S_5\) can also be evidence as they contains numbers (millions, thousands).

Evidence-related words A paragraph containing many evidence-related words, such as numbers, citations, and cue phrases (e.g., ‘because of’) is likely to contain arguments. This feature, thus, is defined as the occurrence count of such evidence indicators.

Example 10

(S1) Processed foods destroy your mind. (S2) If you suffer from chronic bouts of brain ‘fog,’or have difficulty concentrating and thinking normally, chances are your diet has something to do with it. (S3) And a recent study out of Oxford University lends credence to this possibility, having found that junk food consumption can cause people to become angry and irritable.(S4) Nutrient-dense whole foods, on the other hand, can help level out your mood, sustain your energy levels, and leave you feeling calmer and more collected.

Table 5 Features and example values
Full size table

This paragraph may contain an argument as an evidence-related keyword (study) is available in segment \(S_3\) of the paragraph. In addition, the keyword processed foods appears as a subject of the first sentence, which shows that this sentence is likely to be a claim.

Prototypical words Prototypical words are lexical expressions used to formulate arguments, for instance, ‘argue’ or ‘believe.’ As part of our preliminary study, we learnt a list of 97 prototypical words. Each prototypical word is associated with a weight, indicating the likelihood of a paragraph containing the given word to include an argument. This weight is determined based on the relative frequency of the word in the training data and in the overall set of documents. Again, the respective feature is the occurrence count of the prototypical words, normalized by their respective weight.

Example 11

(S1) Believe it or not, almost all the food that you eat, even the foods ‘made from scratch,’ have actually been processed. (S2) According to an article published in the journal, Advances in Nutrition, any food that has been subject to washing, cleaning, milling, cutting, chopping, heating, pasteurizing, blanching, cooking, canning, freezing, mixing, and packaging that alter the food from its natural state is considered a ‘processed food.’

The above paragraph contains an argument as segment \(S_1\) which contains a prototypical word (believe). This prototypical word signifies that it may be a claim. In addition, segment \(S_2\) provides information coming from a reliable source which is the Advances in Nutrition journal

Syntactical features Syntactical features refer to the text structure and capture local relations between words within a sentence. Our preliminary study identified the following syntactical features:

  • Part-of-speech Words in a sentence may be classified into different part-of-speech, such as nouns, verbs and adjectives. Intuitively, this feature exploits the fact that, if the keyword appears to be a subject or object of a sentence, it is likely that the sentence is a relevant claim or evidence. Technically, this feature is defined as the occurrence count of the keyword in a paragraph either as a subject or as an object.

  • Parse structure A sentence can be parsed into a tree-like structure that captures syntactical relations between the phrases within that sentence. In the parse structure, the relation between the phrases is determined by the head word, which is an indicator of the mentioned topic. We use the appearance of the keyword as the head word of phrases in a paragraph as a classification feature indicating whether the paragraph contains arguments.

For illustration, we consider the following paragraph that is relevant for the topic of ‘processed food’ discussed above.

Example 12

Furthermore, the engineering behind processed food makes it virtually addictive. A 2009 study by the Scripps Research Institute indicates that overconsumption of processed food triggers addiction-like neuroaddictive responses in the brain, making it harder to trigger the release of dopamine. In other words, the more processed food we eat, the more we need to give us pleasure; thus the report suggests that the same mechanisms underlie drug addiction and obesity.

This paragraph illustrates various of the features that hint at arguments, see also Table 5. Examples include evidence-related words, such as ‘study’ and prototypical words, such as ‘indicate.’ In addition, thematic words such as ‘addictive’ render the paragraph important for argumentation mining. Further, the keyword ‘processed foods’ appears several times, e.g., in the first sentence, part-of-speech tagging identifies the keyword as the subject of a sentence.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, Q.V.H., Duong, C.T., Nguyen, T.T. et al. Argument discovery via crowdsourcing. The VLDB Journal 26, 511–535 (2017). https://doi.org/10.1007/s00778-017-0462-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-017-0462-9

Keywords

Search

Navigation