Skip to main content
SpringerLink
Log in

Up for Debate: Effects of Formal Structure on Argumentation Quality in a Crowdsourcing Platform

  • Conference paper
  • First Online:
Social Computing and Social Media: Experience Design and Social Network Analysis (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12774))

Included in the following conference series:

Abstract

We examined the use of formal structure (more specifically, the Toulmin model and the use of abstraction laddering) in argument assertion templates in a crowdsourcing platform, to determine its effects on argument quality, as rated by other peer contributors. Contrary to our hypotheses, the attempt to add rigor to asserted arguments resulted in a significant decrease in quality across several measures, including the pathos, kairos, and overall level of agreement with the assertion. We found that the way participants voted (a binary outcome of supporting or dissenting) aligned more strongly with whether they agreed with the assertion (regardless of quality) rather than with the quality of the assertion. We provide multiple potential explanations for why the use of the Toulmin model was not a reliable predictor of argument quality in a crowdsourcing application.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Howe, J.: The rise of crowdsourcing. wired, June 2006. https://www.wired.com/2006/06/crowds/

  2. Brabham, D.C.: Crowdsourcing. The MIT Press, Cambridge, MA (2013)

    Book  Google Scholar 

  3. Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. CHI 2011, 1403–1412 (2011)

    Google Scholar 

  4. Kim, J., Sterman, S., Cohen, A.A.B., Bernstein, M.: Mechanical novel: crowdsourcing complex work through reflection and revision. In: ACM Conference on Computer-Supported Cooperative Work and Social Computing (2017)

    Google Scholar 

  5. Chilton, L., Little, G., Edge, D., Weld, D.S., Landay, J.A.: Cascade: crowdsourcing taxonomy creation (Report No. UW-CSE-12–11–02) (2012)

    Google Scholar 

  6. Drapeau, R., Chilton, L.B., Bragg, J., Weld, D.S.: MicroTalk: using argumentation to improve crowdsourcing accuracy. In: Fourth AAAI Conference on Human Computation and Crowdsourcing, AAAI Press (2016)

    Google Scholar 

  7. Hackman, J.R.: Collaborative Intelligence: Using Teams to Solve Hard Problems. Berrett-Koehler Publishers, San Francisco, CA (2011)

    Google Scholar 

  8. Smith, J.B.: Collective Intelligence in Computer-Based Collaboration. Lawrence Erlbaum, Hillsdale, NJ (1994)

    Google Scholar 

  9. Erduran, S., Simon, S., Osborne, J.: TAPping into argumentation: developments in the application of Toulmin’s argument pattern for studying science discourse. Sci. Educ. 88(6), 015–933 (2004). https://doi.org/10.1002/sce.20012

    Article  Google Scholar 

  10. Verheij, B.: Evaluating arguments based on Toulmin’s scheme. In: OSSA Conference Archive, vol. 115, pp. 1–17 (2001). https://scholar.uwindsor.ca/cgi/viewcontent.cgi?article=1700&context=ossaarchive

  11. Herrick, J.A.: Argumentation Understanding and Shaping Arguments. Pearson, New York (1995)

    Google Scholar 

  12. Toulmin, S.: The Uses of Argument. Cambridge University Press, Cambridge (1958)

    Google Scholar 

  13. Spatariu, A., Hartley, K., Bendixen, L.D.: Defining and measuring quality in online discussions. J. Interact. Online Learn. 2(4), 1–15 (2004)

    Google Scholar 

  14. Kim, D., Benbasat, I.: The effects of trust-assuring arguments on consumer trust in Internet stores: application of Toulmin’s model of argumentation. Inf. Syst. Res. 17(3), 286–300 (2006). https://doi.org/10.1287/isre.1060.0093

    Article  Google Scholar 

  15. Boller, G.W., Swasy, J.L., Munch, J.M.: Conceptualizing argument quality via argument structure. Adv. Consum. Res. 17, 321–328 (1990)

    Google Scholar 

  16. Bizup, J.: The uses of Toulmin in composition studies. CCC 61(1), 1–23 (2009)

    Google Scholar 

  17. Tans, O.: The fluidity of warrants: Using the Toulmin model to analyse practical discourse. In: Hitchcock, D., Verheij, B. (eds.) Arguing on the Toulmin Model, pp. 219–230. Springer, Dordrecht (2006)

    Chapter  Google Scholar 

  18. Schultze, U., Avital, M.: Designing interviews to generate rich data for information systems research. Inf. Organ. 21, 1–16 (2011)

    Article  Google Scholar 

  19. Veludo-de-Oliveira, T.M., Ikeda, A.A., Campomar, M.C.: Laddering in the practice of marketing research: barriers and solutions. Qual. Mark. Res. 9(3), 297–306 (2006). https://doi.org/10.1108/13522750610671707

    Article  Google Scholar 

  20. Senge, P.M., Kleiner, A., Roberts, C., Ross, R.B., Smith, B.J.: The Fifth Discipline Fieldbook: Strategies and Tools for Building a Learning Organization. Doubleday, New York, NY (1994)

    Google Scholar 

  21. Barry, C.: How/why Laddering (2010). https://dschool-old.stanford.edu/groups/k12/wiki/afdc3/HowWhy_Laddering.html

  22. Wenzel, J.W.: Three perspectives on argument: Rhetoric, dialectic, logic. In: Trapp, R., Schuetz, J. (eds.) Perspectives on Argumentation: Essays in Honor of Wayne Brockriede, pp. 9–26. Waveland Press, Prospect Heights, IL (1990)

    Google Scholar 

  23. Cohen, D.H.: Evaluating arguments and making meta-arguments. Inf. Logic 21(2), 73–84 (2001). https://doi.org/10.22329/il.v21i2.2238

    Article  Google Scholar 

  24. Rife, M.C.: Ethos, pathos, logos, kairos: using a rhetorical heuristic to mediate digital-survey recruitment strategies. IEEE . Prof. Commun. 53(3), 260–277 (2010). https://doi.org/10.1109/TPC.2010.2052856

  25. Scriven, M.: Reasoning. McGraw-Hill Book Co, New York (1976)

    Google Scholar 

  26. Walton, D.: The new dialectic: Conversational contexts of argument. University of Toronto Press, Toronto (1998)

    Book  Google Scholar 

  27. Nussbaum, E.M.: Argumentation, dialogue theory, and probability modeling: alternative frameworks for argumentation research in education. Educ. Psychol. 46(2), 84–106 (2011). https://doi.org/10.1080/00461520.2011.558816

    Article  Google Scholar 

  28. Winter, S., Krämer, N.C.: Selecting science information in Web 2.0: how source cues, message sidedness, and need for cognition influence users’ exposure to blog posts. J. Comput. Mediat. Commun. 18(1), 80–96 (2012)

    Article  Google Scholar 

  29. Dorton, S.L., Smith, C.M., Upham J.B.: Applying visualization and collective intelligence for rapid group decision making. In: Proceedings of the 2018 International Annual Meeting of the Human Factors and Ergonomics Society, pp. 167–171 (2018). doi: https://doi.org/10.1177/1541931218621039

  30. Repenning, N.P., Kieffer, D., Astor, T.: The most underrated skill in management. MIT Sloan Manag. Rev. 58(3), 39–48 (2017)

    Google Scholar 

  31. Heuer, R.J.: Psychology of Intelligence Analysis. Echo Point Books & Media, Brattleboro, VT (2017)

    Google Scholar 

  32. Killingsworth, M.J.: Rhetorical appeals: a revision. Rhetor. Rev. 24(3), 249–263 (2005)

    Article  Google Scholar 

  33. Hunt, K.: Establishing a presence on the world wide web: a rhetorical approach. Techn. Commun. 43(4), 376–387 (1996)

    MathSciNet  Google Scholar 

  34. Higgins, C., Walker, R.: Ethos, logs, pathos: Strategies of persuasion in social/environmental reports. Acc. Forum 36, 194–208 (2012). https://doi.org/10.1016/j.accfor.2012.02.003

    Article  Google Scholar 

  35. Wachsmuth, H., Stede, M., El Baff, R., Al-Khatib, K., Skeppstedt, M., Stein, B.: Argumentation synthesis following rhetorical strategies. In: Proceedings of the 27th Annual Conference on Computational Linguistics, pp. 3753–3765 (2018). https://www.aclweb.org/anthology/C18-1318

  36. Petty, R.E., Cacioppo, J.T.: Source factors and the elaboration likelihood model of persuasion. Adv. Consum. Res. 11, 668–672 (1984)

    Article  Google Scholar 

  37. Areni, C.S., Lutz, R.J.: The role of argument quality in the elaboration likelihood model. Adv. Consum. Res. 15, 197–203 (1988)

    Google Scholar 

  38. Kotter, J.: Leading Change. Harvard Business Review Press, Boston, MA (2012)

    Google Scholar 

  39. Corner, A., Hahn, U.: Evaluating science arguments: evidence, uncertainty, and argument strength. J. Exp. Psychol. Appl. 15(3), 199–212 (2009). https://doi.org/10.1037/a0016533

    Article  Google Scholar 

  40. Kahneman, D.: Thinking. Fast and Slow. Farrar Straus and Giroux, New York (2011)

    Google Scholar 

  41. Carenini, G.: A task-based framework to evaluate evaluative arguments. Proc. First Int. Conf. Nat. Lang. Gener. 14, 9–16 (2000). https://doi.org/10.3115/1118253.1118256

    Article  Google Scholar 

  42. Sorokin, P.A.: The Crisis of Our Age. Dutton, New York (1941)

    Google Scholar 

  43. Dunbar, N.E., et al.: Fear appeals, message processing cues, and credibility in the websites of violent, ideological, and nonideological groups. J. Comput.-Mediat. Commun. 19, 871–889 (2014). https://doi.org/10.1111/jcc4.12083

    Article  Google Scholar 

  44. Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv. 51(1), 1–39 (2018). https://doi.org/10.1145/0000000.0000000

    Article  Google Scholar 

  45. Rains, S.A., Karmikel, C.D.: Health information-seeking and perceptions of website credibility: Examining web-use orientation, message characteristics, and structural features of websites. Comput. Hum. Behav. 25(2), 544–553 (2009)

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the Office of Naval Research under Contract No. N00014-19-C-1012. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors, and do not necessarily reflect the views of the Office of Naval Research.

Author information

Authors and Affiliations

  1. Sonalysts, Inc., Waterford, CT, 06385, USA

    Stephen L. Dorton, Samantha B. Harper, Glory A. Creed & H. George Banta

Authors
  1. Stephen L. Dorton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samantha B. Harper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Glory A. Creed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. H. George Banta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen L. Dorton .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Towson University, Towson, MD, USA

    Gabriele Meiselwitz

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dorton, S.L., Harper, S.B., Creed, G.A., Banta, H.G. (2021). Up for Debate: Effects of Formal Structure on Argumentation Quality in a Crowdsourcing Platform. In: Meiselwitz, G. (eds) Social Computing and Social Media: Experience Design and Social Network Analysis . HCII 2021. Lecture Notes in Computer Science(), vol 12774. Springer, Cham. https://doi.org/10.1007/978-3-030-77626-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77626-8_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77625-1

  • Online ISBN: 978-3-030-77626-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation