skip to main content
10.1145/3411764.3445433acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Can Online Juries Make Consistent, Repeatable Decisions?

Published: 07 May 2021 Publication History

Abstract

A jury of one’s peers is a prominent way to adjudicate disputes and is increasingly used in participatory governance online. The fairness of this approach rests on the assumption that juries are consistent: that the same jury would hand down similar judgments to similar cases. However, prior literature suggests that social influence would instead cause early interactions to cascade into different judgments for similar cases. In this paper, we report an online experiment that changes participants’ pseudonyms as they appear to collaborators, temporarily masking a jury’s awareness that they have deliberated together before. This technique allows us to measure consistency by reconvening the same jury on similar cases. Counter to expectation, juries are equally consistent as individuals, a result that is “good for democracy.” But this consistency arises in part due to group polarization, as consensus develops by hardening initial majority opinions. Furthermore, we find that aggregating groups’ perspectives without deliberation erodes consistency.

References

[1]
Tanja Aitamurto and Helene E Landemore. 2015. Five Design Principles for Crowdsourced Policymaking: Assessing the Case of Crowdsourced Off-Road Traffic Law in Finland. Journal of Social Media for Organizations 2, 1 (2015), 1–19.
[2]
Ali Alkhatib and Michael S Bernstein. 2019. Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[3]
Jennifer Nancy Lee Allen, Antonio Alonso Arechar, Gordon Pennycook, and David Rand. 2020. Scaling up Fact-Checking Using the Wisdom of Crowds. (2020). Unpublished preprint.
[4]
Albert W Alschuler and Andrew G Deiss. 1994. A Brief History of the Criminal Jury in the United States. The University of Chicago Law Review 61, 3 (1994), 867–928.
[5]
Julia Angwin and Hannes Grassegger. 2017. Facebook’s Secret Censorship Rules Protect White Men From Hate Speech But Not Black Children. Retrieved September 3, 2020 from https://www.propublica.org/article/facebook-hate-speech-censorship-internal-documents-algorithms
[6]
Solomon E Asch. 1951. Effects of Group Pressure Upon the Modification and Distortion of Judgments. Organizational Influence Processes(1951), 295–303.
[7]
Federico Ast. 2017. Kleros, a Protocol for a Decentralized Justice System. Retrieved October 31, 2019 from https://medium.com/kleros/kleros-a-decentralized-justice-protocol-for-the-internet-38d596a6300d
[8]
Bence Bago, David G Rand, and Gordon Pennycook. 2020. Fake news, Fast and Slow: Deliberation Reduces Belief in False (But Not True) News Headlines. Journal of Experimental Psychology: General(2020).
[9]
Joshua Becker, Abdullah Almaatouq, and Emőke-Ágnes Horvát. 2020. Network Structures of Collective Intelligence: The Contingent Benefits of Group Discussion. (2020). Working Paper.
[10]
Douglas L Bessette, Robyn S Wilson, and Joseph L Arvai. 2019. Do People Disagree with Themselves? Exploring the Internal Consistency of Complex, Unfamiliar, and Risky decisions. Journal of Risk Research(2019), 1–13.
[11]
Laura W Black, Howard T Welser, Dan Cosley, and Jocelyn M DeGroot. 2011. Self-Governance Through Group Discussion in Wikipedia: Measuring Deliberation in Online Groups. Small Group Research 42, 5 (2011), 595–634.
[12]
John Bone, John Hey, and John Suckling. 1999. Are Groups More (or Less) Consistent than Individuals?Journal of Risk and Uncertainty 18, 1 (1999), 63–81.
[13]
Keith Burghardt, William Rand, and Michelle Girvan. 2019. Inferring Models of Opinion Dynamics from Aggregated Jury Data. PLOS One 14, 7 (2019). https://doi.org/10.1371/journal.pone.0218312
[14]
Moira Burke and Robert Kraut. 2008. Mopping Up: Modeling Wikipedia Promotion Decisions. In Proceedings of the 2008 ACM conference on Computer Supported Cooperative Work. 27–36.
[15]
Hancheng Cao, Vivian Yang, Victor Chen, Yu Jin Lee, Lydia Stone, Mark E Whiting, and Michael S Bernstein. 2020. My Team Will Go On: Differentiating High and Low Viability Teams through Team Interaction. Proceedings of the ACM on Human-Computer Interaction 5 (2020).
[16]
Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for the Analysis of Conversations. Proceedings of SIGDIAL(2020).
[17]
Kaiping Chen. 2020. How Deliberative Designs Empower Citizens’ Voices: A Case Study on Ghana’s Deliberative Poll on Agriculture and the Environment. Public Understanding of Science(2020), 0963662520966742.
[18]
Kevin M Clermont and Theodore Eisenberg. 1991. Trial by Jury or Judge: Transcending Empiricism. Cornell L. Rev. 77(1991), 1124.
[19]
Edward Cohen, Mukund Venkateswaran, Nivedita Sankar, and Chris Callison-Burch. 2020. Turkish Judge: A Peer Evaluation Framework for Crowd Work Appeals. (2020).
[20]
Phil F Culverhouse, Robert Williams, Beatriz Reguera, Vincent Herry, and Sonsoles González-Gil. 2003. Do Experts Make Mistakes? A Comparison of Human and Machine Identification of Dinoflagellates. Marine Ecology Progress Series 247 (2003), 17–25.
[21]
Lee J Curley, Jennifer Murray, Rory MacLean, and Phyllis Laybourn. 2017. Are Consistent Juror Decisions Related to Fast and Frugal Decision Making? Investigating the Relationship Between Juror Consistency, Decision Speed and Cue Utilisation. Medicine, Science and the Law 57, 4 (2017), 211–219.
[22]
Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. 2011. Extraneous Factors in Judicial Decisions. Proceedings of the National Academy of Sciences 108, 17(2011), 6889–6892.
[23]
Dennis J Devine, Laura D Clayton, Benjamin B Dunford, Rasmy Seying, and Jennifer Pryce. 2001. Jury Decision Making: 45 Years of Empirical Research on Deliberating Groups. Psychology, Public Policy, and Law 7, 3 (2001), 622.
[24]
Shari Seidman Diamond, Michael J Saks, and Stephan Landsman. 1998. Juror Judgments About Liability and Damages: Sources of Variability and Ways to Increase Consistency. DePaul L. Rev. 48(1998), 301.
[25]
Facebook. 2020. Oversight Board Bylaws. Retrieved September 3, 2020 from https://about.fb.com/wp-content/uploads/2020/01/Bylaws_v6.pdf
[26]
Jenny Fan and Amy X Zhang. 2020. Digital Juries: A Civics-Oriented Approach to Platform Governance. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
[27]
James Fishkin. 2013. Reviving Deliberative Democracy. J. Attali & al., La Démocratie enrayée (2013), 181–200.
[28]
James Fishkin, Nikhil Garg, Lodewijk Gelauff, Ashish Goel, Kamesh Munagala, Sukolsak Sakshuwong, Alice Siu, and Sravya Yandamuri. 2018. Deliberative Democracy with the Online Deliberation Platform. In The 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2019). HCOMP.
[29]
Joshua T Gyory, Jonathan Cagan, and Kenneth Kotovsky. 2019. Are You Better Off Alone? Mitigating the Underperformance of Engineering Teams During Conceptual Design Through Adaptive Process Management. Research in Engineering Design 30, 1 (2019), 85–102.
[30]
J Richard Hackman and Nancy Katz. 2010. Group Behavior and Performance.(2010).
[31]
Reid Hastie, Steven Penrod, and Nancy Pennington. 1983. Inside the Jury. Harvard University Press.
[32]
Kokil Jaidka, Alvin Zhou, Yphtach Lelkes, Jana Egelhofer, and Sophie Lecheler. 2020. Beyond Anonymity: Network Affordances, under Deindividuation, Improve Social Media Discussion Quality. (2020). Unpublished preprint.
[33]
Mahmood Jasim, Pooya Khaloo, Somin Wadhwa, Amy X Zhang, Ali Sarvghad, and Narges Mahyar. 2020. CommunityClick: Capturing and Reporting Community Feedback from Town Halls to Improve Inclusivity. Proceedings of the 2020 ACM Conference on Computer Supported Cooperative Work and Social Computing(2020).
[34]
Malte F Jung, Nikolas Martelaro, and Pamela J Hinds. 2015. Using Robots to Moderate Team Conflict: the Case of Repairing Violations. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction. 229–236.
[35]
Soomin Kim, Jinsu Eun, Changhoon Oh, Bongwon Suh, and Joonhwan Lee. 2020. Bot in the Bunch: Facilitating Group Chat Discussion by Improving Efficiency and Participation with a Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[36]
Yubo Kou and Bonnie Nardi. 2013. Regulating Anti-Social Behavior on the Internet: The Example of League of Legends. (2013).
[37]
Min Kyung Lee and Su Baykal. 2017. Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 1035–1048.
[38]
Kristina Lerman, Xiaoran Yan, and Xin-Zeng Wu. 2016. The ”Majority Illusion” in Social Networks. PloS One 11, 2 (2016), e0147617.
[39]
Robert J MacCoun. 1987. Getting Inside the Black Box: Toward a Better Understanding of Civil Jury Behavior. Rand Corporation.
[40]
Robert J MacCoun and Tom R Tyler. 1988. The Basis of Citizen’s Perceptions of the Criminal Jury. Law and Human Behavior 12, 3 (1988), 333–352.
[41]
Nancy S Marder. 2005. The Jury Process. Foundation Press.
[42]
Nancy S Marder. 2006. Cyberjuries: A New Role as Online Mock Juries. U. Tol. L. Rev. 38(2006), 239.
[43]
Serge Moscovici and Claude Faucheux. 1972. Social Influence, Conformity Bias, and the Study of Active Minorities. In Advances in Experimental Social Psychology. Vol. 6. Elsevier, 149–202.
[44]
Elisabeth Noelle-Neumann. 1974. The Spiral of Silence a Theory of Public Opinion. Journal of Communication 24, 2 (1974), 43–51.
[45]
Beth S Noveck. 2004. Unchat: Democratic Solution for a Wired World. Democracy Online: The Prospects for Political Renewal Through the Internet (2004), 21–34.
[46]
Vincent E Owhoso, William F Messier, Jr, and John G Lynch, Jr. 2002. Error Detection by Industry-Specialized Teams During Sequential Audit Review. Journal of Accounting Research 40, 3 (2002), 883–900.
[47]
John W Payne, John William Payne, James R Bettman, and Eric J Johnson. 1993. The Adaptive Decision Maker. Cambridge university press.
[48]
Marshall Scott Poole, Michael Holmes, Richard Watson, and Gerardine DeSanctis. 1993. Group Decision Support Systems and Group Communication: A Comparison of Decision Making in Computer-Supported and Nonsupported Groups. Communication Research 20, 2 (1993), 176–213.
[49]
Robert T Roper. 1980. Jury Size and Verdict Consistency: ”A Line Has to be Drawn Somewhere”?Law and Society Review(1980), 977–995.
[50]
Niloufar Salehi and Michael S Bernstein. 2018. Hive: Collective Design Through Network Rotation. Proceedings of the ACM on Human-Computer Interaction 2, CSCW(2018), 1–26.
[51]
Niloufar Salehi, Lilly C Irani, Michael S Bernstein, Ali Alkhatib, Eva Ogbe, and Kristy Milland. 2015. We are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers. In CHI ’15: Proceedings of the 33rd annual ACM Conference on Human factors in computing systems. 1621–1630.
[52]
Jeffrey A Segal. 1986. Supreme Court Justices as Human Decision Makers: An Individual-Level Analysis of the Search and Seizure cases. The Journal of Politics 48, 4 (1986), 938–955.
[53]
Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. 2018. Design and Analysis of the NIPS 2016 Review Process. The Journal of Machine Learning Research 19, 1 (2018), 1913–1946.
[54]
Jae-Young Son, Apoorva Bhandari, and Oriel FeldmanHall. 2019. Crowdsourcing Punishment: Individuals Reference Group Preferences to Inform Their Own Punitive Decisions. Scientific Reports 9, 1 (2019), 1–15.
[55]
Garold Stasser and William Titus. 1985. Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion.Journal of Personality and Social Psychology 48, 6(1985), 1467.
[56]
Cass R Sunstein. 1999. The Law of Group Polarization. University of Chicago Law School, John M. Olin Law & Economics Working Paper91(1999).
[57]
Cass R Sunstein. 2000. Deliberative Trouble? Why Groups Go to Extremes. The Yale Law Journal 100, 71 (2000), 71–119.
[58]
Cass R Sunstein and Reid Hastie. 2014. Making Dumb Groups Smarter. Harvard Business Review 92, 12 (2014), 90–98.
[59]
James Surowiecki. 2005. The Wisdom of Crowds. Anchor.
[60]
Ahmed E Taha. 2004. How Panels Affect Judges: Evidence from United States District Courts. U. Rich. L. Rev. 39(2004), 1235.
[61]
Linda Thorne, Dawn W. Massey, and Joanne Jones. 2004. An Investigation of Social Influence: Explaining the Effect of Group Discussion on Consensus in Auditors’ Ethical Reasoning.Business Ethics Quarterly 14, 3 (2004), 525–551. https://doi.org/10.5840/beq200414321
[62]
Nina Totenberg. 2019. Supreme Court Sees 2 Similar Death Penalty Questions Very Differently. (2019). https://www.npr.org/2019/03/30/708238203/supreme-court-sees-2-similar-death-penalty-questions-very-differently
[63]
Mark E Whiting, Allie Blaising, Chloe Barreau, Laura Fiuza, Nik Marda, Melissa Valentine, and Michael S Bernstein. 2019. Did It Have To End This Way? Understanding the Consistency of Team Fracture. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–23.
[64]
Mark E Whiting, Irena Gao, Michelle Xing, Junior Diarrassouba N’Godjigui, Tonya Nguyen, and Michael S Bernstein. 2020. Parallel Worlds: Repeated Initializations of the Same Team To Improve Team Viability. Proceedings of the ACM on Human-Computer Interaction 4 (2020).
[65]
Mark E Whiting, Grant Hugh, and Michael S Bernstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 197–206.
[66]
Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Malone. 2010. Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science 330, 6004 (2010), 686–688.
[67]
Amy X Zhang, Grant Hugh, and Michael S Bernstein. 2020. PolicyKit: Building Governance in Online Communities. UIST ’20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (2020).
[68]
Sharon Zhou, Melissa Valentine, and Michael S Bernstein. 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.

Cited By

View all
  • (2024)"Just a tool, until you stab someone with it": Exploring Reddit Users' Questions and Advice on the Legality of Port ScansProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688469(322-336)Online publication date: 30-Sep-2024
  • (2023)Measuring User-Moderator Alignment on r/ChangeMyViewProceedings of the ACM on Human-Computer Interaction10.1145/36100777:CSCW2(1-36)Online publication date: 4-Oct-2023
  • (2023)Diverse Perspectives Can Mitigate Political Bias in Crowdsourced Content ModerationProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594080(1280-1291)Online publication date: 12-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
10862 pages
ISBN:9781450380966
DOI:10.1145/3411764
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deliberation
  2. Governance
  3. Groups
  4. Individuals
  5. Juries

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

CHI '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)110
  • Downloads (Last 6 weeks)5
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)"Just a tool, until you stab someone with it": Exploring Reddit Users' Questions and Advice on the Legality of Port ScansProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688469(322-336)Online publication date: 30-Sep-2024
  • (2023)Measuring User-Moderator Alignment on r/ChangeMyViewProceedings of the ACM on Human-Computer Interaction10.1145/36100777:CSCW2(1-36)Online publication date: 4-Oct-2023
  • (2023)Diverse Perspectives Can Mitigate Political Bias in Crowdsourced Content ModerationProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594080(1280-1291)Online publication date: 12-Jun-2023
  • (2022)A Comparative Evaluation of Interventions Against Misinformation: Augmenting the WHO ChecklistProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517717(1-21)Online publication date: 29-Apr-2022
  • (2022)The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk EvaluationProceedings of the ACM Web Conference 202210.1145/3485447.3512239(1685-1696)Online publication date: 25-Apr-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media