skip to main content
10.1145/3630106.3659030acmotherconferencesArticle/Chapter ViewAbstractPublication PagesfacctConference Proceedingsconference-collections
research-article

Data, Annotation, and Meaning-Making: The Politics of Categorization in Annotating a Dataset of Faith-based Communal Violence

Published: 05 June 2024 Publication History

Abstract

Data annotation is a process of meaning-making and is inherently political. The literature on ethics in data-driven technologies explores these political aspects, primarily focusing on questions of bias and power. This paper argues that the politics of annotation often overemphasize secular and modern values and overlooks faith-based, religious, and spiritual aspects (FRS) in data annotation. This oversight particularly affects the postcolonial regions of the Global South, where FRS are intertwined with people’s everyday experiences and ethics. We conducted a focus group discussion and contextual inquiries with six annotators who annotated a faith-related “violence” dataset from South Asian YouTube content. Our analysis reveals that FRS blindness in data annotation manifests through the politics of achieving objectivity and the “scientific” process of meaning-making. Due to these goals, which are predominantly shaped by Western values, FRS sensitivities are overlooked from the initial stages of data curation through annotation, ultimately leading to a context collapse within the annotation process. Finally, we advocate for the adaptation of FRS sensitivities into the annotation process and data infrastructure, particularly when the dataset clearly pertains to FRS, to promote greater cultural and contextual inclusivity in annotation practices.

References

[1]
Syed Farid Alatas. 2000. Academic dependency in the social sciences: Reflections on India and Malaysia. American Studies International 38, 2 (2000), 80–96.
[2]
Syed Farid Alatas. 2003. Academic dependency and the global division of labour in the social sciences. Current sociology 51, 6 (2003), 599–613.
[3]
Dina Almanea and Massimo Poesio. 2022. ArMIS-the Arabic misogyny and sexism corpus with annotator subjective disagreements. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2282–2291.
[4]
Claude Alvares. 2011. A critique of Eurocentric social science and the question of alternatives. Economic and Political Weekly (2011), 72–81.
[5]
Lefteris Jason Anastasopoulos and Jake Ryland Williams. 2019. A scalable machine learning approach for measuring violent and peaceful forms of political protest participation with social media data. Plos one 14, 3 (2019), e0212834.
[6]
Margherita Antona, George Margetis, Stavroula Ntoa, and Helmut Degen. 2023. Special Issue on AI in HCI., 1723–1726 pages.
[7]
Ananya Bhattacharjee, Dana Kulzhabayeva, Mohi Reza, Harsh Kumar, Eunchae Seong, Xuening Wu, Mohammad Rashidujjaman Rifat, Robert Bowman, Rachel Kornfield, Alex Mariakakis, 2023. Integrating Individual and Social Contexts into Self-Reflection Technologies. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–6.
[8]
Geoffrey C Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences. MIT press.
[9]
John Hedley Brooke. 1991. Science and religion: Some historical perspectives. Cambridge University Press.
[10]
Pete Burnap and Matthew L Williams. 2016. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. EPJ Data science 5 (2016), 1–15.
[11]
Heidi A Campbell. 2012. Understanding the relationship between religion online and offline in a networked society. Journal of the American Academy of Religion 80, 1 (2012), 64–93.
[12]
Mohit Chandra, Manvith Reddy, Shradha Sehgal, Saurabh Gupta, Arun Balaji Buduru, and Ponnurangam Kumaraguru. 2021. " A Virus Has No Religion": Analyzing Islamophobia on Twitter During the COVID-19 Outbreak. In Proceedings of the 32nd ACM conference on hypertext and social media. 67–77.
[13]
Patricia Hill Collins. 2015. The social construction of black feminist thought. In Women, knowledge, and reality. Routledge, 222–248.
[14]
Kate Crawford and Trevor Paglen. 2021. Excavating AI: The politics of images in machine learning training sets. Ai & Society 36, 4 (2021), 1105–1116.
[15]
Veena Das. 2006. Life and Words: Violence and the Descent into the Ordinary. Univ of California Press.
[16]
Veena Das. 2008. Violence, gender, and subjectivity. Annual review of anthropology 37 (2008), 283–299.
[17]
Veena Das 1990. Mirrors of violence: Communities, riots, and survivors in South Asia. (No Title) (1990).
[18]
Jenny L Davis and Nathan Jurgenson. 2014. Context collapse: Theorizing context collusions and collisions. Information, communication & society 17, 4 (2014), 476–485.
[19]
Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2342–2351.
[20]
Avery Dulles. 1984. Faith, Church, and God: Insights from Michael Polanyi. Theological Studies 45, 3 (1984), 537–550.
[21]
Arturo Escobar. 2018. Designs for the pluriverse: Radical interdependence, autonomy, and the making of worlds. Duke University Press.
[22]
John H Evans and Michael S Evans. 2008. Religion and science: Beyond the epistemological conflict narrative. Annu. Rev. Sociol 34 (2008), 87–105.
[23]
Mary Flanagan, Daniel C Howe, and Helen Nissenbaum. 2008. Embodying values in technology: Theory and practice. na.
[24]
Batya Friedman, Peter H Kahn, Alan Borning, and Alina Huldtgren. 2013. Value sensitive design and information systems. Early engagement and new technologies: Opening up the laboratory (2013), 55–95.
[25]
Johan Galtung. 1990. Cultural violence. Journal of peace research 27, 3 (1990), 291–305.
[26]
Clifford Geertz. 2013. Religion as a cultural system. In Anthropological approaches to the study of religion. Routledge, 1–46.
[27]
Barney G Glaser, Anselm L Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory; strategies for qualitative research. Nursing research 17, 4 (1968), 364.
[28]
Banu Gökarıksel. 2009. Beyond the officially sacred: religion, secularism, and the body in the production of subjectivity. Social & Cultural Geography 10, 6 (2009), 657–674.
[29]
David J Gouwens. 1996. Kierkegaard as religious thinker. Cambridge University Press.
[30]
Nitesh Goyal, Ian D Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–28.
[31]
Charles Hirschkind. 2001. Civic virtue and religious reason: An Islamic counterpublic. Cultural Anthropology 16, 1 (2001), 3–34.
[32]
Charles Hirschkind. 2006. The ethical soundscape: Cassette sermons and Islamic counterpublics. Columbia University Press.
[33]
Kori Inkpen, Stevie Chancellor, Munmun De Choudhury, Michael Veale, and Eric PS Baumer. 2019. Where is the human? Bridging the gap between AI and HCI. In Extended abstracts of the 2019 chi conference on human factors in computing systems. 1–9.
[34]
Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E Grinter. 2010. Postcolonial computing: a lens on design and development. In Proceedings of the SIGCHI conference on human factors in computing systems. 1311–1320.
[35]
Alecia Y Jackson and Lisa A Mazzei. 2008. Experience and “I” in autoethnography: A deconstruction. International review of qualitative research 1, 3 (2008), 299–318.
[36]
Willis Jenkins. 2013. Ecologies of grace: Environmental ethics and Christian theology. Oxford University Press.
[37]
Kevin Jesse, Premkumar T Devanbu, and Toufique Ahmed. 2021. Learning type annotation: is big data enough?. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1483–1486.
[38]
Amba Kak. 2020. " The Global South is everywhere, but also always somewhere" National Policy Narratives and AI Justice. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 307–312.
[39]
Shivani Kapania, Alex S Taylor, and Ding Wang. 2023. A hunt for the Snark: Annotator Diversity in Data Practices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–15.
[40]
Gunay Kazimzade and Milagros Miceli. 2020. Biased priorities, biased outcomes: three recommendations for ethics-oriented data annotation practices. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 71–71.
[41]
Neha Kumar and Naveena Karusala. 2021. Braving citational justice in human-computer interaction. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–9.
[42]
Bruno Latour. 2012. We have never been modern. Harvard university press.
[43]
Bruno Latour. 2013. An inquiry into modes of existence. Harvard University Press.
[44]
Anita Lavorgna and Pamela Ugwudike. 2021. The datafication revolution in criminal justice: An empirical exploration of frames portraying data-driven technologies for crime prevention and control. Big Data & Society 8, 2 (2021), 20539517211049670.
[45]
Christopher A Le Dantec, Erika Shehan Poole, and Susan P Wyche. 2009. Values as lived experience: evolving value sensitive design in support of value discovery. In Proceedings of the SIGCHI conference on human factors in computing systems. 1141–1150.
[46]
Clément Le Ludec, Maxime Cornet, and Antonio A Casilli. 2023. The problem with annotation. Human labour and outsourcing between France and Madagascar. Big Data & Society 10, 2 (2023), 20539517231188723.
[47]
Duri Long and Brian Magerko. 2020. What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–16.
[48]
Peter McIlveen. 2008. Autoethnography as a method for reflexive research and practice in vocational psychology. Australian journal of career development 17, 2 (2008), 13–20.
[49]
Philip A Mellor and Chris Shilling. 2014. Re-conceptualising the religious habitus: Reflexivity and embodied subjectivity in global modernity. Culture and Religion 15, 3 (2014), 275–297.
[50]
Milagros Miceli, Julian Posada, and Tianling Yang. 2022. Studying up machine learning data: Why talk about bias when we mean power?Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1–14.
[51]
Milagros Miceli, Martin Schuessler, and Tianling Yang. 2020. Between subjectivity and imposition: Power dynamics in data annotation for computer vision. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–25.
[52]
Nusrat Jahan Mim. 2021. Gospels of modernity: Digital cattle markets, urban religiosity, and secular computing in the global South. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.
[53]
Michael Muller, Melanie Feinberg, Timothy George, Steven J Jackson, Bonnie E John, Mary Beth Kery, and Samir Passi. 2019. Human-centered study of data science work practices. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1–8.
[54]
Ashis Nandy. 1988. The politics of secularism and the recovery of religious tolerance. Alternatives 13, 2 (1988), 177–194.
[55]
Darcia Narvaez, Irene Getz, James R Rest, and Stephen J Thoma. 1999. Individual moral judgment and cultural ideologies.Developmental psychology 35, 2 (1999), 478.
[56]
Helen Nissenbaum. 2001. How computer systems embody values. Computer 34, 3 (2001), 120–119.
[57]
Cathy O’neil. 2017. Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.
[58]
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. 2022. Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions. arXiv preprint arXiv:2205.00415 (2022).
[59]
Michael Polanyi and Harry Prosch. 1975. Meaning. University of Chicago Press.
[60]
Mohammad Rashidujjaman, Ayesha Bhimdiwala, Ananya Bhattacharjee, Amna Batool, Dipto Das, Nusrat Jahan Mim, Abdullah Hasan Safir, Sharifa Sultana, Taslima Akter, C Estelle Smith, 2023. Many Worlds of Ethics: Ethical Pluralism in CSCW. (2023).
[61]
Noopur Raval. 2019. An agenda for decolonizing data science. Spheres: Journal for Digital Cultures5 (2019), 1–6.
[62]
Mohammad Rashidujjaman Rifat, Ashratuz Zavin Asha, Shivesh Jadon, Xinyi Yan, Shion Guha, and Syed Ishtiaque Ahmed. 2024. Combating Islamophobia: Compromise, Community, and Harmony in Mitigating Harmful Online Content. ACM Transactions on Social Computing (2024).
[63]
Mohammad Rashidujjaman Rifat, Firaz Ahmed Peer, Hawra Rabaan, Nusrat Jahan Mim, Maryam Mustafa, Kentaro Toyama, Robert B Markum, Elizabeth Buie, Jessica Hammer, Sharifa Sultana, 2022. Integrating Religion, Faith, and Spirituality in HCI. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–6.
[64]
Mohammad Rashidujjaman Rifat, Toha Toriq, and Syed Ishtiaque Ahmed. 2020. Religion and sustainability: Lessons of sustainable computing from Islamic religious communities. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–32.
[65]
Holmes Rolston. 2006. Science and religion: A critical survey. Number 188. Templeton Foundation Press.
[66]
Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 315–328.
[67]
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[68]
Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021. Do datasets have politics? Disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–37.
[69]
Caroline M Schöpf. 2020. The Coloniality of Global Knowledge Production: Theorizing the Mechanisms of Academic Dependency.Social Transformations: Journal of the Global South 8, 2 (2020).
[70]
Helaine Selin. 2013. Encyclopaedia of the history of science, technology, and medicine in non-westen cultures. Springer Science & Business Media.
[71]
Cheng Feng Shih. 2010. Academic colonialism and the struggle for indigenous knowledge systems in Taiwan.Social Alternatives 29, 1 (2010), 44–47.
[72]
Linda Tuhiwai Smith. 2021. Decolonizing methodologies: Research and indigenous peoples. Bloomsbury Publishing.
[73]
Sharifa Sultana and Syed Ishtiaque Ahmed. 2019. Witchcraft and hci: Morality, modernity, and postcolonial computing in rural bangladesh. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–15.
[74]
Scott Timcke. 2023. Towards a More Comprehensive AI Ethics: How Global South Perspectives Can Enrich Current Approaches to AI Governance. Scott Timcke and Theresa Schültken (2023) Towards a More Comprehensive AI Ethics: How Global South Perspectives Can Enrich Current Approaches to AI Governance, UN Unite Brief (2023).
[75]
Tânia Valente. 2023. Future Visions for a Decolonized Future of HCI. In HCI International 2023 Posters: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part I. Springer Nature, 109.
[76]
Bertie Vidgen and Taha Yasseri. 2020. Detecting weak and strong Islamophobic hate speech on social media. Journal of Information Technology & Politics 17, 1 (2020), 66–78.
[77]
Ding Wang, Shantanu Prabhat, and Nithya Sambasivan. 2022. Whose AI Dream? In search of the aspiration in data annotation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–16.
[78]
Jutta Weber and Bianca Prietl. 2021. AI in the age of technoscience: On the rise of data-driven AI and its epistem-ontological foundations. In The Routledge social science handbook of AI. Routledge, 58–73.
[79]
Susann Wiedlitzka, Gabriele Prati, Rupert Brown, Josh Smith, and Mark A Walters. 2023. Hate in word and deed: the temporal association between online and offline islamophobia. Journal of quantitative criminology 39, 1 (2023), 75–96.
[80]
Sue Wilkinson. 1998. Focus group methodology: a review. International journal of social research methodology 1, 3 (1998), 181–203.
[81]
Angie Zhang, Alexander Boltz, Chun Wei Wang, and Min Kyung Lee. 2022. Algorithmic management reimagined for workers and by workers: Centering worker well-being in gig work. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–20.

Index Terms

  1. Data, Annotation, and Meaning-Making: The Politics of Categorization in Annotating a Dataset of Faith-based Communal Violence

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      FAccT '24: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency
      June 2024
      2580 pages
      ISBN:9798400704505
      DOI:10.1145/3630106
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 05 June 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Faith
      2. Religion
      3. and Spirituality
      4. annotation
      5. communal violence
      6. decolonizing knowledge practices
      7. meaning-making

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      FAccT '24

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 135
        Total Downloads
      • Downloads (Last 12 months)135
      • Downloads (Last 6 weeks)10
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media