skip to main content
10.1145/3589334.3645519acmconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article
Open access

SymLearn: A Symbiotic Crowd-AI Collective Learning Framework to Web-based Healthcare Policy Adherence Assessment

Published: 13 May 2024 Publication History

Abstract

This paper develops a symbiotic human-AI collective learning framework that explores the complementary strengths of both AI and crowdsourced human intelligence to address a novel Web-based healthcare-policy-adherence assessment (WebHA) problem. In particular, the objective of the WebHA problem is to automatically assess people's public health policy adherence during emergent global health crisis events (e.g., COVID-19, MonkeyPox) by exploring massive social media imagery data. Recent advances in human-AI systems exhibit a significant potential in addressing the intricate imagery-based classification problems like WebHA by leveraging the collective intelligence of both humans and AI. This paper aims to address the limitation of existing human-AI systems that often rely heavily on human intelligence to improve AI model performance while overlooking the fact that humans themselves can be fallible and prone to errors. To address the above limitation, this paper develops SymLearn, a symbiotic human-AI co-learning framework that leverages human intelligence to troubleshoot and fine-tune the AI model while using AI models to guide human crowd workers to reduce the inherent human errors in their labels. Extensive experiments on two real-world WebHA applications show that SymLearn clearly outperforms the state-of-the-art baselines by improving WebHA performance and reducing crowd response delay.

Supplemental Material

MP4 File
Supplemental video

References

[1]
Alaa Awad Abdellatif, Lutfi Samara, Amr Mohamed, Abdulla Al-Ali, Aiman Erbad, and Mohsen Guizani. 2019. Compress or Interfere?. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--6.
[2]
Ebtesam Alomari, Iyad Katib, Aiiad Albeshri, Tan Yigitcanlar, and Rashid Mehmood. 2021. Iktishaf: a big data tool with automatic labeling for road traffic social sensing and event detection using distributed machine learning. Sensors 21, 9 (2021), 2993.
[3]
Noor Awad, Neeratyoy Mallik, and Frank Hutter. 2021. DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization. arXiv preprint arXiv:2105.09821 (2021).
[4]
Panagiotis Barmpoutis, Kosmas Dimitropoulos, Kyriaki Kaza, and Nikos Grammalidis. 2019. Fire detection from images using faster R-CNN and multidimensional texture analysis. In ICASSP 2019--2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 8301--8305.
[5]
Jobie Budd, Benjamin S Miller, Erin M Manning, Vasileios Lampos, Mengdie Zhuang, Michael Edelstein, Geraint Rees, Vincent C Emery, Molly M Stevens, Neil Keegan, et al. 2020. Digital technologies in the public-health response to COVID-19. Nature medicine 26, 8 (2020), 1183--1192.
[6]
CDC. 2021. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/ masks.html
[7]
CDC. 2021. https://www.cdc.gov/coronavirus/2019-ncov/community/tribal/social-distancing.html
[8]
Chacha Chen, Junjie Liang, Fenglong Ma, Lucas Glass, Jimeng Sun, and Cao Xiao. 2021. Unite: Uncertainty-based health risk prediction leveraging multi-sourced data. In Proceedings of the Web Conference 2021. 217--226.
[9]
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. 2023. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675 (2023).
[10]
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. 2022. Davit: Dual attention vision transformers. In European Conference on Computer Vision. Springer, 74--92.
[11]
Tom Edixhoven, Sihang Qiu, Lucie Kuiper, Olivier Dikken, Gwennan Smitskamp, and Ujwal Gadiraju. 2021. Improving reactions to rejection in crowdsourcing through self-reflection. In 13th ACM Web Science Conference 2021. 74--83.
[12]
Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning. PMLR, 1437--1446.
[13]
Mirta Galesic, Wändi Bruine de Bruin, Jonas Dalege, Scott L Feld, Frauke Kreuter, Henrik Olsson, Drazen Prelec, Daniel L Stein, and Tamara van Der Does. 2021. Human social sensing is an untapped resource for computational social science. Nature 595, 7866 (2021), 214--222.
[14]
Meric Altug Gemalmaz and Ming Yin. 2021. Accounting for Confirmation Bias in Crowdsourced Label Aggregation. In IJCAI. 1729--1735.
[15]
RA Gilyazev and D Yu Turdakov. 2018. Active learning and crowdsourcing: A survey of optimization methods for data labeling. Programming and Computer Software 44 (2018), 476--491.
[16]
Michael H Haischer, Rachel Beilfuss, Meggie Rose Hart, Lauren Opielinski, David Wrucke, Gretchen Zirgaitis, Toni D Uhrich, and Sandra K Hunter. 2020. Who is wearing a mask? Gender-, age-, and location-related differences during the COVID-19 pandemic. PloS one 15, 10 (2020), e0240785.
[17]
Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. 2019. Bootstrapping upper confidence bound. Advances in neural information processing systems 32 (2019).
[18]
Haiyan Hao and Yan Wang. 2020. Leveraging multimodal social media data for rapid disaster damage assessment. International Journal of Disaster Risk Reduction 51 (2020), 101760.
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[20]
Mehdi Hosseini, Ingemar J Cox, Nata?a Mili?-Frayling, Gabriella Kazai, and Vishwa Vinay. 2012. On aggregating labels from multiple crowd workers to infer relevance of documents. In Advances in Information Retrieval: 34th European Conference on IR Research, ECIR 2012, Barcelona, Spain, April 1--5, 2012. Proceedings 34. Springer, 182--194.
[21]
Xiao Hu, Haobo Wang, Anirudh Vegesana, Somesh Dube, Kaiwen Yu, Gore Kao, Shuo-Han Chen, Yung-Hsiang Lu, George K Thiruvathukal, and Ming Yin. 2020. Crowdsourcing detection of sampling biases in image datasets. In Proceedings of The Web Conference 2020. 2955--2961.
[22]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In CVPR, Vol. 1. 3.
[23]
Yandao Huang, Wenqiang Chen, Hongjie Chen, Lu Wang, and Kaishun Wu. 2019. G-fall: device-free and training-free fall detection with geophones. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--9.
[24]
Michael N Katehakis and Arthur F Veinott Jr. 1987. The multi-armed bandit problem: decomposition and computation. Mathematics of Operations Research 12, 2 (1987), 262--268.
[25]
Ding Li, Wenzhong Li, Xiaoliang Wang, Cam-Tu Nguyen, and Sanglu Lu. 2019. Activetracker: Uncovering the trajectory of app activities over encrypted internet traffic streams. In 2019 16th Annual IEEE international conference on sensing, communication, and networking (SECON). IEEE, 1--9.
[26]
Lisha Li and Kevin Jamieson. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research 18 (2018), 1--52.
[27]
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research 18, 185 (2018), 1--52. http: //jmlr.org/papers/v18/16--558.html
[28]
Danielle McDuffie. 2019. Using Amazon's mechanical Turk: benefits, drawbacks, and suggestions. APS Observer 32, 2 (2019).
[29]
Virginia Negri, Dario Scuratti, Stefano Agresti, Donya Rooein, Gabriele Scalia, Amudha Ravi Shankar, Jose Luis Fernandez Marquez, Mark James Carman, and Barbara Pernici. 2021. Image-based social sensing: combining AI and the crowd to mine policy-adherence indicators from Twitter. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 92--101.
[30]
An Nguyen, Byron Wallace, and Matthew Lease. 2015. Combining crowd and expert labels using decision theoretic active learning. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3. 120--129.
[31]
Ahmed Husseini Orabi, Prasadith Buddhitha, Mahmoud Husseini Orabi, and Diana Inkpen. 2018. Deep learning for depression detection of twitter users. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. 88--97.
[32]
Matteo Polsinelli, Luigi Cinque, and Giuseppe Placidi. 2020. A light CNN for detecting COVID-19 from CT scans of the chest. Pattern recognition letters 140 (2020), 95--100.
[33]
Marisa Ponti and Alena Seredko. 2022. Human-machine-learning integration and task allocation in citizen science. Humanities and Social Sciences Communications 9, 1 (2022), 1--15.
[34]
PyTorch. [n. d.]. https://pytorch.org/
[35]
Agha Ali Raza, Mustafa Naseem, Namoos Hayat Qasmi, Shan Randhawa, Fizzah Malik, Behzad Taimur, Sacha St-Onge Ahmad, Sarojini Hirshleifer, Arman Rezaee, and Aditya Vashistha. 2022. Fostering Engagement of Underserved Communities with Credible Health Information on Social Media. In Proceedings of the ACM Web Conference 2022. 3718--3727.
[36]
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. 2017. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694 (2017).
[37]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296 (2017).
[38]
Ozan Sener and Silvio Savarese. 2018. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In International Conference on Learning Representations.
[39]
Megh Shukla and Shuaib Ahmed. 2021. A mathematical analysis of learning loss for active learning in regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3320--3328.
[40]
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2820--2828.
[41]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, Vol. 139. 10347--10357.
[42]
Hoang Duy Trinh, Lorenza Giupponi, and Paolo Dini. 2019. Urban anomaly detection by processing mobile traffic traces with LSTM neural networks. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--8.
[43]
Shu-Feng Tsao, Helen Chen, Therese Tisseverasinghe, Yang Yang, Lianghua Li, and Zahid A Butt. 2021. What social media told us in the time of COVID-19: a scoping review. The Lancet Digital Health 3, 3 (2021), e175--e194.
[44]
Amazon Mechanical Turk. [n. d.]. https://www.mturk.com/
[45]
Twitter. [n. d.]. https://www.twitter.com/
[46]
Yan Yan, Glenn M Fung, Rómer Rosales, and Jennifer G Dy. 2011. Active learning from crowds. In Proceedings of the 28th international conference on machine learning (ICML-11). 1161--1168.
[47]
Chenhao Ying, Haiming Jin, XudongWang, and Yuan Luo. 2020. Chaste: Incentive mechanism in edge-assisted mobile crowdsensing. In 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--9.
[48]
Donggeun Yoo and In So Kweon. 2019. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 93--102.
[49]
Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, and Dong Wang. 2022. Contrastive domain adaptation for early misinformation detection: A case study on covid-19. In Proceedings of the 31st ACMInternational Conference on Information & Knowledge Management. 2423--2433.
[50]
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. 2018. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906 (2018).
[51]
Anran Zhang, Jiayi Shen, Zehao Xiao, Fan Zhu, Xiantong Zhen, Xianbin Cao, and Ling Shao. 2019. Relational attention network for crowd counting. In Proceedings of the IEEE/CVF international conference on computer vision. 6788--6797.
[52]
Daniel Zhang, Yang Zhang, Qi Li, Thomas Plummer, and Dong Wang. 2019. Crowdlearn: A crowd-ai hybrid system for deep learning-based damage assessment applications. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 1221--1232.
[53]
Weitong Zhang, Dongruo Zhou, and Quanquan Gu. 2021. Reward-free model based reinforcement learning with linear function approximation. Advances in Neural Information Processing Systems 34 (2021), 1582--1593.
[54]
Yang Zhang, Xiangyu Dong, Lanyu Shang, Daniel Zhang, and Dong Wang. 2020. A multi-modal graph neural network approach to traffic risk forecasting in smart urban sensing. In 2020 17th Annual IEEE international conference on sensing, communication, and networking (SECON). IEEE, 1--9.
[55]
Yang Zhang, Ruohan Zong, Ziyi Kou, Lanyu Shang, and DongWang. 2022. Crowd-NAS: A Crowd-guided Neural Architecture Searching Approach to Disaster Damage Assessment. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1--29.
[56]
Yang Zhang, Ruohan Zong, Ziyi Kou, Lanyu Shang, and DongWang. 2022. Crowd-NAS: A Crowd-guided Neural Architecture Searching Approach to Disaster Damage Assessment. Proceedings of the ACM on Human-Computer Interaction CSCW (2022).

Index Terms

  1. SymLearn: A Symbiotic Crowd-AI Collective Learning Framework to Web-based Healthcare Policy Adherence Assessment

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WWW '24: Proceedings of the ACM Web Conference 2024
    May 2024
    4826 pages
    ISBN:9798400701719
    DOI:10.1145/3589334
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 May 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. crowdsourcing
    2. human-ai collaboration
    3. public health
    4. social media

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    WWW '24
    Sponsor:
    WWW '24: The ACM Web Conference 2024
    May 13 - 17, 2024
    Singapore, Singapore

    Acceptance Rates

    Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 243
      Total Downloads
    • Downloads (Last 12 months)243
    • Downloads (Last 6 weeks)43
    Reflects downloads up to 27 Jan 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media