skip to main content
10.1145/3476098.3485055acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Improving Multimodal Data Labeling with Deep Active Learning for Post Classification in Social Networks

Published: 20 October 2021 Publication History

Abstract

Automatic user post classification is an important task in the field of social network analysis. Being effectively solved, post classification could be used for thematic user feed composition or inappropriate content identification. Commonly addressed by applying various Machine Learning approaches, the task often involves manual processes related to ground truth sourcing, which is known to be a hardly-scalable and increasingly expensive procedure. At the same time, Active Learning for automatic user post classification is a promising way to bridge such a gap, as it does not require massive ground truth availability aligning our research with the real world settings. In this work, we put our focus on leveraging textual and visual data modalities for the application of user post classification and investigate how batch size and batch normalization disabling techniques could affect active deep neural network learning process. We solve the problem of automatic user post classification by employing our novel multimodal neural network architecture with multi-head tunable loss function components. We show that the proposed approach, coupled with Active Learning, allows for the achievement of a significant classification performance boost in terms of crowd assessing resources as compared to the passive learning approaches.

References

[1]
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. In International conference on machine learning. PMLR, 1247-- 1255.
[2]
Nicolas Audebert, Catherine Herold, Kuider Slimani, and Cédric Vidal. 2019. Multimodal deep networks for text and image-based document classification. arXiv:arXiv:1907.06370
[3]
Tadas Baltruaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41, 2 (2018), 423--443.
[4]
Luciana Bencke, Cristian Cechinel, and Roberto Munoz. 2020. Automated classification of social network messages into Smart Cities dimensions. Future Generation Computer Systems 109 (2020), 218--237.
[5]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5 (2017), 135--146.
[6]
William J Brady, MJ Crockett, and Jay J Van Bavel. 2020. The MAD model of moral contagion: The role of motivation, attention, and design in the spread of moralized content online. Perspectives on Psychological Science 15, 4 (2020), 978--1010.
[7]
Kseniya Buraya, Aleksandr Farseev, and Andrey Filchenkov. 2018. Multi-view personality profiling based on longitudinal data. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 15--27.
[8]
Serhii Chalyi and Inna Pribylnova. 2019. The method of constructing recommendations online on the temporal dynamics of user interests using multilayer graph. EUREKA: Physics and Engineering 3 (2019), 13--19.
[9]
Mauro Conti, Daniele Lain, Riccardo Lazzeretti, Giulio Lovisotto, and Walter Quattrociocchi. 2017. It's always April fools' day!: On the difficulty of social network misinformation classification via propagation features. In 2017 IEEE Workshop on Information Forensics and Security (WIFS). IEEE, 1--6.
[10]
Hang Cui, Tarek Abdelzaher, and Lance Kaplan. 2019. A semi-supervised activelearning truth estimator for social networks. In The World Wide Web Conference. 296--306.
[11]
Jia Deng,Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.
[12]
Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. 2015. Multimodal Deep Learning for Robust RGB-D Object Recognition. arXiv:arXiv:1507.06821
[13]
Aleksandr Farseev and Tat-Seng Chua. 2017. Tweet can be fit: Integrating data from wearable sensors and multiple social networks for wellness profile learning. ACM Transactions on Information Systems (TOIS) 35, 4 (2017), 1--34.
[14]
Aleksandr Farseev, Liqiang Nie, Mohammad Akbari, and Tat-Seng Chua. 2015. Harvesting multiple sources for user profile learning: a big data study. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. 235--242.
[15]
Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. 2019. Deep Session Interest Network for Click-Through Rate Prediction. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10--16, 2019, Sarit Kraus (Ed.). ijcai.org, 2301--2307. https://doi.org/10.24963/ijcai.2019/319
[16]
Mafalda Falcao Ferreira, Rui Camacho, and Luis F. Teixeira. 2020. Autoencoders asWeight Initialization of Deep Classification Networks for Cancer versus Cancer Studies. arXiv:2001.05253 [cs.LG]
[17]
Alvaro Figueira, Miguel Sandim, and Paula Fortuna. 2016. An approach to relevancy detection: contributions to the automatic detection of relevance in social networks. In New Advances in Information Systems and Technologies. Springer, 89--99.
[18]
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (New York, NY, USA) (ICML'16). JMLR.org, 1050--1059.
[19]
RA Gilyazev and D Yu Turdakov. 2018. Active Learning and Crowdsourcing: A Survey of Optimization Methods for Data Labeling. Programming and Computer Software 44, 6 (2018), 476--491.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[21]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.
[22]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735--1780.
[23]
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 (2011).
[24]
Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. CoRR abs/1502.03167 (2015). arXiv:1502.03167 http://arxiv.org/abs/1502.03167
[25]
Md Rafiqul Islam, Shaowu Liu, Xianzhi Wang, and Guandong Xu. 2020. Deep learning for misinformation detection on online social networks: a survey and new perspectives. Social Network Analysis and Mining 10, 1 (2020), 1--20.
[26]
A Kendall, Y Gal, and R Cipolla. 2017. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. arXiv:1705.07115 (2017).
[27]
Xiaolin Lin, Saonee Sarker, and Mauricio Featherman. 2019. Users' psychological perceptions of information sharing in the context of social media: a comprehensive model. International Journal of Electronic Commerce 23, 4 (2019), 453--491.
[28]
Hui-Shu Liu, Gi-Young Jung, and Hyung-Ho Kim. 2020. A Study on Information Consumption Behavior Structure of Rural Residents in Changchun, China. Journal of the Korea Convergence Society 11, 1 (2020), 9--16.
[29]
Liyuan Liu, Haoming Jiang, Pengcheng He,Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the Variance of the Adaptive Learning Rate and Beyond. arXiv:1908.03265 [cs.LG]
[30]
Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M Bronstein. 2019. Fake news detection on social media using geometric deep learning. arXiv preprint arXiv:1902.06673 (2019).
[31]
Maximiliano Perez-Cepeda and Leopoldo Arias-Bolzmann. [n.d.]. Influence of Ecuadorian Homosexual Subculture in Consumption Culture: Study about Information Consumption on Twitter. Journal of Promotion Management ([n. d.]), 1--22.
[32]
Daniela Pohl, Abdelhamid Bouchachia, and Hermann Hellwagner. 2018. Batchbased active learning: Application to social media data for crisis management. Expert Systems with Applications 93 (2018), 232--244.
[33]
Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668 (2018).
[34]
Andrey Savchenko, Anton Alekseev, Sejeong Kwon, Elena Tutubalina, Evgeny Myasnikov, and Sergey Nikolenko. 2020. Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements. In Proceedings of the 28th International Conference on Computational Linguistics. 1886--1892.
[35]
Burr Settles. 2009. Active learning literature survey. Technical Report. University of Wisconsin-Madison Department of Computer Sciences.
[36]
H. S. Seung, M. Opper, and H. Sompolinsky. 1992. Query by committee. In Proceedings of the Fifth Annual ACM Workshop on Computational Learning Theory (Proceedings of the Fifth Annual ACM Workshop on Computational Learning Theory). Publ by ACM, 287--294.
[37]
Abraham Silberschatz, Peter B Galvin, and Greg Gagne. 2009. Operating system concepts with Java. Wiley Publishing.
[38]
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387 (2015).
[39]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1--9.
[40]
Eugenio Tacchini, Gabriele Ballarin, Marco L Della Vedova, Stefano Moret, and Luca de Alfaro. 2017. Some like it hoax: Automated fake news detection in social networks. arXiv preprint arXiv:1704.07506 (2017).
[41]
Takumi Takahashi, Takuji Tahara, Koki Nagatani, Yasuhide Miura, Tomoki Taniguchi, and Tomoko Ohkuma. 2018. Text and Image Synergy with Feature Cross Technique for Gender Identification: Notebook for PAN at CLEF 2018. In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10--14, 2018 (CEURWorkshop Proceedings, Vol. 2125), Linda Cappellato, Nicola Ferro, Jian-Yun Nie, and Laure Soulier (Eds.). CEUR-WS.org. http://ceur-ws.org/Vol-2125/paper_83.pdf
[42]
Qiaozhi Wang, Jaisneet Bhandal, Shu Huang, and Bo Luo. 2017. Classification of private tweets using tweet content. In 2017 IEEE 11th International Conference on Semantic Computing (ICSC). IEEE, 65--68.
[43]
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035--2043.
[44]
Donggeun Yoo and In So Kweon. 2019. Learning loss for active learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition.
[45]
Zhou Yu, Yuhao Cui, Jun Yu, Dacheng Tao, and Qi Tian. 2019. Multimodal Unified Attention Networks for Vision-and-Language Interactions. arXiv preprint arXiv:1908.04107 (2019).
[46]
Chao Zhang, Zichao Yang, Xiaodong He, and Li Deng. 2020. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing 14, 3 (2020), 478--493.
[47]
Qiao Zhang, Shuiyuan Zhang, Jian Dong, Jinhua Xiong, and Xueqi Cheng. 2015. Automatic detection of rumor on social network. In Natural Language Processing and Chinese Computing. Springer, 113--122.
[48]
Tingshao Zhu, Russell Greiner, and Gerald Häubl. 2003. Learning a Model of a Web User's Interests. In User Modeling 2003, 9th International Conference, UM 2003, Johnstown, PA, USA, June 22--26, 2003, Proceedings (Lecture Notes in Computer Science, Vol. 2702), Peter Brusilovsky, Albert T. Corbett, and Fiorella de Rosis (Eds.). Springer, 65--75. https://doi.org/10.1007/3--540--44963--9_10

Cited By

View all
  • (2024)Label Engineering Methods for ML SystemsIntelligent Systems and Applications10.1007/978-3-031-66336-9_33(464-474)Online publication date: 1-Aug-2024
  • (2022)Modeling the trajectories of interests and preferences of users in digital social systemsProcedia Computer Science10.1016/j.procs.2022.10.212212:C(104-113)Online publication date: 1-Jan-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MULL'21: Multimedia Understanding with Less Labeling on Multimedia Understanding with Less Labeling
October 2021
64 pages
ISBN:9781450386814
DOI:10.1145/3476098
  • Program Chairs:
  • Xiu-Shen Wei,
  • Han-Jia Ye,
  • Jufeng Yang,
  • Jian Yang
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. active learning
  2. data crowdsourcing
  3. deep learning
  4. multimodal data classification
  5. social networks

Qualifiers

  • Research-article

Conference

MM '21
Sponsor:
MM '21: ACM Multimedia Conference
October 24, 2021
Virtual Event, China

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)25
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Label Engineering Methods for ML SystemsIntelligent Systems and Applications10.1007/978-3-031-66336-9_33(464-474)Online publication date: 1-Aug-2024
  • (2022)Modeling the trajectories of interests and preferences of users in digital social systemsProcedia Computer Science10.1016/j.procs.2022.10.212212:C(104-113)Online publication date: 1-Jan-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media