Abstract
The article describes a software pipeline for identifying and classifying the interests of users in social networks using modern models and deep learning methods. The developed program is able to detect the presence of bad habits (smoking, alcohol), a sporting lifestyle, as well as determine the user's addiction to travel by an available set of photos. The software includes modules that implement deep learning algorithms for the object detection and semantic segmentation of images using the Cascade-R-CNN and DeepLabv3+ models, and the module for converting annotations of the images from COCO, ImageNet, OpenImagesV6 datasets and manually labeled images to the unified format. The models were trained on the created original datasets which include 90200 photos in total. The accuracy of the developed models is from 83.7% up to 86.6% mAP for object detection depending on a specific category of objects and 78.4% pixel accuracy for segmentation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Burrell, J.: How the machine ‘thinks’: understanding opacity in machine learning algorithms. Big Data Soc. (2016). https://doi.org/10.1177/2053951715622512
Schwartz, S.H.: An overview of the schwartz theory of basic values. Online Readings Psychol. Cult. (2012). https://doi.org/10.9707/2307-0919.1116
Bachrach, Y., Kosinski, M., Graepel, T., Kohli, P., Stillwell, D.: Personality and patterns of Facebook usage. In: Proceedings of the 4th Annual ACM Web Science Conference, WebSci 2012 (2012). https://doi.org/10.1145/2380718.2380722.
Hua, W., Huynh, D.T., Hosseini, S., Lu, J., Zhou, X.: Information extraction from microblogs: a survey Information extraction from mi- croblogs: a survey. Int J Softw. Inf. 66(44), 495–522 (2012)
Gemp, I., Nallapati, R., Ding, R., Nan, F., Xiang, B.: Weakly semi-supervised neural topic models. In: ICLR (2019)
Fang, G., Su, L., Jiang, D., Wu, L.: Group recommendation systems based on external social-trust networks. Wirel. Commun. Mob. Comput. (2018). https://doi.org/10.1155/2018/6709607
Hossain, M.D., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. 51(6), 118 (2019)
Zhang, H., et al.: ResNeSt: Split-Attention Networks (2020)
Zhai, A., et al.: Visual discovery at Pinterest. In: 26th International World Wide Web Conference 2017, WWW 2017 Companion (2019). https://doi.org/10.1145/3041021.3054201
Grechikhin, I., Savchenko, A.V.: User modeling on mobile device based on facial clustering and object detection in photos and videos. In: Morales, A., Fierrez, J., Sánchez, J.S., Ribeiro, B. (eds.) IbPRIA 2019. LNCS, vol. 11868, pp. 429–440. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31321-0_37
Demochkin, K.V., Savchenko, A.V.: User preference prediction in a set of photos based on neural aggregation network. In: Bychkov, I., Kalyagin, V.A., Pardalos, P.M., Prokopyev, O. (eds.) Network Algorithms, Data Mining, and Applications: NET, Moscow, Russia, May 2018, pp. 121–127. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-37157-9_8
Wieczorek, S., Filipiak, D., Filipowska, A.: Semantic image-based profiling of users’ interests with neural networks. In: 4th Working Semantics Deep Learning International Semantics Web Conference 2018 (2018). https://doi.org/10.3233/978-1-61499-894-5-179
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2018.2858826
The latest in machine learning | Papers With Code. https://paperswithcode.com/, Accessed 14 Jul 2020
Cai, Z., Vasconcelos, N.: Cascade R-CNN: high quality object detection and instance segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (2019). https://doi.org/10.1109/tpami.2019.2956516
Cai, Z., Vasconcelos, N.: Cascade R-CNN: delving into high quality object detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/CVPR.2018.00644
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49
Chollet,F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (2017). https://doi.org/10.1109/CVPR.2017.195
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Rethinking atrous convolution for semantic image segmentation liang-chieh. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://doi.org/10.1109/TPAMI.2017.2699184
Acknowledgments
This research is financially supported by The Russian Science Foundation, Agreement №17-71-30029 with co-financing of Bank Saint Petersburg.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Volokha, V., Gladilin, P. (2021). Identifying User Interests and Habits Using Object Detection and Semantic Segmentation Models. In: van der Aalst, W.M.P., et al. Analysis of Images, Social Networks and Texts. AIST 2020. Lecture Notes in Computer Science(), vol 12602. Springer, Cham. https://doi.org/10.1007/978-3-030-72610-2_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-72610-2_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72609-6
Online ISBN: 978-3-030-72610-2
eBook Packages: Computer ScienceComputer Science (R0)