Skip to main content

Advertisement

SpringerLink
Log in

Automatic content moderation on social media

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Millions of users produce and consume billions of content on social media. Therefore, human-reviewed content moderation is not achievable in such volume. Automating content moderation is a scalable solution for social media platforms. In this research work, we propose an automatic content moderation pipeline based on deep neural networks. Our solution consists of two main parts: the first part classifies a given image into granular content classes; and a second part obfuscates the part of a given image that might be inappropriate for the target audience. Our proposed solution is a cost-efficient in terms of human labour and practical for deploying the real-time systems. Our classification network is trained with automatically labelled data using noise-robust techniques. Our automatic obfuscation algorithm uses the information obtained from the classification network and does not require additional annotation or supplementary training. This obfuscation algorithm presents a novel-use case of class-specific activation mappings for censoring regional explicit nudity in images. The classification network achieves a top-1 accuracy of 0.903 and a top-2 accuracy of 0.986. The obfuscation algorithm covers a minimum explicitly nude area of 0.68 on average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Abadpour A, Kasaei S (2005) Pixel-based skin detection for pornography filtering. Iranian J Electr Electron Eng 1(3):21–41

    Google Scholar 

  2. Agrawal A, Batra D, Parikh D (2016) Analyzing the behavior of visual question answering models, arXiv:1606.07356

  3. Arentz WA, Olstad B (2004) Classifying offensive sites based on image content. Comput Vis Image Underst 94(1-3):295–310

    Article  Google Scholar 

  4. Arsht A, Etcovitch D (2018) The human cost of online content moderation. Harvard J Law Technol:1

  5. Batrinca B, Treleaven PC (2015) Social media analytics: a survey of techniques, tools and platforms. Ai Soc 30(1):89–116

    Article  Google Scholar 

  6. Caetano C, Avila S, Schwartz WR, Guimaraes SJF, Araújo ADA (2016) A mid-level video representation based on binary descriptors: a case study for pornography detection. Neurocomputing 213:102–114

    Article  Google Scholar 

  7. da Silva MV, Marana AN (2018) Spatiotemporal cnns for pornography detection in videos. In: Iberoamerican congress on pattern recognition. Springer, pp 547–555

  8. Duarte N, Llanso E, Loup AC (2018) Mixed messages? the limits of automated social media content analysis. In: FAT, pp 106

  9. Fan X, Beaupre T, Webb B (2019) Social media based content selection system. uS Patent App 14(/343):931

    Google Scholar 

  10. Fleck MM, Forsyth DA, Bregler C (1996) Finding naked people. In: European conference on computer vision. Springer, pp 593–602

  11. Fleck MM, Forsyth DA, Bregler C (1996) Finding naked people. In: European conference on computer vision. Springer, pp 593–602

  12. Gajula G, Hundiwale A, Mujumdar S, Saritha L (2020) A machine learning based adult content detection using support vector machine. In: 2020 7Th international conference on computing for sustainable global development (INDIACom). IEEE, pp 181–185

  13. Gangwar A, Fidalgo E, Alegre E, González-castro V (2017) Pornography and child sexual abuse detection in image and video: A comparative evaluation

  14. Gorbova J, Colovic M, Marjanovic M, Njegus A, Anbarjafari G (2019) Going deeper in hidden sadness recognition using spontaneous micro expressions database. Multimed Tools Appl 78(16):23 161–23 178

    Article  Google Scholar 

  15. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior JCSJ, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26 391–26 403

    Article  Google Scholar 

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  17. Hoiem D, Chodpathumwan Y, Dai Q (2012) Diagnosing error in object detectors. In: European conference on computer vision. Springer, pp 340–353

  18. Hu W, Wu O, Chen Z, Fu Z, Maybank S (2007) Recognition of pornographic web pages by classifying texts and images. IEEE Trans Pattern Anal Mach Intell 29(6):1019–1034

    Article  Google Scholar 

  19. Jahromi MNS, Buch-Cardona P, Avots E, Nasrollahi K, Escalera S, Moeslund TB, Anbarjafari G (2019) Privacy-constrained biometric system for non-cooperative users. Entropy 21(11):1033

    Article  Google Scholar 

  20. Jeong H, Park J, Park J, Lee B (2017) Inconsistent work performance in automation, can we measure trust in automation. Int Robot Autom J 6:3

    Google Scholar 

  21. Jhaver S, Birman I, Gilbert E, Bruckman A (2019) Human-machine collaboration for content regulation: the case of reddit automoderator. ACM Trans Comput-Human Interact (TOCHI) 26(5):1–35

    Article  Google Scholar 

  22. Jones MJ, Rehg JM (2002) Statistical color models with application to skin detection. Int J Comput Vis 46(1):81–96

    Article  MATH  Google Scholar 

  23. Kakumanu P, Makrogiannis S, Bourbakis N (2007) A survey of skin-color modeling and detection methods. Pattern Recogn 40(3):1106–1122

    Article  MATH  Google Scholar 

  24. Karabulut D, Tertychnyi P, Arslan HS, Ozcinar C, Nasrollahi K, Valls J, Vilaseca J, Moeslund TB, Anbarjafari G (2020) Cycle-consistent generative adversarial neural networks based low quality fingerprint enhancement. Multimed Tools Appli:1–21

  25. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization, arXiv:1412.6980

  26. Kotenko I, Chechulin A, Komashinsky D (2017) Categorisation of web pages for protection against inappropriate content in the internet. Int J Internet Protocol Technol 10(1):61–71

    Article  Google Scholar 

  27. Lipton ZC (2018) The mythos of model interpretability. Queue 16 (3):31–57

    Article  Google Scholar 

  28. Litvin A, Nasrollahi K, Escalera S, Ozcinar C, Moeslund TB, Anbarjafari G (2019) A novel deep network architecture for reconstructing rgb facial images from thermal for face recognition. Multimed Tools Appl 78(18):25 259–25 271

    Article  Google Scholar 

  29. Lopes AP, de Avila SE, Peixoto AN, Oliveira RS, Araújo AdA (2009) A bag-of-features approach based on hue-sift descriptor for nude detection. In: 2009 17Th european signal processing conference. IEEE, pp 1552–1556

  30. Mahadeokar J, Pesavento G (2016) Open sourcing a deep learning solution for detecting nsfw images. Retr August 24:2018

    Google Scholar 

  31. Malamuth NM (1996) Sexually explicit media, gender differences and evolutionary theory. J Commun 46(3):1460–2466

    Google Scholar 

  32. Moustafa M (2015) Applying deep learning to classify pornographic images and videos, arXiv:1511.088991511.08899

  33. Narayanan BK, Nirmala M et al (2018) Adult content filtering: restricting minor audience from accessing inappropriate internet content. Educ Inf Technol 23(6):2719–2735

    Article  Google Scholar 

  34. Noroozi F, Kaminska D, Corneanu C, Sapinski T, Escalera S, Anbarjafari G (2018) Survey on emotional body gesture recognition. IEEE Transactions on Affective Computing

  35. Noroozi F, Marjanovic M, Njegus A, Escalera S, Anbarjafari G (2017) Audio-visual emotion recognition in video clips. IEEE Trans Affect Comput 10(1):60–75

    Article  Google Scholar 

  36. Ofodile I, Helmi A, Clapés A, Avots E, Peensoo KM, Valdma S-M, Valdmann A, Valtna-Lukner H, Omelkov S, Escalera S et al (2019) Action recognition using single-pixel time-of-flight detection. Entropy 21(4):414

    Article  Google Scholar 

  37. Park N, Kim Y (2010) Harmful adult multimedia contents filtering method in mobile rfid service environment. In: International conference on computational collective intelligence. Springer, pp 193–202

  38. Perez M, Avila S, Moreira D, Moraes D, Testoni V, Valle E, Goldenstein S, Rocha A (2017) Video pornography detection through deep learning techniques and motion information. Neurocomputing 230:279–293

    Article  Google Scholar 

  39. Rowley HA, Jing Y, Baluja S (2006) Large scale image-based adult-content filtering. In: International conference on computer vision theory and applications. IEEE, pp 1–7

  40. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  41. Sapiński T, Kamińska D, Pelikant A, Anbarjafari G (2019) Emotion recognition from skeletal movements. Entropy 21(7):646

    Article  Google Scholar 

  42. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  43. Singh M, Bansal D, Sofat S (2016) Behavioral analysis and classification of spammers distributing pornographic content in social media. Soc Netw Anal Min 6(1):41

    Article  Google Scholar 

  44. Sugimoto CR, Work S, Larivière V, Haustein S (2017) Scholarly use of social media and altmetrics: a review of the literature. J Assoc Inf Sci Technol 68(9):2037–2062

    Article  Google Scholar 

  45. Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, pp 3319–3328

  46. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  47. Tan WR, Chan CS, Aguirre HE, Tanaka K (2018) Improved artgan for conditional synthesis of natural image and artwork. IEEE Trans Image Process 28(1):394–409

    Article  MathSciNet  MATH  Google Scholar 

  48. Wang Y, Ma X, Chen Z, Luo Y, Yi J, Bailey J (2019) Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the IEEE International Conference on Computer Vision, pp 322–330

  49. Wehrmann J, Simoes GS, Barros RC, Cavalcante VF (2018) Adult content detection in videos with convolutional and recurrent neural networks. Neurocomputing 272:432–438

    Article  Google Scholar 

  50. Yuan K, Tang D, Liao X, Wang X, Feng X, Chen Y, Sun M, Lu H, Zhang K (2019) Stealthy porn: Understanding real-world adversarial images for illicit online promotion. In: 2019 IEEE Symposium on security and privacy (SP). IEEE, pp 952–966

  51. Zheng Q-F, Zeng W, Wang W-Q, Gao W (2006) Shape-based adult image detection. Int J Image Graph 6(01):115–124

    Article  Google Scholar 

  52. Zhu Y, Chen Z, Wu F (2020) Affective video content analysis via multimodal deep quality embedding network. IEEE Transactions on Affective Computing

  53. Zhu H, Zhou S, Wang J, Yin Z (2007) An algorithm of pornographic image detection. In: Fourth international conference on image and graphics (ICIG 2007). IEEE, pp 801–804

Download references

Author information

Authors and Affiliations

  1. iCV Research Lab, Institute of Technology, University of Tartu, Tartu, 50411, Estonia

    Dogus Karabulut, Cagri Ozcinar & Gholamreza Anbarjafari

  2. Yildiz Technical University, Istanbul, Turkey

    Gholamreza Anbarjafari

  3. iVCV OÜ, Tartu, Estonia

    Gholamreza Anbarjafari

  4. PwC Advisory, Helsinki, Finland

    Gholamreza Anbarjafari

Authors
  1. Dogus Karabulut
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cagri Ozcinar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gholamreza Anbarjafari.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work has been partially supported by the Estonian Centre of Excellence in IT (EXCITE) funded by the European Regional Development Fund. The authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V and X Pascal GPU.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karabulut, D., Ozcinar, C. & Anbarjafari, G. Automatic content moderation on social media. Multimed Tools Appl 82, 4439–4463 (2023). https://doi.org/10.1007/s11042-022-11968-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-11968-3

Keywords

Search

Navigation