Skip to main content
SpringerLink
Log in

Toward efficient and intelligent video analytics with visual privacy protection for large-scale surveillance

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Nowadays, the explosion of CCTV cameras has resulted in an increasing demand for distributed solutions to efficiently process the vast volume of video data. Otherwise, the use of surveillance when people are being watched remotely and recorded continuously has raised a significant threat to visual privacy. Using existing systems cannot prevent any party from exploiting unwanted personal data of others. In this paper, we develop an intelligent surveillance system with integrated privacy protection, where it is built on the top of big data tools, i.e., Kafka and Spark Streaming. To protect individual privacy, we propose a privacy-preserving solution based on effective face recognition and tracking mechanisms. Particularly, we associate body pose with face to reduce privacy leaks across video frames. The body pose is also exploited to infer person-centric information like human activities. Extensive experiments conducted on benchmark datasets further demonstrate the efficiency of our system for various vision tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Apache Kafka (2020) https://kafka.apache.org/

  2. Ajunwa I, Crawford K, Schultz J (2017) Limitless worker surveillance. Calif Law Rev 105:735

    Google Scholar 

  3. Alabdulatif A, Khalil I, Yi X (2020) Towards secure big data analytic for cloud-enabled applications with fully homomorphic encryption. J Parallel Distrib Comput 137:192–204

    Article  Google Scholar 

  4. Avola D, Cascio M, Cinque L, Foresti GL, Massaroni C, Rodolà E (2019) 2-D skeleton-based action recognition via two-branch stacked LSTM-RNNS. IEEE Trans Multimed 22(10):2481–2496

    Article  Google Scholar 

  5. Balapour A, Nikkhah HR, Sabherwal R (2020) Mobile application security: role of perceived privacy as the predictor of security perceptions. Int J Inf Manag 52:102063

    Article  Google Scholar 

  6. Bitouk D, Kumar N, Dhillon S, Belhumeur P, Nayar SK (2008) Face swapping: automatically replacing faces in photographs. In: ACM SIGGRAPH 2008 papers, pp 1–8

  7. Brkic K, Sikiric I, Hrkac T, Kalafatic Z (2017) I know that person: Generative full body and face de-identification of people in images. In: CVPRW, IEEE, pp 1319–1328

  8. Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: CVPR, pp 7291–7299

  9. Chamikara MAP, Bertok P, Khalil I, Liu D, Camtepe S (2020) Privacy preserving face recognition utilizing differential privacy. Comput Secur 97:101951

    Article  Google Scholar 

  10. Chen J, Li K, Deng Q, Li K, Philip SY (2019) Distributed deep learning model for intelligent video surveillance systems with edge computing. IEEE Trans Ind Inf

  11. Chen JC, Patel VM, Chellappa R (2016) Unconstrained face verification using deep cnn features. In: IEEE WACV, pp 1–9

  12. Chhabra S, Singh R, Vatsa M, Gupta G (2018) Anonymizing k-facial attributes via adversarial perturbations. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp 656–662

  13. Cloud data access (2018) https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/intro.html/

  14. DAR P (2018) A chinese school is using facial recognition to analyze students’ behavior. https://www.analyticsvidhya.com/blog/2018/06/china-school-facial-recognition-analyse-students/

  15. Dai J, Saghafi B, Wu J, Konrad J, Ishwar P (2015) Towards privacy-preserving recognition of human activities. In: ICIP, IEEE, pp 4238–4242

  16. Domingo-Ferrer J (2010) Coprivacy: towards a theory of sustainable privacy. In: International Conference on Privacy in Statistical Databases, Springer, pp 258–268

  17. Gafni O, Wolf L, Taigman Y (2019) Live face de-identification in video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 9378–9387

  18. Garfinkel SL (2015) De-identification of personal information. National institute of standards and technology

  19. Gentry C et al (2009) Fully homomorphic encryption using ideal lattices. In: Stoc, vol 9, pp 169–178

  20. Gong S (2012) Learning user interest model for content-based filtering in personalized recommendation system. Int J Digital Content Technol Appl 6(11):155–162

    Article  Google Scholar 

  21. De Guzman JA, Thilakarathna K, Seneviratne A (2019) Security and privacy approaches in mixed reality: a literature survey. ACM Comput Surv 52(6):1–37

    Article  Google Scholar 

  22. Henriksen-Bulmer J, Jeary S (2016) Re-identification attacks–a systematic literature review. Int J Inf Manag 36(6):1184–1192

    Article  Google Scholar 

  23. Huynh-The T, Hua CH, Tu NA, Hur T, Bang J, Kim D, Amin MB, Kang BH, Seung H, Shin SY et al (2018) Hierarchical topic modeling with pose-transition feature for action recognition using 3d skeleton data. Inf Sci 444:20–35

    Article  MathSciNet  Google Scholar 

  24. Kim K, Yang Z, Masi I, Nevatia R, Medioni G (2018) Face and body association for video-based face recognition. In: IEEE WACV, pp 39–48

  25. Landset S, Khoshgoftaar TM, Richter AN, Hasanin T (2015) A survey of open source tools for machine learning with big data in the hadoop ecosystem. J Big Data 2(1):24

    Article  Google Scholar 

  26. Li H, Gu Z, Deng L, Han Y, Yang C, Tian Z (2019) A fine-grained video encryption service based on the cloud-fog-local architecture for public and private videos. Sensors 19(24):5366

    Article  Google Scholar 

  27. Lin J, Li Y, Yang G (2021) Fpgan: Face de-identification method with generative adversarial networks for social robots. Neural Netw 133:132–147

    Article  Google Scholar 

  28. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: Single shot multibox detector. In: ECCV, Springer, pp 21–37

  29. Ludl D, Gulde T, Curio C (2019) Simple yet efficient real-time pose-based action recognition. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), IEEE, pp 581–588

  30. Lv J, Wu B, Liu C, Gu X (2018) Pf-face: A parallel framework for face classification and search from massive videos based on spark. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), IEEE, pp 1–7

  31. Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S et al (2016) Mllib: Machine learning in apache spark. J Mach Learn Res 17(1):1235–1241

    MathSciNet  MATH  Google Scholar 

  32. Nazare AC Jr, Schwartz WR (2016) A scalable and flexible framework for smart video surveillance. CVIU 144:258–275

    Google Scholar 

  33. Newton EM, Sweeney L, Malin B (2005) Preserving privacy by de-identifying face images. IEEE Trans Knowl Data Eng 17(2):232–243

    Article  Google Scholar 

  34. Orekondy T, Schiele B, Fritz M (2017) Towards a visual privacy advisor: Understanding and predicting privacy risks in images. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3686–3695

  35. Orekondy T, Fritz M, Schiele B (2018) Connecting pixels to privacy and utility: automatic redaction of private information in images. In: CVPR, pp 8466–8475

  36. O’Toole AJ, Phillips PJ, Weimer S, Roark DA, Ayyad J, Barwick R, Dunlop J (2011) Recognizing people from dynamic and static faces and bodies: dissecting identity with a fusion approach. Vis Res 51(1):74–83

    Article  Google Scholar 

  37. Pei D, Guo X, Zhang J (2017) A video encryption service based on cloud computing. In: 2017 7th IEEE International Conference on Electronics Information and Emergency Communication (ICEIEC), IEEE, pp 167–171

  38. Plageras AP, Psannis KE, Ishibashi Y, Kim BG (2016) Iot-based surveillance system for ubiquitous healthcare. In: IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society, IEEE, pp 6226–6230

  39. Ren Z, Jae Lee Y, Ryoo MS (2018) Learning to anonymize faces for privacy preserving action detection. In: ECCV, pp 620–636

  40. Ryoo MS, Rothrock B, Fleming C, Yang HJ (2017) Privacy-preserving human activity recognition from extreme low resolution. In: Thirty-First AAAI Conference on Artificial Intelligence

  41. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: CVPR, pp 815–823

  42. Shahroudy A, Liu J, Ng TT, Wang G (2016) Ntu rgb+ d: a large scale dataset for 3d human activity analysis. In: CVPR, pp 1010–1019

  43. Shao Z, Cai J, Wang Z (2017) Smart monitoring cameras driven intelligent processing to big surveillance video data. IEEE Trans Big Data 4(1):105–116

    Article  Google Scholar 

  44. Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: CVPR 2011, IEEE, pp 1297–1304

  45. Simoens P, Xiao Y, Pillai P, Chen Z, Ha K, Satyanarayanan M (2013) Scalable crowd-sourcing of video from mobile devices. In: Proceeding of the 11th Annual International Conference on MoBile Systems, Applications, and Services, pp 139–152

  46. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, pp 568–576

  47. Subudhi BN, Rout DK, Ghosh A (2019) Big data analytics for video surveillance. Multimed Tools Appl 78(18):26129–26162

    Article  Google Scholar 

  48. Sultana T, Wahid KA (2019) Choice of application layer protocols for next generation video surveillance using internet of video things. IEEE Access 7:41607–41624

    Article  Google Scholar 

  49. Tan H, Chen L (2014) An approach for fast and parallel video processing on apache hadoop clusters. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), IEEE, pp 1–6

  50. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: IEEE ICCV, pp 4489–4497

  51. Tu NA, Huynh-The T, Khan KU, Lee YK (2018) Ml-hdp: A hierarchical Bayesian nonparametric model for recognizing human actions in video. IEEE Trans Circuits Syst Video Technol 29(3):800–814

    Article  Google Scholar 

  52. Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S, et al. (2013) Apache Hadoop YARN: Yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, ACM, p 5

  53. Wang H, Kläser A, Schmid C, Liu CL (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79

    Article  MathSciNet  Google Scholar 

  54. Wang J, Liu Z, Wu Y, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. In: CVPR, pp 1290–1297

  55. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, Van Gool L (2016) Temporal segment networks: towards good practices for deep action recognition. In: ECCV, Springer, pp 20–36

  56. Whitelam C, Taborsky E, Blanton A, Maze B, Adams J, Miller T, Kalka N, Jain AK, Duncan JA, Allen K, et al. (2017) IARPA Janus benchmark-B face dataset. In: IEEE CVPRW, pp 90–98

  57. Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. In: ICIP, pp 3645–3649

  58. Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: IEEE CVPR

  59. Wu Y, Yang F, Xu Y, Ling H (2019) Privacy-protective-gan for privacy preserving face de-identification. J Comput Sci Technol 34(1):47–60

    Article  Google Scholar 

  60. Xu S, Yang G, Mu Y, Deng RH (2018) Secure fine-grained access control and data sharing for dynamic groups in the cloud. IEEE Trans Inf Forensics Secur 13(8):2101–2113

    Article  Google Scholar 

  61. Yang A, Zhang C, Chen Y, Zhuansun Y, Liu H (2019) Security and privacy of smart home systems based on the internet of things and stereo matching algorithms. IEEE Internet Things J 7(4):2521–2530

    Article  Google Scholar 

  62. Yang S, Wu B (2015) Large scale video data analysis based on spark. In: 2015 International Conference on Cloud Computing and Big Data, IEEE, pp 209–212

  63. Yaseen MU, Anjum A, Rana O, Hill R (2018) Cloud-based scalable object detection and classification in video streams. Futur Gener Comput Syst 80:286–298

    Article  Google Scholar 

  64. Yi S, Hao Z, Zhang Q, Zhang Q, Shi W, Li Q (2017) Lavea: Latency-aware video analytics on edge computing platform. In: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, pp 1–13

  65. Yu JY, Kim Y, Kim YG (2021) Intelligent video data security: a survey and open challenges. IEEE Access 9:26948–26967

    Article  Google Scholar 

  66. Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, USENIX Association, pp 2–2

  67. Zhang W, Xu L, Duan P, Gong W, Lu Q, Yang S (2015) A video cloud platform combing online and offline cloud computing technologies. Pers Ubiquitous Comput 19(7):1099–1110

    Article  Google Scholar 

  68. Zhang Y, Yang M, Zheng D, Lang P, Wu A, Chen C (2018) Efficient and secure big data storage system with leakage resilience in cloud computing. Soft Comput 22(23):7763–7772

    Article  Google Scholar 

  69. Zhou L, Pan S, Wang J, Vasilakos AV (2017) Machine learning on big data: opportunities and challenges. Neurocomputing 237:350–361

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Social Policy Grant and funded by the Nazarbayev University.

Author information

Authors and Affiliations

  1. Department of Computer Science, Nazarbayev University, 53 Kabanbay Batyr Ave, 010000, Nur-Sultan, Astana, Republic of Kazakhstan

    Nguyen Anh Tu & M. Fatih Demirci

  2. ICT Convergence Research Center, Kumoh National Institute of Technology, 61, Daehak-ro, Gumi-si, Gyeongsangbuk-do, 39177, Republic of Korea

    Thien Huynh-The

  3. College of Engineering and Computer Science, VinUniversity, Vinhomes Ocean Park, Gia Lam District, Hanoi, Vietnam

    Kok-Seng Wong

  4. Department of Computer Science and Engineering, Kyung Hee University, Global Campus, 1732 Deokyoungdae-ro, Giheung-gu, Yongin-si, Gyeonggi-do, 446-701, Republic of Korea

    Young-Koo Lee

  5. Department of Computer Engineering, TOBB University of Economics and Technology, Sogutozu Cad, No: 43, Ankara, Turkey

    M. Fatih Demirci

Authors
  1. Nguyen Anh Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thien Huynh-The
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kok-Seng Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Fatih Demirci
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Young-Koo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nguyen Anh Tu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tu, N.A., Huynh-The, T., Wong, KS. et al. Toward efficient and intelligent video analytics with visual privacy protection for large-scale surveillance. J Supercomput 77, 14374–14404 (2021). https://doi.org/10.1007/s11227-021-03865-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03865-7

Keywords

Search

Navigation