Skip to main content
SpringerLink
Log in

Person Retrieval in Surveillance Videos Using Deep Soft Biometrics

  • Chapter
  • First Online:
Deep Biometrics

Part of the book series: Unsupervised and Semi-Supervised Learning ((UNSESUL))

Abstract

In the world of security and surveillance, it is very common to localize the person in the video. The localization turns to be non-trivial when the search is based on a linguistic query. This is due to the semantic gap between language based query and its processing by a machine. Typically query uses attributes like height, cloth color, cloth type, gender, and hair color, e.g., a tall male with a black t-shirt and pink short. Such attributes are known as soft biometrics. Conventionally, searching the person in the surveillance video is done manually by scouring through hours of videos, which is inefficient and time-consuming. Thus, an automatic person retrieval algorithm is an active area of research.

The chapter discusses automatic person retrieval algorithm using soft biometric attributes namely height, cloth color, gender. The algorithm uses Mask R-CNN for detection and semantic segmentation of the person. This removes background clutter and allows a precise color patch extraction for better classification. The algorithm uses the geometric model for height classification while color and gender models are built using the AlexNet: a deep neural network. The proposed algorithm is tested on the AVSS 2018 challenge II dataset and it analyzes person retrieval in challenging conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. A. Dantcheva, C. Velardo, A. D’Angelo, J.L. Dugelay, Bag of soft biometrics for person identification. Multimed. Tools Appl. 51(2), 739–77 (2011)

    Article  Google Scholar 

  2. A. Dantcheva, P. Elia, A. Ross, What else does your biometric data reveal? a survey on soft biometrics. IEEE Trans. Inf. Forensics Secur. 11(3), 441–67 (2016)

    Article  Google Scholar 

  3. J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, ImageNet: a large-scale hierarchical image database, in IEEE Conference on 2009 Computer Vision and Pattern Recognition, CVPR (IEEE, Piscataway, 2009), pp. 248–255

    Google Scholar 

  4. Y. Deng, P. Luo, C.C. Loy, X. Tang, Pedestrian attribute recognition at far distance, in Proceedings of the 22nd ACM international conference on Multimedia (ACM, New York, 2014), pp. 789–792

    Google Scholar 

  5. S. Denman, M. Halstead, C. Fookes, S. Sridharan, Searching for people using semantic soft biometric descriptions. Pattern Recogn. Lett. 68, 306–315 (2015)

    Article  Google Scholar 

  6. H. Galiyawala, K. Shah, V. Gajjar, M.S. Raval, Person retrieval in surveillance video using height, color, and gender, in 15th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) (IEEE, New Zealand 2018)

    Google Scholar 

  7. D. Gray, S. Brennan, H. Tao, Evaluating appearance models for recognition, reacquisition, and tracking. In Proceedings of IEEE International Workshop on Performance Evaluation for Tracking and Surveillance (PETS), vol. 3, no. 5, (2007), pp. 1–7. Citeseer

    Google Scholar 

  8. M. Halstead, S. Denman, S. Sridharan, C.B. Fookes, Locating people in video from semantic descriptions: a new database and approach, in Proceedings of the 22nd International Conference on Pattern Recognition (IEEE, Piscataway, 2014), pp. 4501–4506

    Google Scholar 

  9. M. Halstead, S. Denman, C. Fookes, Y. Tian, M. Nixon, Semantic person retrieval in surveillance using soft biometrics: AVSS 2018 challenge II, in IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) (IEEE, New Zealand, 2018)

    Google Scholar 

  10. K. He, G. Gkioxari, P. Dollar, R. Girshick, Mask R-CNN, in 2017 IEEE International Conference on Computer Vision (ICCV) (IEEE, Piscataway, 2017), pp. 2980–2988

    Book  Google Scholar 

  11. A.K. Jain, S.C. Dass, K. Nandakumar, Soft biometric traits for personal recognition systems, in International Conference on Biometric Authentication (ICBA) (Springer, Berlin, 2004), pp. 731–738

    Google Scholar 

  12. A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012), pp. 1097–1105

    Google Scholar 

  13. R. Layne, T.M. Hospedales, S. Gong, Q. Mary, Person re-identification by attributes. In BMVC, vol. 2, no. 3, (2012), p. 8

    Google Scholar 

  14. D. Li, X. Chen, K. Huang, Multi-attribute learning for pedestrian attribute recognition in surveillance scenarios, in 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), (IEEE, Piscataway, 2015), pp. 111–115

    Google Scholar 

  15. D. Li, Z. Zhang, X. Chen, K. Huang, A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE Trans. Image Process. 28(4), 1575–1590 (2019)

    Article  MathSciNet  Google Scholar 

  16. T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C.L. Zitnick, Microsoft coco: common objects in context, in European Conference on Computer Vision (Springer, Cham, 2014), pp. 740–755

    Google Scholar 

  17. C. Liu, S. Gong, C.C. Loy, X. Lin, Person re-identification: what features are important?, in European Conference on Computer Vision (Springer, Berlin, 2012) pp. 391–401

    Google Scholar 

  18. M.D. MacLeod, J.N. Frowley, J.W. Shepherd, Whole body information: its relevance to eyewitnesses, in Adult Eyewitness Testimony: Current Trends and Developments, eds. by D.F. Ross, J.D. Read, M.P. Toglia (Cambridge University Press, New York, 1994), pp. 125–143

    Chapter  Google Scholar 

  19. M.S. Raval, Digital video forensics: description based person identification. CSI Commun. 39(12), 9–11 (2016)

    Google Scholar 

  20. D.A. Reid, M.S. Nixon, Imputing human descriptions in semantic biometrics, in Proceedings of the 2nd ACM Workshop on Multimedia in Forensics, Security and Intelligence 2010 (ACM, New York, 2010), pp. 25–30

    Google Scholar 

  21. D.A. Reid, S. Samangooei, C. Chen, M.S. Nixon, A. Ross, Soft biometrics for surveillance: an overview, in Handbook of Statistics, vol. 31 (Elsevier, Amsterdam, 2013), pp. 327–352

    Google Scholar 

  22. S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks, in Advances in Neural Information Processing Systems (2015), pp. 91–99

    Google Scholar 

  23. S. Samangooei, M. Nixon, B. Guo, The use of semantic human description as a soft biometric, in Proceedings of 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems (IEEE, Arlington, 2008)

    Google Scholar 

  24. M.S. Sarfraz, A. Schumann, Y. Wang, R. Stiefelhagen, Deep view-sensitive pedestrian attribute inference in an end-to-end model, in BMVC, 2017.

    Google Scholar 

  25. P. Shah, M.S. Raval, S. Pandya, S. Chaudhary, A. Laddha, H. Galiyawala, Description based person identification: use of clothes color and type, in Computer Vision, Pattern Recognition, Image Processing, and Graphics: 6th National Conference, NCVPRIPG 2017 (Mandi, 2017), Revised Selected Papers 6 (Springer, Singapore, 2018), pp. 457–469

    Google Scholar 

  26. P. Sudowe, H. Spitzer, B. Leibe, Person attribute recognition with a jointly-trained holistic CNN model, in Proceedings of the IEEE International Conference on Computer Vision Workshops (2015), pp. 87–95

    Google Scholar 

  27. R. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. 3(4), 323–344 (1987)

    Article  Google Scholar 

  28. D.A. Vaquero, R.S. Feris, D. Tran, L. Brown, A. Hampapur, M. Turk, Attribute-based people search in surveillance environments, in 2009 Workshop on Applications of Computer Vision (WACV) (IEEE, Piscataway, 2009), pp. 1–8

    Book  Google Scholar 

  29. J. Wang, X. Zhu, S. Gong, W. Li, Attribute recognition by joint recurrent learning of context and correlation, in IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  30. C. Yao, B. Feng, D. Li, J. Li, Hierarchical pedestrian attribute recognition based on adaptive region localization, in 2017 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) (IEEE, Piscataway, 2017), pp. 471–476

    Google Scholar 

  31. J. Zhu, S. Liao, Z. Lei, D. Yi, S. Li, Pedestrian attribute classification in surveillance: database and evaluation, in Proceedings of the IEEE International Conference on Computer Vision Workshops (2013), pp. 331–338

    Google Scholar 

  32. J. Zhu, S. Liao, D. Yi, Z. Lei, S.Z. Li, Multi-label CNN based pedestrian attribute learning for soft biometrics, in 2015 International Conference on Biometrics (ICB) (IEEE, Piscataway, 2015), pp. 535–540

    Book  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Board of Research in Nuclear Sciences (BRNS) for a generous grant (36(3)/14/20/2016-BRNS/36020). We acknowledge the support of NVIDIA Corporation for a donation of the Quadro K5200 GPU used for this research. We would also like to thank the SAIVT Research Labs at Queensland University of Technology (QUT) for creating the challenging dataset. Thanks to Kenil Shah and Vandit Gajjar for their help during the implementation of the work.

Author information

Authors and Affiliations

  1. School of Technology, Pandit Deendayal Petroleum University, Gandhinagar, India

    Hiren J. Galiyawala & Mehul S. Raval

  2. Bhabha Atomic Research Centre (BARC), Mumbai, India

    Anand Laddha

Authors
  1. Hiren J. Galiyawala
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehul S. Raval
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anand Laddha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hiren J. Galiyawala .

Editor information

Editors and Affiliations

  1. School of Computing and Communication, InfoLab21, Lancaster University, Lancaster, UK

    Richard Jiang

  2. School of Information Technology, Deakin University, Waurn Ponds, VIC, Australia

    Chang-Tsun Li

  3. Electrical Engineering and Computer Science, School of Electronics, Queen’s University Belfast, Belfast, UK

    Danny Crookes

  4. Technical University of Denmark, Kgs. Lyngby, Denmark

    Weizhi Meng

  5. GREYC research Lab, ENSICAEN, Caen, France

    Christophe Rosenberger

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Galiyawala, H.J., Raval, M.S., Laddha, A. (2020). Person Retrieval in Surveillance Videos Using Deep Soft Biometrics. In: Jiang, R., Li, CT., Crookes, D., Meng, W., Rosenberger, C. (eds) Deep Biometrics. Unsupervised and Semi-Supervised Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-32583-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32583-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32582-4

  • Online ISBN: 978-3-030-32583-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation