Skip to main content
Log in

Context-based person identification framework for smart video surveillance

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Smart video surveillance (SVS) applications enhance situational awareness by allowing domain analysts to focus on the events of higher priority. SVS approaches operate by trying to extract and interpret higher “semantic” level events that occur in video. One of the key challenges of SVS is that of person identification where the task is for each subject that occurs in a video shot to identify the person it corresponds to. The problem of person identification is especially challenging in resource-constrained environments where transmission delay, bandwidth restriction, and packet loss may prevent the capture of high-quality data. Conventional person identification approaches which primarily are based on analyzing facial features are often not sufficient to deal with poor-quality data. To address this challenge, we propose a framework that leverages heterogeneous contextual information together with facial features to handle the problem of person identification for low-quality data. We first investigate the appropriate methods to utilize heterogeneous context features including clothing, activity, human attributes, gait, people co-occurrence, and so on. We then propose a unified approach for person identification that builds on top of our generic entity resolution framework called RelDC, which can integrate all these context features to improve the quality of person identification. This work thus links one well-known problem of person identification from the computer vision research area (that deals with video/images) with another well-recognized challenge known as entity resolution from the database and AI/ML areas (that deals with textual data). We apply the proposed solution to a real-world dataset consisting of several weeks of surveillance videos. The results demonstrate the effectiveness and efficiency of our approach even on low-quality video data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. \(704 \times 480\) resolution per frame.

  2. We use the standard definition of relational datasets as used in the database literature.

References

  1. Project sherlock @ uci. http://sherlock.ics.uci.edu

  2. Ahonen, T., Hadid, A., Pietik, M.: Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Anal. (2006)

  3. An, L., Kafai, M., Bhanu, B.: Dynamic bayesian network for unconstrained face recognition in surveillance camera networks. IEEE J. Emerg. Sel. Top. Circuits Syst. 3(2), 155–164 (2013)

    Article  Google Scholar 

  4. Balcan, M., Blum, A., Choi, P.P., Lafferty, J., Pantano, B., Rwebangira, M.R., Zhu, X.: Person identification in webcam images: An application of semi-supervised learning. In: ICML Workshop on Learning from Partially Classified Training Data (2005)

  5. Chaudhuri, S., Ganjam, K., Ganti, V., Kapoor, R., Narasayya, V., Vassilakis, T.: Data cleaning in Microsoft SQL Server 2005. In: ACM SIGMOD Conference (2005)

  6. Chen, S., Kalashnikov, D.V., Mehrotra, S.: Adaptive graphical approach to entity resolution. In: Proceedings of ACM IEEE Joint Conference on Digital Libraries (JCDL 2007), Vancouver, British Columbia, Canada, June 17–23 (2007)

  7. Chen, Z., Kalashnikov, D.V., Mehrotra, S.: Exploiting relationships for object consolidation. In: Proceedings of International ACM SIGMOD Workshop on Information Quality in Information Systems (ACM IQIS 2005), Baltimore, MD, USA, June 17 (2005)

  8. Chen, Z.S., Kalashnikov, D.V., Mehrotra, S.: Exploiting context analysis for combining multiple entity resolution systems. In: Proceedings of ACM SIGMOD International Conference on Management of Data (ACM SIGMOD 2009), Providence, RI, USA, June 29–July 2 (2009)

  9. Cristani, M., Bicego, M., Murino, V.: Audio-visual event recognition in surveillance video sequences. IEEE Trans. Multimed. (2007)

  10. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)

  11. Gallagher, A., Chen, T.: Clothing cosegmentation for recognizing people. In: IEEE CVPR (2008)

  12. Gao, Y., Tang, J., Hong, R., Yan, S., Dai, Q., Zhang, N., Chua, T.: Camera constraint-free view-based 3D object retrieval. IEEE Trans. Image Process. 21(4), 2269–2281 (2012)

    Article  MathSciNet  Google Scholar 

  13. Gao, Y., Wang, M., Tao, D., Ji, R., Dai, Q.: 3D object retrieval and recognition with hypergraph analysis. IEEE Trans. Image Process. 21(9), 4290–4303 (2012)

    Article  MathSciNet  Google Scholar 

  14. Gao, Y., Wang, M., Zha, Z., Shen, J., Li, X., Wu, X.: Visual-textual joint relevance learning for tag-based social image search. IEEE Trans. Image Process. 22(1), 363–376 (2013)

    Article  MathSciNet  Google Scholar 

  15. Grossman, R.L., Kamath, C., Kegelmeyer, P., Kumar, V., Namburu, R.R.: Data Mining for Scientific and Engineering Applications. Kluwer, Dordrecht (2001)

    Book  MATH  Google Scholar 

  16. Han, J., Altman, R.B., Kumar, V., Mannila, H., Pregibon, D.: Emerging scientific applications in data mining. Commun. ACM 45(8), 54–58 (2002)

    Article  Google Scholar 

  17. Hong, R., Tang, J., Tan, H.-K., Ngo, C.-W., Yan, S., Chua, T.-S.: Beyond search: event-driven summarization for web videos. TOMCCAP 7(4), 35 (2011)

    Article  Google Scholar 

  18. Hong, R., Wang, M., Li, G., Nie, L., Zha, Z.-J., Chua, T.-S.: Multimedia question answering. IEEE MultiMed. 19(4), 72–78 (2012)

    Article  Google Scholar 

  19. Kalashnikov, D.V.: Super-EGO: fast multi-dimensional similarity join. Int. J. Very Large Data Bases 4(2), 561–585 (2013)

    Article  MathSciNet  Google Scholar 

  20. Kalashnikov, D.V., Chen, Z., Mehrotra, S., Nuray, R.: Web people search via connection analysis. IEEE Trans. Knowl. Data Eng. 20(11) (2008)

  21. Kalashnikov, D.V., Mehrotra, S.: Domain-independent data cleaning via analysis of entity-relationship graph. ACM Trans Database Syst. 31(2), 716–767 (2006)

    Article  Google Scholar 

  22. Kalashnikov, D.V., Mehrotra, S., Chen, S., Nuray, R., Ashish, N.: Disambiguation algorithm for people search on the web. In: Proceedings of the IEEE 23rd International Conference on Data Engineering (IEEE ICDE 2007), Istanbul, Turkey, April 16–20 2007 (short publication)

  23. Kalashnikov, D.V., Mehrotra, S., Chen, Z.: Exploiting relationships for domain-independent data cleaning. In: SIAM International Conference on Data Mining (SDM 2005), Newport Beach, CA, USA, April 21–23 (2005)

  24. Kantardzic, M., Zurada, J.: Next Generation of Data-Mining Applications. Wiley, London (2005)

    MATH  Google Scholar 

  25. Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Describable visual attributes for face verification and image search. In: IEEE TPAMI (2011)

  26. McCallum, A., Wellner, B.: Object consolidation by graph partitioning with a conditionally-trained distance metric. In: KDD Workshop on Data Cleaning, Record Linkage and Object Consolidation (2003)

  27. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Self-tuning in graph-based reference disambiguation. In: Proceedings of the 12th International Conference on Database Systems for Advanced Applications (DASFAA 2007), Springer LNCS, Bangkok, Thailand, April 9–12 (2007)

  28. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Exploiting web querying for web people search. ACM Trans. Database Syst. 37(1) (2012)

  29. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Adaptive connection strength models for relationship-based entity resolution. ACM J. Data Inf. Qual. 4(2) (2013)

  30. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S., Yu, Y.: Attribute and object selection queries on objects with probabilistic attributes. ACM Trans. Database Syst. 37(1) (2012)

  31. Tang, J., Hong, R., Yan, S., Chua, T.-S., Qi, G.-J., Jain, R.: Image annotation by knn-sparse graph-based label propagation over noisily tagged web images. ACM TIST 2(2), 14 (2011)

    Google Scholar 

  32. Tang, J., Yan, S., Hong, R., Qi, G.-J., Chua, T.-S.: Inferring semantic concepts from community-contributed images and noisy tags. ACM Multimed., pp. 223–232 (2009)

  33. Tian, Y., Brown, L.M.G., Hampapur, A., Senior, M.L., Shu, C.: Ibm smart surveillance system (s3): event based video surveillance system with an open and extensible framework. Mach. Vis. Appl. (2008)

  34. Vaisenberg, R., Mehrotra, S., Ramanan, D.: Semartcam scheduler: emantics driven eal- ime data collection from indoor amera networks to maximize event detection. J. Real Time Image Process. 5(4) (2010)

  35. Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J., Song, Y.: Unified video annotation via multigraph learning. IEEE Trans. Circuits Syst. Video Technol. 19(5), 733–746 (2009)

    Article  Google Scholar 

  36. Wang, M., Li, H., Tao, D., Lu, K., Wu, X.: Multimodal graph-based reranking for web image search. IEEE Trans. Image Process. 21(11), 4649–4661 (2012)

    Article  MathSciNet  Google Scholar 

  37. Yang, M., Yu, K.: Real-time clothing recognition in surveillance videos. In: ICIP (2011)

  38. Zhang, L., Kalashnikov, D.V., Mehrotra, S.: A unified framework for context assisted face clustering. In: ACM International Conference on Multimedia Retrieval (ACM ICMR 2013), Dallas, Texas, USA, April 16–19 (2013)

  39. Zhang, L., Vaisenberg, R., Mehrotra, S., Kalashnikov, D.V.: Video entity resolution: applying er techniques for smart video surveillance. In: IQ2S Workshop in Conjunction with IEEE PERCOM 2011 (2011)

  40. Zhong, H., Shi, J., Visontai, M.: Detecting unusual activity in video. In: CVPR (2004)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Liyan Zhang.

Additional information

This work was supported in part by NSF grants CNS-1118114, CNS-1059436, CNS-1063596. It is part of NSF supported project Sherlock @ UCI (http://sherlock.ics.uci.edu): a UC Irvine project on Data Quality and Entity Resolution [1].

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, L., Kalashnikov, D.V., Mehrotra, S. et al. Context-based person identification framework for smart video surveillance. Machine Vision and Applications 25, 1711–1725 (2014). https://doi.org/10.1007/s00138-013-0535-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-013-0535-8

Keywords

Navigation