Skip to main content
SpringerLink
Log in

Local instance and context dictionary-based detection and localization of abnormalities

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Studies on contextual abnormality detection and localization for images and videos are presented in this work. The task of detecting abnormalities becomes challenging while considering the context in the scene. Some object which is normal in one scenario may be considered as abnormal in another. We present conceptually simple, flexible and a general framework, by incorporating instance segmentation, skip-gram with negative sampling and isolation forest for detecting and localizing contextual abnormality in images and videos. The skip-gram-based model is generally used for word2vec in natural language processing for finding the similarity between words. In this work, we extended them to detect the object-based abnormality in the images and video. Then we introduce the voting technique, which overcomes the variable-length feature vector issues; the decision of normal or abnormal object is based on this technique by considering the output from the isolation forest. We consider the anomalous events as scenarios having a different distribution from the normal settings such as a less frequently seen object in a given combination, the increase in the number of specific objects category, the object’s presence at unseen distance and occupancy of the out-of-vocabulary object. We observed that the proposed framework works in the proximity of multiple object categories and camera motion in the natural capture videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Aggarwal, C.C.: Outlier Analysis. Springer, Berlin (2013)

    Book  Google Scholar 

  2. Andrews, J.T., Tanay, T., Morton, E.J., Griffin, L.D.: Transfer representation-learning for anomaly detection. In: International Conference on Machine Learning (2016)

  3. Bastan, M., Gudukbay, U., Ulusoy, O.: Segmentation-based extraction of important objects from video for object-based indexing. In: IEEE International Conference on Multimedia and Expo, pp. 1357–1360. IEEE (2008)

  4. Bengio, Y.: Learning deep architectures for AI. Found. Trends ® Mach. Learn. 2(1), 1–127 (2009)

    Article  MathSciNet  Google Scholar 

  5. Bengio, Y., Goodfellow, I.J., Courville, A.: Deep Learning (2015). http://www.iro.umontreal.ca/~bengioy/dlbook. Book in preparation for MIT Press

  6. Biswas, S., Babu, R.V.: Anomaly detection in compressed h.264/avc video. Multimed. Tools Appl. 74(24), 11099–11115 (2015)

    Article  Google Scholar 

  7. Bontemps, L., McDermott, J., Le-Khac, N.A., et al.: Collective anomaly detection based on long short-term memory recurrent neural networks. In: International Conference on Future Data and Security Engineering, pp. 141–152. Springer (2016)

  8. Chalapathy, R., Chawla, S.: Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407 (2019)

  9. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: A survey. ACM Comp. Surv. 41(3), 15 (2009)

    Article  Google Scholar 

  10. Choi, M.J., Torralba, A., Willsky, A.S.: Context models and out-of-context objects. Pattern Recogn. Lett. 33(7), 853–862 (2012)

    Article  Google Scholar 

  11. Chong, Y.S., Tay, Y.H.: Modeling representation of videos for anomaly detection using deep learning: A review. arXiv preprint arXiv:1505.00523 (2015)

  12. Dai, J., Li, Y., He, K., Sun, J.: R-fcn: Object detection via region-based fully convolutional networks. In: Advances in Neural Information Processing Systems, pp. 379–387 (2016)

  13. Diamantopoulos, G., Spann, M.: Event detection for intelligent car park video surveillance. Real-Time Imaging 11(3), 233–243 (2005)

    Article  Google Scholar 

  14. Goldberg, Y., Levy, O.: word2vec explained: Deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722 (2014)

  15. Google: Word2vec in c. https://code.google.com/archive/p/word2vec/. Accessed: 2018

  16. Habeeb, R.A.A., Nasaruddin, F., Gani, A., Hashem, I.A.T., Ahmed, E., Imran, M.: Real-time big data processing for anomaly detection: A survey. Int. J. Inf. Manag. 45, 289–307 (2019)

    Article  Google Scholar 

  17. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988. IEEE (2017)

  18. Javan-Roshtkhari, M.: Visual event description in videos. Ph.D. thesis, McGill University (2014)

  19. Javan Roshtkhari, M., Levine, M.D.: Online dominant and anomalous behavior detection in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2611–2618 (2013)

  20. Kiran, B., Thomas, D., Parakkal, R.: An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. J. Imag. 4(2), 36 (2018)

    Article  Google Scholar 

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  22. Leach, M.J., Sparks, E.P., Robertson, N.M.: Contextual anomaly detection in crowded surveillance scenes. Pattern Recogn. Lett. 44, 71–79 (2014)

    Article  Google Scholar 

  23. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  24. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)

    Google Scholar 

  25. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer (2014)

  26. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE (2008)

  27. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation-based anomaly detection. ACM Trans. Knowl. Discovery Data (TKDD) 6(1), 3 (2012)

    Google Scholar 

  28. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  29. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1975–1981. IEEE (2010)

  30. Matlab: Specify polygonal region of interest. https://www.mathworks.com/help/images/ref/roipoly.html. Accessed: 2018

  31. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems. Curran Associates, Inc. pp. 3111–3119 (2013)

  32. Oh, J., Kim, H.I., Park, R.H.: Context-based abnormal object detection using the fully-connected conditional random rields. Pattern Recogn. Lett. 98, 16–25 (2017)

    Article  Google Scholar 

  33. Popoola, O.P., Wang, K.: Video-based abnormal human behavior recognition - a review. IEEE Trans. Sys. Man., Cyber. Part C: Appl., Rev. 42(6), 865–878 (2012)

    Article  Google Scholar 

  34. Ravanbakhsh, M., Nabi, M., Mousavi, H., Sangineto, E., Sebe, N.: Plug-and-play cnn for crowd motion analysis: An application in abnormal event detection. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1689–1698. IEEE (2018)

  35. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

  36. Ryan, D., Denman, S., Fookes, C., Sridharan, S.: Textures of optical flow for real-time anomaly detection in crowds. In: 8th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS), pp. 230–235. IEEE (2011)

  37. Sabokrou, M., Fayyaz, M., Fathy, M., Klette, R.: Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26(4), 1992–2004 (2017)

    Article  MathSciNet  Google Scholar 

  38. Sabokrou, M., Fayyaz, M., Fathy, M., Moayed, Z., Klette, R.: Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Underst. 172, 88–97 (2018)

    Article  Google Scholar 

  39. Saleh, B., Farhadi, A., Elgammal, A.: Object-centric anomaly detection by attribute-based reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 787–794 (2013)

  40. Saykol, E., Bastan, M., Güdükbay, U., Ulusoy, Ö.: Keyframe labeling technique for surveillance event classification. Optical Eng. 49(11), 117203-1–117203-12 (2010)

  41. Şaykol, E., Güdükbay, U., Ulusoy, Ö.: Scenario-based query processing for video-surveillance archives. Eng. Appl. Artif. Intell. 23(3), 331–345 (2010)

    Article  Google Scholar 

  42. Sharma, M.K., Sarcar, S., Sheet, D., Biswas, P.K.: Limitations with measuring performance of techniques for abnormality localization in surveillance video and how to overcome them? In: Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, p. 75. ACM (2016)

  43. Sharma, M.K., Sheet, D., Biswas, P.K.: Abnormality detecting deep belief network. In: Proceedings of the International Conference on Advances in Information Communication Technology & Computing, p. 11. ACM (2016)

  44. Sharma, M.K., Sheet, D., Biswas, P.K.: Image embedding for detecting irregularity. In: Proceedings of 3rd International Conference on Computer Vision and Image Processing (CVIP), vol. 2, p. 243. Springer Nature (2019)

  45. Sharma, M.K., Sheet, D., Biswas, P.K.: Spatiotemporal deep networks for detecting abnormality in videos. Multimedia Tools and Applications pp. 1–32 (2020)

  46. Sodemann, A.A., Ross, M.P., Borghetti, B.J.: A review of anomaly detection in automated surveillance. IEEE Trans. Sys. Man., Cyber. Part C: Appl., Rev. 42(6), 1257–1272 (2012)

    Article  Google Scholar 

  47. Tan, H., Zhai, Y., Liu, Y., Zhang, M.: Fast anomaly detection in traffic surveillance video based on robust sparse optical flow. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1976–1980. IEEE (2016)

  48. Thi, N.N., Cao, V.L., Le-Khac, N.A.: One-class collective anomaly detection based on long short-term memory recurrent neural networks. (2018). arXiv preprint arXiv:1802.00324

  49. Weng, Y., Liu, L.: A collective anomaly detection approach for multidimensional streams in mobile service security. IEEE Access 7, 49157–49168 (2019). https://doi.org/10.1109/ACCESS.2019.2909750

    Article  Google Scholar 

  50. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. (2015). arXiv preprint arXiv:1510.01553

  51. Yan, W., Yu, L.: On accurate and reliable anomaly detection for gas turbine combustors: A deep learning approach. (2019). arXiv preprint arXiv:1908.09238

  52. Yong, S.P., Deng, J.D., Purvis, M.K.: Novelty detection in wildlife scenes through semantic context modelling. Pattern Recogn. 45(9), 3439–3450 (2012)

    Article  Google Scholar 

  53. yoonkim: Word2vec in lua. https://github.com/yoonkim/word2vec_torch. Accessed: 2018

  54. Zhou, G., Wu, Y.: Anomalous event detection based on self-organizing map for supermarket monitoring. In: Proceedings of International Conference on Information Engineering and Computer Science, pp. 1–4. IEEE (2009)

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology, Kharagpur, India

    M. K. Sharma, D. Sheet & P. K. Biswas

Authors
  1. M. K. Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. D. Sheet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. K. Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. K. Sharma.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, M.K., Sheet, D. & Biswas, P.K. Local instance and context dictionary-based detection and localization of abnormalities. Machine Vision and Applications 32, 69 (2021). https://doi.org/10.1007/s00138-021-01179-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-021-01179-5

Keywords

Search

Navigation