Deep Learning-Based Image Captioning Used to Recognize Malicious or Emergency Activity in an Image

Abstract

Image captioning is one of the fundamental problems of computer vision technology. Our objective in this work is to use existing image captioning architectures and extend it to develop an architecture to recognize activity happening in an image. We will be focusing on classifying the output of the image captioning model to either positive activity or negative activity. This architecture can be used to detect malicious or emergencies from a CCTV camera automatically. Videos are nothing but a series of frames. In other words, videos are images with temporal dimensions added to it. We will solve this problem in two steps. First, we will have an image captioning model where we will use deep neural networks to produce caption, and then we will use classification models to produce our final output. There are existing deep learning architectures in computer vision that can detect fire, smoke, accident on highways, or any other malicious or emergencies, but these systems exist individually. The proposed system has the potential of automatically detecting any emergency situations from an image.