Elsevier

Computer Communications

Volume 184, 15 February 2022, Pages 128-136
Computer Communications

A lightweight framework for abnormal driving behavior detection

https://doi.org/10.1016/j.comcom.2021.12.007Get rights and content

Abstract

As one of the most serious hazards in the world, traffic accidents have caused huge casualties and property losses. The detection of abnormal behaviors of drivers is of great importance for intelligence transportation systems. However, the training of most abnormal behavior detection artificial intelligence algorithms demands huge computation and memory resources. Therefore, the training process is preferred to be done in cloud. In this paper, we propose a lightweight abnormal driving behavior detection framework, which applies intelligence to the edge and gives the IoT (Internet of Things) devices the ability to process and understand the data. The framework consists of four modules: bus driver wearing mask detection, bus driver abnormal motion detection, bus driver fatigue driving detection, and video recovery. Comprehensive experiments on real-world datasets have demonstrated the effectiveness and applicability of the proposed framework.

Introduction

Traffic safety servers as an important topic in the field of Intelligent Transportation Systems (ITS). As one of the most serious hazards, traffic accidents have resulted in huge property losses and casualties. It is reported that the number of traffic accidents deaths increases by 1.35 million every year [1]. The official report of China concludes that one of the main reasons behind accidents is improper driving behavior. Most accidents are caused by improper operation of the driver. On June 11, 2017, two buses collided head-on due to fatigue driving of the bus driver, resulting in 8 people being injured [2]. On August 13, 2015, due to the bus driver’s improper use of smartphones, 3 buses were rear-ended, and 15 people were injured [3]. Therefore, detecting abnormal behaviors of bus drivers can either provide an early warning of driver’s misbehavior, or be greatly significant for ensuring traffic safety, or both.

Moreover, wearing a mask for passengers and drivers can significantly reduce the risk of infection during the time of the COVID-19 pandemic spreading. Although wearing the mask can effectively cut off the path of infection, it also brings great challenges to detecting abnormal behaviors of drivers, such as yawning. Therefore, the detection of abnormal behaviors for faces with masks is an interesting and meaningful task.

As we all known, the traditional manual detection for abnormal driving behaviors is demonstrated to be inefficient, delay prompt response, and heavy monitoring tasks [4]. Recent years have witnessed the breakthroughs of Convolutional Neural Network (CNN)-based deep learning models in computer vision tasks, such as target detection, image classification, emotion recognition, and scene segmentation [5], [6]. However, the camera equipment cannot guarantee real-time collection and transmission of high-quality images in the process of driving leads to many problems such as noise, blur, and jitter in the collected video [7]. Therefore, it is necessary to improve the video resolution in question, namely, video recovery.

Recently, multimedia Internet of Things (IoT) has attracted much attention by integrating multiple capabilities such as computer vision, data processing, and network communication [8], [9], [10], [11]. These capabilities have been widely used in monitoring, event recognition, and automatic behavior analysis. Previous multimedia IoT systems process and analyze the video data collected by remote surveillance cameras in the central server [12]. Long-distance transmission of large amounts of video data may cause network delays and congestion [13], [14]. Edge computing is introduced as an emerging technology that can preprocess video data, resulting in distributed computing and transmission delay reduction. Edge computing actually provides a new paradigm to real-time video processing tasks.

In this paper, we propose a novel lightweight abnormal behavior detection framework for bus drivers capable of deploying in the IoT devices. The overall framework consists of four modules, including video recovery, mask detection, abnormal motion detection, and fatigue detection. Three contributions can be summarized as follows:

  • We propose a lightweight abnormal behavior detection framework that enables the IoT devices as the carriers of artificial intelligence algorithms.

  • By incorporating video recovery into the abnormal behavior detection framework, the problems of noise, blur, and jitter in the process of camera equipment acquisition and image transmission can be solved. Moreover, we adopt Intel Neural Compute Stick 2 (NCS) to improve the detection efficiency of IoT devices.

  • We conduct extensive experiments on real bus driver monitoring video datasets. The experiment verifies the excellent performance and applicability on IoT devices of the proposed framework.

The rest of this article is summarized as follows. In Section 2, we briefly introduce the related work. We elaborate on the proposed framework in Section 3. In Section 4, we exemplify the description of the experimental data and the analysis of the results. Finally, we summarize our paper in Section 5.

Section snippets

Mask detection

Generally, existing mask detection methods focus on face construction and identity recognition when wearing face masks. Qin et al. [15] proposed a new facemask-wearing condition model capable of classifying three kinds of facemask-wearing conditions. Sabbir et al. [16] have adopted the Principal Component Analysis (PCA) to recognize the person who is wearing a mask. Li et al. [17] utilized YOLOv3 [18] to finish the face detection task. The proposed method achieved nearly 94% accuracy. Nizam

Design of the proposed framework

The overall framework consists of four key modules: face mask detection module, abnormal behavior detection module, fatigue detection module, and video recovery module, as shown in Fig. 1.

To guarantee the accuracy of these detections, the original video datasets require to be restored and deburred. We adopt the Laplacian method to perform blur detection on the video and put the video that needs to be restored into the EDVR algorithm for recovery.

Face mask detection module can be divided into

Experiments on bus driver monitoring video restoration

We verify the necessity of video restoration by detecting bus drivers wearing masks in the video restoration experiment. We confirm that a small number of videos need to be restored. Therefore, we set the threshold to 115 by the variance of the Laplacian operator based on the statistical analysis of the bus driver monitoring dataset. A few clips that require video recovery account for 3%. We mark the video data with or without a mask. Following that, we detect the marked video through the above

Conclusion

We propose a lightweight abnormal driving behavior detection framework, including video restoration, bus driver wearing mask detection, bus driver abnormal motion detection, bus driver fatigue driving detection. Video restoration module filters the video with noise and fuzzy. In the mask detection module, we put forward a model based on face anchor detection and verify the performance of the model by comparing the experimental results of different datasets. Secondly, we propose a bus driver’s

CRediT authorship contribution statement

Mingliang Hou: Performed the analysis, Wrote the paper. Mengyuan Wang: Performed the analysis, Wrote the paper. Wenhong Zhao: Conceived and designed the analysis, Wrote the paper. Qichao Ni: Collected the data. Zhen Cai: Wrote the paper. Xiangjie Kong: Conceived and designed the analysis.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (49)

  • WanL. et al.

    Deep matrix factorization for trust-aware recommendation in social networks

    IEEE Trans. Netw. Sci. Eng.

    (2021)
  • JinS. et al.

    Application of deep learning methods in biological networks

    Brief. Bioinform.

    (2020)
  • KongX. et al.

    FedVCP: A federated-learning-based cooperative positioning scheme for social internet of vehicles

    IEEE Trans. Comput. Soc. Syst.

    (2021)
  • XiaF. et al.

    Exploring human mobility patterns in urban scenarios: A trajectory data perspective

    IEEE Commun. Mag.

    (2018)
  • KongX. et al.

    A federated learning-based license plate recognition scheme for 5G-enabled internet of vehicles

    IEEE Trans. Ind. Inf.

    (2021)
  • XiaF. et al.

    Graph learning: A survey

    IEEE Trans. Artif. Intell.

    (2021)
  • WanL. et al.

    Task-driven resource assignment in mobile edge computing exploiting evolutionary computation

    IEEE Wirel. Commun.

    (2019)
  • XiaF. et al.

    Ranking station importance with human mobility patterns using subway network datasets

    IEEE Trans. Intell. Transp. Syst.

    (2019)
  • KongX. et al.

    Real-time mask identification for COVID-19: An edge computing-based deep learning framework

    IEEE Internet Things J.

    (2021)
  • QinB. et al.

    Identifying facemask-wearing condition using image super-resolution with classification network to prevent COVID-19

    Sensors

    (2020)
  • M.S. Ejaz, M.R. Islam, M. Sifatullah, A. Sarker, Implementation of principal component analysis on masked and...
  • LiC. et al.

    Face detection based on YOLOv3

  • RedmonJ. et al.

    YOLOV3: An incremental improvement

    (2018)
  • DinN.U. et al.

    A novel GAN-based network for unmasking of masked face

    IEEE Access

    (2020)
  • Cited by (12)

    • Real-time fatigue driving detection system based on multi-module fusion

      2022, Computers and Graphics (Pergamon)
      Citation Excerpt :

      The third method is to judge the fatigue state according to the driver’s facial features [9,16]. This method mainly detects facial information by computer vision detection and processing technology [17,18], obtains the head position, blink frequency and mouth information [19–21], and evaluates the driver’s mental state after comprehensively processing various characteristic information [22]. When the driver is tired, the opening degree of the eyes will become smaller, and even completely close the eyes, the frequency of the eyes will also be lower than that in the normal state.

    View all citing articles on Scopus

    This work is partially supported by the National Natural Science Foundation of China (62072409), Zhejiang Provincial Natural Science Foundation, China (LR21F020003), and Fundamental Research Funds for the Provincial Universities of Zhejiang, China (RF-B2020001).

    View full text