A lightweight framework for abnormal driving behavior detection☆
Introduction
Traffic safety servers as an important topic in the field of Intelligent Transportation Systems (ITS). As one of the most serious hazards, traffic accidents have resulted in huge property losses and casualties. It is reported that the number of traffic accidents deaths increases by 1.35 million every year [1]. The official report of China concludes that one of the main reasons behind accidents is improper driving behavior. Most accidents are caused by improper operation of the driver. On June 11, 2017, two buses collided head-on due to fatigue driving of the bus driver, resulting in 8 people being injured [2]. On August 13, 2015, due to the bus driver’s improper use of smartphones, 3 buses were rear-ended, and 15 people were injured [3]. Therefore, detecting abnormal behaviors of bus drivers can either provide an early warning of driver’s misbehavior, or be greatly significant for ensuring traffic safety, or both.
Moreover, wearing a mask for passengers and drivers can significantly reduce the risk of infection during the time of the COVID-19 pandemic spreading. Although wearing the mask can effectively cut off the path of infection, it also brings great challenges to detecting abnormal behaviors of drivers, such as yawning. Therefore, the detection of abnormal behaviors for faces with masks is an interesting and meaningful task.
As we all known, the traditional manual detection for abnormal driving behaviors is demonstrated to be inefficient, delay prompt response, and heavy monitoring tasks [4]. Recent years have witnessed the breakthroughs of Convolutional Neural Network (CNN)-based deep learning models in computer vision tasks, such as target detection, image classification, emotion recognition, and scene segmentation [5], [6]. However, the camera equipment cannot guarantee real-time collection and transmission of high-quality images in the process of driving leads to many problems such as noise, blur, and jitter in the collected video [7]. Therefore, it is necessary to improve the video resolution in question, namely, video recovery.
Recently, multimedia Internet of Things (IoT) has attracted much attention by integrating multiple capabilities such as computer vision, data processing, and network communication [8], [9], [10], [11]. These capabilities have been widely used in monitoring, event recognition, and automatic behavior analysis. Previous multimedia IoT systems process and analyze the video data collected by remote surveillance cameras in the central server [12]. Long-distance transmission of large amounts of video data may cause network delays and congestion [13], [14]. Edge computing is introduced as an emerging technology that can preprocess video data, resulting in distributed computing and transmission delay reduction. Edge computing actually provides a new paradigm to real-time video processing tasks.
In this paper, we propose a novel lightweight abnormal behavior detection framework for bus drivers capable of deploying in the IoT devices. The overall framework consists of four modules, including video recovery, mask detection, abnormal motion detection, and fatigue detection. Three contributions can be summarized as follows:
- •
We propose a lightweight abnormal behavior detection framework that enables the IoT devices as the carriers of artificial intelligence algorithms.
- •
By incorporating video recovery into the abnormal behavior detection framework, the problems of noise, blur, and jitter in the process of camera equipment acquisition and image transmission can be solved. Moreover, we adopt Intel Neural Compute Stick 2 (NCS) to improve the detection efficiency of IoT devices.
- •
We conduct extensive experiments on real bus driver monitoring video datasets. The experiment verifies the excellent performance and applicability on IoT devices of the proposed framework.
The rest of this article is summarized as follows. In Section 2, we briefly introduce the related work. We elaborate on the proposed framework in Section 3. In Section 4, we exemplify the description of the experimental data and the analysis of the results. Finally, we summarize our paper in Section 5.
Section snippets
Mask detection
Generally, existing mask detection methods focus on face construction and identity recognition when wearing face masks. Qin et al. [15] proposed a new facemask-wearing condition model capable of classifying three kinds of facemask-wearing conditions. Sabbir et al. [16] have adopted the Principal Component Analysis (PCA) to recognize the person who is wearing a mask. Li et al. [17] utilized YOLOv3 [18] to finish the face detection task. The proposed method achieved nearly 94% accuracy. Nizam
Design of the proposed framework
The overall framework consists of four key modules: face mask detection module, abnormal behavior detection module, fatigue detection module, and video recovery module, as shown in Fig. 1.
To guarantee the accuracy of these detections, the original video datasets require to be restored and deburred. We adopt the Laplacian method to perform blur detection on the video and put the video that needs to be restored into the EDVR algorithm for recovery.
Face mask detection module can be divided into
Experiments on bus driver monitoring video restoration
We verify the necessity of video restoration by detecting bus drivers wearing masks in the video restoration experiment. We confirm that a small number of videos need to be restored. Therefore, we set the threshold to 115 by the variance of the Laplacian operator based on the statistical analysis of the bus driver monitoring dataset. A few clips that require video recovery account for 3%. We mark the video data with or without a mask. Following that, we detect the marked video through the above
Conclusion
We propose a lightweight abnormal driving behavior detection framework, including video restoration, bus driver wearing mask detection, bus driver abnormal motion detection, bus driver fatigue driving detection. Video restoration module filters the video with noise and fuzzy. In the mask detection module, we put forward a model based on face anchor detection and verify the performance of the model by comparing the experimental results of different datasets. Secondly, we propose a bus driver’s
CRediT authorship contribution statement
Mingliang Hou: Performed the analysis, Wrote the paper. Mengyuan Wang: Performed the analysis, Wrote the paper. Wenhong Zhao: Conceived and designed the analysis, Wrote the paper. Qichao Ni: Collected the data. Zhen Cai: Wrote the paper. Xiangjie Kong: Conceived and designed the analysis.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (49)
- et al.
Real-time dissemination of emergency warning messages in 5G enabled selfish vehicular social networks
Comput. Netw.
(2020) - et al.
A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic
Measurement
(2021) - et al.
Separate visual pathways for perception and action
Trends Neurosci.
(1992) - et al.
Automatic recognition of alertness and drowsiness from EEG by an artificial neural network
Med. Eng. Phys.
(2002) - et al.
Dynamic driver fatigue detection using hidden Markov model in real driving condition
Expert Syst. Appl.
(2016) - et al.
Can SVM be used for automatic EEG detection of drowsiness during car driving?
Saf. Sci.
(2009) - et al.
An investigation into unsafe behaviors and traffic accidents involving unlicensed drivers: a perspective for alignment measurement
Int. J. Environ. Res. Public Health
(2020) - et al.
A qualitative study exploring how city bus drivers manage sleepiness and fatigue
Chronobiol. Int.
(2020) - et al.
Driving behavior analysis of city buses based on real-time GNSS traces and road information
Sensors
(2021) - et al.
TBI2Flow: Travel behavioral inertia based long-term taxi passenger flow prediction
World Wide Web
(2020)
Deep matrix factorization for trust-aware recommendation in social networks
IEEE Trans. Netw. Sci. Eng.
Application of deep learning methods in biological networks
Brief. Bioinform.
FedVCP: A federated-learning-based cooperative positioning scheme for social internet of vehicles
IEEE Trans. Comput. Soc. Syst.
Exploring human mobility patterns in urban scenarios: A trajectory data perspective
IEEE Commun. Mag.
A federated learning-based license plate recognition scheme for 5G-enabled internet of vehicles
IEEE Trans. Ind. Inf.
Graph learning: A survey
IEEE Trans. Artif. Intell.
Task-driven resource assignment in mobile edge computing exploiting evolutionary computation
IEEE Wirel. Commun.
Ranking station importance with human mobility patterns using subway network datasets
IEEE Trans. Intell. Transp. Syst.
Real-time mask identification for COVID-19: An edge computing-based deep learning framework
IEEE Internet Things J.
Identifying facemask-wearing condition using image super-resolution with classification network to prevent COVID-19
Sensors
Face detection based on YOLOv3
YOLOV3: An incremental improvement
A novel GAN-based network for unmasking of masked face
IEEE Access
Cited by (12)
Real-time fatigue driving detection system based on multi-module fusion
2022, Computers and Graphics (Pergamon)Citation Excerpt :The third method is to judge the fatigue state according to the driver’s facial features [9,16]. This method mainly detects facial information by computer vision detection and processing technology [17,18], obtains the head position, blink frequency and mouth information [19–21], and evaluates the driver’s mental state after comprehensively processing various characteristic information [22]. When the driver is tired, the opening degree of the eyes will become smaller, and even completely close the eyes, the frequency of the eyes will also be lower than that in the normal state.
CDCL-VRE: An ensemble deep learning-based model for distracted driver behavior detection
2024, Journal of Intelligent and Fuzzy SystemsA comprehensive survey on communication techniques for the realization of intelligent transportation systems in IoT based smart cities
2024, Peer-to-Peer Networking and ApplicationsVision Image Monitoring on Transportation Infrastructures: A Lightweight Transfer Learning Approach
2023, IEEE Transactions on Intelligent Transportation Systems
- ☆
This work is partially supported by the National Natural Science Foundation of China (62072409), Zhejiang Provincial Natural Science Foundation, China (LR21F020003), and Fundamental Research Funds for the Provincial Universities of Zhejiang, China (RF-B2020001).