A distributed sensor management for large-scale IoT indoor acoustic surveillance
Introduction
The Pervasive Internet of Things (IoT) is an emerging global infrastructure where embedded smart objects are coupled with pervasive communication to enable detailed characterization of the real physical applications. Recognizing that modern society faces new types of threats; security measures call for a paradigm shift from centralized and investigative solutions to distributed and preventative ones. As such, the complexity and spread of security threats have made it clear that pervasive IoT surveillance is the most effective solution. Such IoT systems comprise a heterogeneous collection of sensor nodes spread over a large area where each sensor has a partial view of the environment, and the network collectively monitors the entire area under surveillance. In such large-scale resource-constrained networks, the use of visual sensors is hurdled by numerous challenges, including; a higher processing complexity, sensitivity to illumination conditions, limitations of the field of view, privacy concerns, bandwidth limitations, finite power supply, and high degree of centralization. However, audio cues in the dynamic scene provide vital information, e.g., explosion, gunshot, or loud scream, that can be captured by microphones. Thus, utilizing the acoustic information from the dynamic scene can provide rich information about the state of the environment with lower processing, storage, and transmission costs. Although acoustic surveillance started getting more attention from the research community in recent years, most state-of-the-art focus only on abnormal event recognition using centralized systems [[1], [2], [3], [4], [5]], thus, ignoring the need for distributed operation to enable large-scale IoT surveillance systems that operates under energy and resource constraints. This work addresses these issues and proposes a large-scale distributed SM for IoT indoor acoustics surveillance that can significantly enhance the process of information gathering by coordinating the sensing resources.
SM is essential in IoT surveillance systems as there is a imperative need to manage and coordinate the sensing resources to collect the most complete and accurate data from a dynamic scene under resource constraints. Achieving efficient distributed SM is challenging due to the spatially distributed nature and the limited resources of sensor networks, specially in indoor environments where situations are characterized by a high density of targets, stochastic environments, and dynamic threats. To-date, distributed large-scale SM is still a challenge [6]. Significant research efforts focused on centralized approaches [[7], [8], [9], [10]]. Although the research community proposed a number of decentralized SM approaches [[11], [12]], limited research effort has been directed in applying of such decentralized SM approaches in pervasive IoT surveillance applications. This paper introduces a novel large-scale SM framework, IntelliSurv, that detects and localizes acoustic abnormalities to address the challenges of IoT indoor acoustic surveillance in a distributed collaborative manner. The proposed framework exhibits capabilities such as collaborative control and SM, heterogeneity, self-organized behavior, autonomous abnormal event detection, and indoor localization. It aims to minimize energy consumption and communication overhead while maintaining high tracking quality and scalability. The key contributions of this paper are summarized as follows:
Designing a scalable pervasive IoT acoustic surveillance system, named IntelliSurv, for indoor environments with integrated abnormal event detection and localization modules.
Formulating a team-theoretic SM for autonomous decision-making and resource management;
Developing a distributed abnormality detection and localization that utilizes acoustic information for identifying critical situations.
The remainder of this paper is organized as follows: Section 2 provides a background overview of surveillance systems and summary of state-of-the-art SM. The system model is presented in Section 3. The proposed IntelliSurv framework is introduced in Section 4, along with the design details of its various system components and the system integration. The performance evaluation is discussed in Section 5. Finally, Section 6 concludes this work.
Section snippets
Acoustic surveillance systems overview
Although, visual sensors are the most adopted modality in smart pervasive surveillance systems, the high processing and communication costs, as well as its subjectivity to illumination, occlusions, and field of view have proven that video surveillance system is an expensive commodity. In addition, concerns about privacy issues heighten with the use of visual surveillance. Not only can audio-based surveillance systems overcome the aforementioned challenges, but also provide vital information
System model
The operation of sensor nodes plays an important role in the effectiveness of the overall system performance. Sensor nodes have four modes of operation, active/sensing, idle/listening, transmitting/receiving, and sleeping. A sensor node must be in one of these four modes at any given time. If there is no limitation on the energy reserve, the sensor network would collect every possible measurement to maximize the situation-awareness. However, the limited energy budget dictates that the sensor
The proposed IntelliSurv SM
The primary contribution of this paper is the design and development of an autonomous SM system for indoor acoustic IoT surveillance, named IntelliSurv. The proposed framework monitors the Volume of Interest (VoI) in an energy-efficient manner through a distributed network of autonomous sensors that uses audio cues to detect anomalies, classifies these anomalies by using trained models, then localizes them, and allocates the needed resources to track such anomalies through the VoI. Fig. 1
Simulation setup
The adopted scenario is the surveillance of the Waterloo International Airport. The airport halls are virtually divided into mesh grid cells using stationary sensors. Each sensor has a sensing range of 3 3 grid cells. The sensors are stationary possessing heterogeneous modalities. Each sensor is equipped with a battery of 100 power units. Initially, the VoI is fully monitored by sensors. The passengers; i.e., targets, enter and leave the airport randomly. Also, impulsive bursts of passengers’
Conclusion and future work
The complexity of emerging threats has stimulated the development of pervasive IoT surveillance systems. A novel sensor management for pervasive IoT surveillance, named IntelliSurv, is proposed in this paper. It automatically detects and localizes abnormal acoustic events in a distributed collaborative manner. The proposed abnormality detection module employs Support Vector Machines (SVM) and Linear Discriminate Analysis (LDA) classifiers to identify and localize human screams or high-stress
Acknowledgment
This work has been supported by Ontario Research Fund-Research Excellence (ORE-RE) program funded by the Government of Ontario, Canada.
Allaa R. Hilal, Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Allaa Hilal received B.S. from Cairo University, M.Sc. the German University in Cairo and a Ph.D. degree in Electrical and Computer Engineering from the University of Waterloo in 2005, 2008 and 2013 respectively. She is currently a Director, Innovation at Intelligent Mechatronic Systems (IMS) and an adjunct assistant professor at the
References (57)
- et al.
Application of acoustic directional data for audio event recognition via HMM/CRF in perimeter surveillance systems
Robot. Auton. Syst.
(2015) - et al.
Approximate stochastic dynamic programming for sensor scheduling to track multiple targets
Digital Signal Process.
(2009) - et al.
Fear-type emotion recognition for future audio-based surveillance systems
Speech Commun.
(2008) - et al.
A survey on sound source localization in robotics: From binaural to array processing methods
Comput. Speech Lang.
(2015) - et al.
Combining visual and acoustic features for audio classification tasks
Pattern Recognit. Lett.
(2017) - et al.
Comparison of multiclass svm classification techniques in an audio surveillance application under mismatched conditions
- et al.
Generalized cross-correlation based noise robust abnormal acoustic event localization utilizing non-negative matrix factorization
- et al.
Evaluation of sound event detection, classification and localization in the presence of background noise for acoustic surveillance of hazardous situations
- et al.
Detection and localization of selected acoustic events in acoustic field for smart surveillance applications
Multimedia Tools Appl.
(2014) - et al.
Pervasive surveillance system management
Sensor scheduling for energy-efficient target tracking in sensor networks
IEEE Trans. Signal Process.
A framework for information-based sensor management for the detection of static targets
IEEE Trans. Syst. Man Cybern. A
A scalable sensor management architecture using BDI model for pervasive surveillance
IEEE Syst. J.
A collaborative energy-aware sensor management system using team theory
ACM Trans. Embed. Comput. Syst.
An adaptive framework for acoustic monitoring of potential hazards
EURASIP J. Audio Speech Music Process.
Using one-class SVMs and wavelets for audio surveillance
IEEE Trans. Inf. Forensics Secur.
Affect recognition in real life scenarios
Audio event detection and localisation for directing video surveillance — a survey of potential techniques
IET Conf. Proc.
Cited by (19)
Future air transportation and digital work at airports – Review and developments
2023, Transportation Research Interdisciplinary PerspectivesSurvey on recent advances in IoT application layer protocols and machine learning scope for research directions
2022, Digital Communications and NetworksCitation Excerpt :The protocols are needed to handle low bandwidth and high latency communication channels when transmitting data between applications and servers [179,180]. In Ref. [181], Hilal et al. proposed IoT acoustic surveillance using Linear Discriminate Analysis (LDA) and a Support Vector Machine (SVM). The primary goal of this approach was to recognize human screams or vocal stress, and it was tested at Waterloo International Airport.
Developing smart devices with automated Machine learning Approach: A review
2021, Materials Today: ProceedingsCitation Excerpt :Applications for detecting failed engine coats, for example, may be used to calculate coating temperature using a thermocouple or IR sensor, microphone noise, piezo sensor and MEMS vibration, torque, or engine drag or some combination. Each can be correlated to wear but will have various response characteristics, signal-to-noise ratios, and other limiting constraints that provide a selective approach over different approaches [24]. In order to achieve the desired results, the Smart Edge AI approach to developing ML algorithm uses training and test data [25].
IRESE: An intelligent rare-event detection system using unsupervised learning on the IoT edge
2019, Engineering Applications of Artificial IntelligenceCitation Excerpt :AD algorithms have been adopted also in IoT contexts, by creating more intelligent, reactive and secure environments. Hilal et al. (2018) present and describe a Sensor Management framework called IntelliSurv. It realize an acoustic surveillance system that follows the pervasive IoT paradigm, being it able to detect and localize anomalous audio events using different kinds of distributed devices: smart sensors for environmental monitoring, and delegate sensors devoted to sensor management, localization and identification of anomalous events.
Acoustic sensor networks in the Internet of Things applications
2018, Future Generation Computer SystemsImproved convolutional neural network and spectrogram image feature for traffic sound event classification
2023, Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering
Allaa R. Hilal, Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Allaa Hilal received B.S. from Cairo University, M.Sc. the German University in Cairo and a Ph.D. degree in Electrical and Computer Engineering from the University of Waterloo in 2005, 2008 and 2013 respectively. She is currently a Director, Innovation at Intelligent Mechatronic Systems (IMS) and an adjunct assistant professor at the University of Waterloo. Allaa has been recognized and awarded numerous prestigious scholarships, most prominent of which are NSERC IRDF, NSERC CGS, Fulbright, DAAD, and OGS. She was also recognized by the Google Anita Borg Memorial Scholarship.
Aya Sayedelahl, Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Aya Sayedelahl received B.S. and MSc. From Mansoura University in Egypt, and a Ph.D. degree in Electrical and Computer Engineering from the United States. After her studies, Aya joined as a post-doctoral researcher at Centre for Pattern Analysis and Machine Intelligence in the University of Waterloo.
Arash Tabibiazar, Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Arash Tabibiazar received the B.Sc. and M.Sc. degrees in computer engineering from Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran, in 1995 and 1999, respectively. He finished his Ph.D. degree in communication and information systems in the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada.
Mohamed S. Kamel, Center for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Mohamed S. Kamel (S’74M’80SM’95F’05) received the B.Sc. (Hons) EE (Alexandria University), M.A.Sc. (McMaster University), and Ph.D. (University of Toronto). He joined the University of Waterloo, Canada, in 1985, where he is a Professor and Director of the Center for Pattern Analysis and Machine Intelligence at the Department of Electrical and Computer Engineering. Professor He currently holds the position of University Research Chair. His research interests are in computational intelligence, pattern recognition, machine learning and cooperative intelligent systems. He has authored and coauthored over 500 papers in journals and conference proceedings, 13 edited volumes, 16 chapters in edited books, four patents, and numerous technical and industrial project reports. Under his supervision, 88 Ph.D. and M.A.Sc. students have completed their degrees.
Dr. Kamel is member of ACM, PEO, Fellow of the Engineering Institute of Canada (EIC), Fellow of the Canadian Academy of Engineering (CAE), and Fellow of the International Association of Pattern Recognition (IAPR).
Otman A. Basir, Center for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.
Otman A. Basir received the M.Sc. degree in electrical engineering from Queen’s University, Kingston, Canada and the Ph.D. degree in systems design engineering from University of Waterloo, Waterloo, Canada, in 1989 and 1993, respectively.
He is currently a Professor with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada.