Skip to main content
SpringerLink
Log in

Cooperative abnormal sound event detection in end-edge-cloud orchestrated systems

  • Regular Paper
  • Published:
CCF Transactions on Networking

Abstract

In this paper, we propose a novel cooperative abnormal sound event detection framework for city surveillance in end-edge-cloud orchestrated systems. A novel offloading decision-making scheme that leverages hierarchical computational capabilities is proposed to speed up the detection process. The audio pre-processing (feature extraction) and post-processing (sound source localization and sound event classification) can be locally executed or offloaded to the edge or cloud based on the calculation of the so-called communication-to-computation ratio. Furthermore, considering the biased audio information due to source-sensor geometries, a cooperative decision-making algorithm is proposed to aggregate the sound event classification results with adaptive control and ensemble learning. In the audio pre-processing, the log-mel spectrogram and time of arrival information are first extracted from the audio waveform captured by the distributed acoustic sensors and then sent to the computation entity assigned by the offloading scheme. In the audio post-processing, the sound source is localized through least-square minimization. Guided by the localized sound source, the spectrograms are fed into the pre-trained neural networks and then the result aggregation algorithm for further classification. Extensive experiments regarding latency and detection accuracy show the superiority and robustness of the proposed scheme, avoiding the cumulative latency caused by the increased number of sensors while maintaining high detection accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Other sound source localization and sound event classification techniques are also applicable in the proposed framework.

  2. The AudioSet ontology can be found at https://research.google.com/audioset/ontology.

References

  • Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., Vijayanarasimhan, S.: Youtube-8M: a large-scale video classification benchmark.arXiv:1609.08675 (2016)

  • Adavanne, S., Politis, A., Nikunen, J., Virtanen, T.: Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J. Sel. Top. Signal Process. 13(1), 34–48 (2018)

    Article  Google Scholar 

  • AudioSet download. https://research.google.com/audioset/download.html (2019)

  • Baucas, M.J., Spachos, P.: Using cloud and fog computing for large scale iot-based urban sound classification. Simul. Modell. Practice Theory 101, 102013 (2020)

    Article  Google Scholar 

  • Bauer, S., Clark, D.D., Lehr, W.: Understanding broadband speed measurements. In: Proc. of MIT TPRC (2010)

  • Cerutti, G., Prasad, R., Brutti, A., Farella, E.: Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms. IEEE J. Sel. Top. Signal Process. (2020)

  • Cobos, M., Antonacci, F., Alexandridis, A., Mouchtaris, A., Lee, B.: A survey of sound source localization methods in wireless acoustic sensor networks. Wirel. Commun, Mobile Comput (2017)

  • Embleton, T.F.: Tutorial on sound propagation outdoors. J. Acoust. Soc. Am. 100(1), 31–48 (1996)

    Article  Google Scholar 

  • Fabregat, G., Belloch, J.A., Badía, J.M., Cobos, M.: Design and implementation of acoustic source localization on a low-cost IoT edge platform. Express Briefs IEEE Trans. Circ. Syst. II, (2020)

  • Goldsmith, A.: Wireless Communications. Cambridge University Press, Cambridge (2005)

    Book  Google Scholar 

  • Guo, Y., Zou, B., Ren, J., Liu, Q., Zhang, D., Zhang, Y.: Distributed and efficient object detection via interactions among devices, edge, and cloud. IEEE Trans. Multimedia 21(11), 2903–2915 (2019)

    Article  Google Scholar 

  • He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al.: Streaming end-to-end speech recognition for mobile devices. In: Proc. of IEEE ICASSP, pp. 6381–6385 (2019)

  • Heittola, T., Mesaros, A., Eronen, A., Virtanen, T.: Context-dependent sound event detection. EURASIP J. Audio Speech Music Process. 1, 1–13 (2013)

    Article  Google Scholar 

  • Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., Slaney, M., Weiss, R.J., Wilson, K.: CNN architectures for large-scale audio classification. In: Proc. of IEEE ICASSP, pp. 131–135 (2017)

  • Hong, K., Lillethun, D., Ramachandran, U., Ottenwälder, B., Koldehofe, B.: Mobile fog: a programming model for large-scale applications on the internet of things. In: Proc. of ACM SIGCOMM workshop on Mobile cloud computing, pp. 15–20 (2013)

  • Huang, Y., Zhu, Y., Fan, X., Ma, X., Wang, F., Liu, J., Wang, Z., Cui, Y.: Task scheduling with optimized transmission time in collaborative cloud-edge learning. In: Proc. of ICCCN, pp. 1–9 (2018)

  • Junior, W., França, A., Dias, K., de Souza, J.N.: Supporting mobility-aware computational offloading in mobile cloud environment. J. Netw. Comput. Appl. 94, 93–108 (2017)

    Article  Google Scholar 

  • Kong, Q., Xu, Y., Wang, W., Plumbley, M.D.: Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 2450–2460 (2020)

    Article  Google Scholar 

  • Li, Y., Liu, M., Drossos, K., Virtanen, T.: Sound event detection via dilated convolutional recurrent neural networks. In: Proc. of IEEE ICASSP, pp. 286–290 (2020)

  • Liu, K., Peng, J., Li, H., Zhang, X., Liu, W.: Multi-device task offloading with time-constraints for energy efficiency in mobile cloud computing. Future Gener. Comput. Syst. 64, 1–14 (2016)

    Article  Google Scholar 

  • Nguyen, T.N.T., Jones, D.L., Gan, W.-S.: A sequence matching network for polyphonic sound event localization and detection. In: Proc. of IEEE ICASSP, pp. 71–75 (2020)

  • Nirjon, S., Dickerson, R.F., Asare, P., Li, Q., Hong, D., Stankovic, J.A., Hu, P., Shen, G., Jiang, X.: Auditeur: a mobile-cloud service platform for acoustic event detection on smartphones. In: Proc. of MobiSys, pp. 403–416 (2013)

  • Niu, C., Li, Y., Hu, R.Q., Ye, F.: Fast and efficient radio resource allocation in dynamic ultra-dense heterogeneous networks. IEEE Access 5, 1911–1924 (2017)

    Google Scholar 

  • Noble, B.D., Satyanarayanan, M., Narayanan, D., Tilton, J.E., Flinn, J., Walker, K.R.: Agile application-aware adaptation for mobility. Proc. ACM SIGOPS 31(5), 276–287 (1997)

    Google Scholar 

  • Noghabi, S. A., Kolb, J., Bodik, P., Cuervo, E.: Steel: simplified development and deployment of edge-cloud applications. In: Proc. of USENIX HotCloud (2018)

  • Phan, H., Maaß, M., Mazur, R., Mertins, A.: Random regression forests for acoustic event detection and classification. IEEE/ACM Trans. Audio Speech Lang. Process. 23(1), 20–31 (2014)

    Article  Google Scholar 

  • Ran, X., Chen, H., Zhu, X., Liu, Z., Chen, J.: DeepDecision: a mobile deep learning framework for edge video analytics. In: Proc. of IEEE INFOCOM, pp. 1421–1429 (2018)

  • Ren, J., Zhang, D., He, S., Zhang, Y., Li, T.: A survey on end-edge-cloud orchestrated network computing paradigms: transparent computing, mobile edge computing, fog computing, and cloudlet. ACM Comput. Surv. 52(6), 1–36 (2019)

    Article  Google Scholar 

  • Shotspotter. http://www.shotspotter.com/technology (2019)

  • Sun, C., Li, H., Li, X., We, J., Xiong, Q., Wang, X., Leung, V.C.: Task offloading for end-edge-cloud orchestrated computing in mobile networks. In: Proc. of IEEE WCNC, pp. 1–6 (2020)

  • Tang, L., He, S.: Multi-user computation offloading in mobile edge computing: a behavioral perspective. IEEE Netw. 32(1), 48–53 (2018)

    Article  Google Scholar 

  • Tong, F., Pan, J.: Random-to-random nodal distance distributions in finite wireless networks. IEEE Trans. Vehic. Technol. 66(11), 10070–10083 (2017)

    Article  Google Scholar 

  • Trowitzsch, I., Schymura, C., Kolossa, D., Obermayer, K.: Joining sound event detection and localization through spatial segregation. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 487–502 (2019)

    Article  Google Scholar 

  • Wang, J., Liu, K., Tzanetakis, G., Pan, J.: Learning-based cooperative sound event detection with edge computing. In: Proc. of IEEE IPCCC, pp. 1–8 (2018)

  • Wang, J., Ai, F., Sun, Q., Liu, T., Li, H., Yan, Z., Liu, D.: Diaphragm-based optical fiber sensor array for multipoint acoustic detection. Opt. Express 26(19), 25293–25304 (2018)

    Article  Google Scholar 

  • Yang, B., Cao, X., Bassey, J., Li, X., Qian, L.: Computation offloading in multi-access edge computing: a multi-task learning approach. IEEE Trans, Mobile Comput (2020)

    Google Scholar 

  • Zhang, Y., Lan, X., Ren, J., Cai, L.: Efficient computing resource sharing for mobile edge-cloud computing networks. IEEE/ACM Trans. Netw. 28(3), 1227–1240 (2020)

    Article  Google Scholar 

  • Zhu, W., Luo, C., Wang, J., Li, S.: Multimedia cloud computing. IEEE Signal Process. Mag. 28(3), 59–69 (2011)

    Article  Google Scholar 

Download references

Funding

This work was supported in part by NSERC, CFI, BCKDF, and China Postdoctoral Science Foundation.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Victoria, Victoria, BC, V8W 2Y2, Canada

    Jingrong Wang, Kaiyang Liu, George Tzanetakis & Jianping Pan

  2. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON, M5S 3G4, Canada

    Jingrong Wang

  3. School of Computer Science and Engineering, Central South University, Changsha, 410075, China

    Kaiyang Liu

Authors
  1. Jingrong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kaiyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. George Tzanetakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianping Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingrong Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Liu, K., Tzanetakis, G. et al. Cooperative abnormal sound event detection in end-edge-cloud orchestrated systems. CCF Trans. Netw. 3, 158–170 (2020). https://doi.org/10.1007/s42045-020-00042-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42045-020-00042-x

Keywords

Search

Navigation