Skip to main content
SpringerLink
Log in

Blockchain-driven anomaly detection framework on edge intelligence

  • Regular Paper
  • Published:
CCF Transactions on Networking

Abstract

There are a large number of end devices in an IoT system, which may malfunction due to various reasons, such as being attacked. Anomaly detection of the devices and the whole IoT system normally rely on the analysis of the huge amount of log records generated by the end devices. How to protect the log records from being tampered with and realize the real-time anomaly detection is a challenging task which is still not addressed. Existing works on anomaly detection by the emerging and effective deep learning algorithms require the transfer of log data to cloud servers which incurs high communication overhead and long detection latency, and is subject to the risk of being tampered. In this paper, we propose a novel and efficient hierarchical framework for online anomaly detection in IoT systems atop Blockchain and smart contracts. At the device layer of the hierarchical framework, an efficient feature extractor is developed to preprocess the raw log data which greatly reduces the size of data to be transferred while keeps sufficient information for the anomaly detection model to use. At the cloud layer of the framework, deep learning models use the processed data from the device layer to build the detection model and output normal workflow patterns. In the edge layer of the framework, a permissioned blockchain is built and a series of smart contracts are developed which can guarantee data integrity and achieve automatic anomaly detection based on the model output from the cloud layer. Extensive experiments demonstrate that our framework can reduce the ledger size by 7.1% without detection accuracy reduction compared with traditional centralized solutions and the detection latency is only 0.47ms in our prototype. Our feature extractor can speed up by 3.6x–7.3x times on the execution time with almost the same CPU usage rate compared with state-of-the-art log parsers and encryption solutions, such as AES and RSA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Belchior, R., Correia, M., Vasconcelos, A.: Justicechain: Using blockchain to protect justice logs. In: OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”, pp. 318–325. Springer (2019)

  • Du, M., Li, F.: Atom: efficient tracking, monitoring, and orchestration of cloud resources. IEEE Trans. Parallel Distrib. Syst. 28(8), 2172–2189 (2017)

    Article  Google Scholar 

  • Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298. ACM (2017)

  • Fu, Q., Lou, J.G., Wang, Y., Li, J.: Execution anomaly detection in distributed systems through unstructured log analysis. In: Data Mining, 2009. ICDM’09. Ninth IEEE International Conference on, pp. 149–158. IEEE (2009)

  • Hamooni, H., Debnath, B., Xu, J., Zhang, H., Jiang, G., Mueen, A.: Logmine: fast pattern recognition for log analytics. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1573–1582. ACM (2016)

  • He, P., Zhu, J., He, S., Li, J., Lyu, M.R.: Towards automated log parsing for large-scale log data analysis. IEEE Trans. Dependable Secure Comput. 15(6), 931–944 (2018)

    Article  Google Scholar 

  • He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: An online log parsing approach with fixed depth tree. In: Web Services (ICWS), 2017 IEEE International Conference on, pp. 33–40. IEEE (2017)

  • He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: system log analysis for anomaly detection. In: Software Reliability Engineering (ISSRE), 2016 IEEE 27th International Symposium on, pp. 207–218. IEEE (2016)

  • Huang, W.: A blockchain-based framework for secure log storage. In: 2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology (CCET), pp. 96–100. IEEE (2019)

  • Jiang, Z.M., Hassan, A.E., Hamann, G., Flora, P.: An automated approach for abstracting execution logs to execution events. J. Softw. Mainten. Evolut. Res. Pract. 20(4), 249–267 (2008)

    Article  Google Scholar 

  • Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., Tang, L.: Neurosurgeon: collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Comput. Architect. News 45(1), 615–629 (2017)

    Article  Google Scholar 

  • Liu, J., Ren, J., Dai, W., Zhang, D., Zhou, P., Zhang, Y., Min, G., Najjari, N.: Online multi-workflow scheduling under uncertain task execution time in iaas clouds. IEEE Transactions on Cloud Computing (2019)

  • Lou, J.G., Qiang, F., Yang, S., Jiang, L., Wu, B.: Mining program workflow from interleaved traces. In: ACM Sigkdd International Conference on Knowledge Discovery & Data Mining (2010)

  • Lyu, F., Ren, J., Cheng, N., Yang, P., Li, M., Zhang, Y., Shen, X.: Lead: large-scale edge cache deployment based on spatio-temporal wifi traffic statistics. IEEE Trans. Mob. Comput. (2020)

  • Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: A lightweight algorithm for message type extraction in system application logs. IEEE Trans. Knowl. Data Eng. 24(11), 1921–1936 (2012)

    Article  Google Scholar 

  • Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., Sasnauskas, R.: A search-based approach for accurate identification of log message formats. In: Proceedings of the 26th IEEE/ACM International Conference on Program Comprehension (ICPC18). ACM (2018)

  • Min, D., Li, F.: Spell: Streaming parsing of system event logs. In: IEEE International Conference on Data Mining (2017)

  • Mizutani, M.: Incremental mining of system log format. In: Services Computing (SCC), 2013 IEEE International Conference on, pp. 595–602. IEEE (2013)

  • Nagappan, M., Vouk, M.A.: Abstracting log lines to log event types for mining software system logs. In: Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on, pp. 114–117. IEEE (2010)

  • Osia, S.A., Shamsabadi, A.S., Sajadmanesh, S., Taheri, A., Katevas, K., Rabiee, H.R., Lane, N.D., Haddadi, H.: A hybrid deep learning architecture for privacy-preserving mobile analytics. IEEE Internet of Things Journal (2020)

  • Osia, S.A., Taheri, A., Shamsabadi, A.S., Katevas, K., Haddadi, H., Rabiee, H.R.: Deep private-feature extraction. IEEE Trans. Knowl. Data Eng. 32(1), 54–66 (2018)

    Article  Google Scholar 

  • Pourmajidi, W.: Scalable blockchain-assisted log storage system for cloud-generated logs (2018)

  • Pourmajidi, W., Miranskyy, A.: Logchain: Blockchain-assisted log storage. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 978–982. IEEE (2018)

  • Pourmajidi, W., Zhang, L., Steinbacher, J., Erwin, T., Miranskyy, A.: Immutable log storage as a service. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 280–281. IEEE (2019)

  • Rane, S., Dixit, A.: Blockslaas: Blockchain assisted secure logging-as-a-service for cloud forensics. In: International Conference on Security& Privacy, pp. 77–88. Springer (2019)

  • Ren, J., Zhang, D., He, S., Zhang, Y., Li, T.: A survey on end-edge-cloud orchestrated network computing paradigms: transparent computing, mobile edge computing, fog computing, and cloudlet. ACM Comput. Surv. (CSUR) 52(6), 1–36 (2019)

    Article  Google Scholar 

  • Shao, W., Wang, Z., Wang, X., Qiu, K., Jia, C., Jiang, C.: Lsc: online auto-update smart contracts for fortifying blockchain-based log systems. Inf. Sci. 512, 506–517 (2020)

    Article  Google Scholar 

  • Shima, K.: Length matters: clustering system log messages using length of words. arXiv:1611.03213 (2016)

  • Tang, L., Li, T., Perng, C.S.: Logsig: Generating system events from raw textual logs. In: Proceedings of the 20th ACM international conference on Information and knowledge management, pp. 785–794. ACM (2011)

  • Tang, W., Ren, J., Zhang, K., Zhang, D., Zhang, Y., Shen, X.: Efficient and privacy-preserving fog-assisted health data sharing scheme. ACM TIST 10(6), 1–23 (2019)

    Article  Google Scholar 

  • Thomas, A., Guo, Y., Kim, Y., Aksanli, B., Kumar, A., Rosing, T.S.: Hierarchical and distributed machine learning inference beyond the edge. In: 2019 IEEE 16th International Conference on Networking, Sensing and Control (ICNSC), pp. 18–23. IEEE (2019)

  • Tsankov, P., Dan, A., Drachsler-Cohen, D., Gervais, A., Buenzli, F., Vechev, M.: Securify: Practical security analysis of smart contracts. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 67–82 (2018)

  • Vaarandi, R.: A data clustering algorithm for mining patterns from event logs. In: IP Operations & Management, 2003.(IPOM 2003). 3rd IEEE Workshop on, pp. 119–126. IEEE (2003)

  • Vaarandi, R., Pihelgas, M.: Logcluster-a data clustering and pattern mining algorithm for event logs. In: Network and Service Management (CNSM), 2015 11th International Conference on, pp. 1–7. IEEE (2015)

  • Wang, H., Yang, D., Duan, N., Guo, Y., Zhang, L.: Medusa: Blockchain powered log storage system. In: 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), pp. 518–521. IEEE (2018)

  • Xiao, Y., Joshi, P., Xu, J., Jin, G., Hui, Z., Jiang, G.: Cloudseer: workflow monitoring of cloud infrastructures via interleaved logs. ACM Sigarch Comput. Architect. News 44(2), 489–502 (2016)

    Article  Google Scholar 

  • Xie, X., Jin, Z., Han, Q., Huang, S., Li, T.: A confidence-guided anomaly detection approach jointly using multiple machine learning algorithms. In: International symposium on cyberspace safety and security, pp. 93–100. Springer (2019)

  • Xie, X., Jin, Z., Wang, J., Yang, L., Lu, Y., Li, T.: Confidence guided anomaly detection model for anti-concept drift in dynamic logs. Journal of Network and Computer Applications, pp. 102659 (2020)

  • Xie, X., Wang, Z., Xiao, X., Lu, Y., Huang, S., Li, T.: A confidence-guided evaluation for log parsers inner quality. Mobile Networks and Applications, pp. 1–12 (2020)

  • Xu, S., Qian, Y., Hu, R.Q.: Data-driven network intelligence for anomaly detection. IEEE Netw. 33(3), 88–95 (2019)

    Article  Google Scholar 

  • Xu, X., Pautasso, C., Zhu, L., Lu, Q., Weber, I.: A pattern collection for blockchain-based applications. In: Proceedings of the 23rd European Conference on Pattern Languages of Programs, pp. 1–20 (2018)

  • Yin, H., Wang, Z., Jha, N.K.: A hierarchical inference model for internet-of-things. IEEE Trans. Multi-Scale Comput. Syst. 4(3), 260–271 (2018)

    Article  Google Scholar 

  • Zhang, L., Xie, X., Xie, K., Wang, Z., Lu, Y., Zhang, Y.: An efficient log parsing algorithm based on heuristic rules. In: International Symposium on Advanced Parallel Processing Technologies, pp. 123–134. Springer (2019)

  • Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., Zhang, J.: Edge intelligence: paving the last mile of artificial intelligence with edge computing. Proc. IEEE 107(8), 1738–1762 (2019)

    Article  Google Scholar 

  • Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., Lyu, M.R.: Tools and benchmarks for automated log parsing. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 121–130. IEEE (2019)

Download references

Acknowledgements

This work is partially supported by the National Key Research and Development Program of China (2018YFB2100300), the National Natural Science Foundation (61872200), the People’s Republic of China ministry of education science and technology development center (2019J02019), the CERNET Innovation Project (NGII20180306, NGII20190402) and the Natural Science Foundation of Tianjin (19JCZDJC31600, 19JCQNJC00600).

Author information

Authors and Affiliations

  1. College of Computer Science, Nankai University, Tianjin, 300350, China

    Xueshuo Xie, Zhaolong Jian & Ye Lu

  2. Tianjin Key Laboratory of Network and Data Security Technology, Tianjin, 300350, China

    Yaozheng Fang & Tao Li

  3. New Jersey Institute of Technology, Newark, NJ, 07102, USA

    Guiling Wang

Authors
  1. Xueshuo Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaozheng Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhaolong Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ye Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guiling Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ye Lu or Tao Li.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, X., Fang, Y., Jian, Z. et al. Blockchain-driven anomaly detection framework on edge intelligence. CCF Trans. Netw. 3, 171–192 (2020). https://doi.org/10.1007/s42045-020-00044-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42045-020-00044-9

Keywords

Search

Navigation