Skip to main content
SpringerLink
Log in

Implementation of Infrastructure for Streaming Outlier Detection in Big Data

  • Conference paper
  • First Online:
Recent Advances in Information Systems and Technologies (WorldCIST 2017)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 570))

Included in the following conference series:

Abstract

In recent years the amount of data available at World Wide Web grew drastically. Therefore the need to analyze these large amount of data is a problem faced by researchers in this field. The problem that occurs is analyzing this data in real time before they are stored. Analyzing network log data in order to find system anomalies is another problem that researchers face. There are many tools that are used in order to detect anomalies in real time big data but in this research Fluentd is used. Submission of questionnaires in such large quantities of data is a way to take our decisions in various businesses and organizations. Another challenge is how to monitor the data generated in the network and to detect errors in order to scale up performance. To enable the infrastructure to detect anomalies in streaming data outlier detection plugin for Fluentd is implemented. Another important thing is how to visualize the result in order to have a legible result. In this paper we show the visualization of Syslog by using Kibana, and also how the Fluentd plugin helps us to identify anomalies in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://ednevnik.edu.mk/.

References

  1. Aggarwal, C.C.: Outlier Analysis. Springer, New York (2013)

    Google Scholar 

  2. Hasani, Z., Kon-Popovska, M., Velinov, G.: Survey of technologies for real time big data streams analytic. In: 11th International Conference on Informatics and Information Technologies, 11–13 April 2014, Bitola, Macedonia (2014)

    Google Scholar 

  3. Hasani, Z., Kon-Popovska, M., Velinov, G.: Lambda architecture for real time big data analytic. In: ICT Innovations 2014 Web Proceedings. ISSN 1857-7288 (2014)

    Google Scholar 

  4. Hasani, Z.: Performance comparison throw running job in Hadoop by defining the number of maps and reduces. In: 12th International Conference on Informatics and Information Technologies 2015, 24–26 April 2015, Bitola, Macedonia (2015)

    Google Scholar 

  5. Hasani, Z.: Virtuoso, system for saving semantic data. In: 12th International Conference on Informatics and Information Technologies 2015. 24–26 April 2015, Bitola, Macedonia (2015)

    Google Scholar 

  6. Tamura, K.: Elasticsearch, Fluentd, and Kibana: Open Source Log Search and Visualization. https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization. Accessed 8 Oct 2015

  7. GitHub: fluent-plugin-anomalydetect. https://github.com/muddydixon/fluent-plugin-anomalydetect. Accessed 8 Oct 2015

  8. Fluentd. http://www.fluentd.org/architecture. Accessed 8 Oct 2015

  9. Apache Lucena. https://lucene.apache.org/. Accessed 30 April 2015

  10. Hasani, Z., Jakimovski, B., Kon-Popovska, M., Velinov, G.: Real time analytic of SQL queries based on log analytic. In: ICT Innovations 2015 Web Proceedings (2015). ISSN 1857-7288. http://proceedings.ictinnovations.org/attachment/conference/12/ict-innovations-2015-web-proceedings.pdf

  11. Tamura, K.: Elasticsearch, Fluentd, and Kibana: Open Source Log Search and Visualization. https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization. Accessed 7 Jan 2016

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University, Skopje, Macedonia

    Zirije Hasani

Authors
  1. Zirije Hasani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zirije Hasani .

Editor information

Editors and Affiliations

  1. DEI/FCT, Universidade de Coimbra, Coimbra, Baixo Mondego, Portugal

    Álvaro Rocha

  2. Nova IMS, Universidade Nova de Lisboa, Lisboa, Portugal

    Ana Maria Correia

  3. College of Engineering, The Ohio State University, Columbus, Ohio, USA

    Hojjat Adeli

  4. DSI/EEUM, Universidade do Minho, Guimarães, Portugal

    Luís Paulo Reis

  5. DIMES, Università della Calabria, Arcavacata di Rende, Italy

    Sandra Costanzo

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Hasani, Z. (2017). Implementation of Infrastructure for Streaming Outlier Detection in Big Data. In: Rocha, Á., Correia, A., Adeli, H., Reis, L., Costanzo, S. (eds) Recent Advances in Information Systems and Technologies. WorldCIST 2017. Advances in Intelligent Systems and Computing, vol 570. Springer, Cham. https://doi.org/10.1007/978-3-319-56538-5_51

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56538-5_51

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56537-8

  • Online ISBN: 978-3-319-56538-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation