Skip to main content
Springer Nature Link
Log in

Enhancing Observability: Real-Time Application Health Checks

  • Conference paper
  • First Online:
Enterprise Design, Operations, and Computing. EDOC 2024 Workshops (EDOC 2024)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 537))

Abstract

Log management and application health monitoring practices are cumbersome and often still require significant human intervention to prevent inaccurate data and information. Existing technologies like Elasticsearch and Grafana offer opportunities to automate and improve these practices. This paper reports on a design solution aimed at enhancing log categorization, anomaly detection, and real-time application health reporting for CAPE Groep’s service application. The proposed solution leverages Elasticsearch’s Machine Learning capabilities and Grafana’s dynamic visualization tools, alongside a newly developed dashboard named Horus, to centralize log data and automate monitoring processes. Preliminary results indicate that the proposed solution significantly improves the accuracy and timeliness of health reports, reduces manual intervention, and provides comprehensive real-time insights into application performance. This paper outlines the requirements, architectural design, and phased implementation plan, demonstrating the potential to streamline operations, enhance service delivery, and support future more stringent scalability requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://capegroep.nl/.

  2. 2.

    https://www.mendix.com/.

  3. 3.

    https://emagiz.com/.

  4. 4.

    Out-of-vocabulary (OOV) words refer to words that appear in a text but were not included in the training set vocabulary when a language model or a word embedding system, like Word2Vec, was initially trained.

References

  1. Ali, S., Boufaied, C., Bianculli, D., Branco, P., Briand, L.C., Aschbacher, N.: An empirical study on log-based anomaly detection using machine learning. ArXiv abs/2307.16714 (2023). https://api.semanticscholar.org/CorpusID:260334470

  2. Bertero, C., Roy, M., Sauvanaud, C., Tredan, G.: Experience report: log mining using natural language processing and application to anomaly detection. In: 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), pp. 351–360 (2017). https://doi.org/10.1109/ISSRE.2017.43

  3. Białecki, A., Muir, R., Ingersoll, G.: Apache Lucene 4 (2012)

    Google Scholar 

  4. Cândido, J., Aniche, M., van Deursen, A.: Log-based software monitoring: a systematic mapping study. PeerJ Comput. Sci. 7, e489 (2021). https://doi.org/10.7717/peerj-cs.489

    Article  Google Scholar 

  5. CAPE Groep: Mendix Marketplace - LogTransporter (2024). https://marketplace.mendix.com/link/component/218262

  6. Courcy, D.: Elastic 7.16: Streamlined data integrations drive results that matter (2021). https://www.elastic.co/blog/whats-new-elastic-7-16-0

  7. Du, S., Cao, J.: Behavioral anomaly detection approach based on log monitoring. In: 2015 International Conference on Behavioral, Economic and Socio-cultural Computing (BESC), pp. 188–194 (2015). https://doi.org/10.1109/BESC.2015.7365981

  8. Elastic: Anomaly detection job types. https://www.elastic.co/guide/en/machine-learning/current/ml-anomaly-detection-job-types.html

  9. Elastic: Categorize text aggregation. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-categorize-text-aggregation.html

  10. Elastic: Elastic Common Schema. https://www.elastic.co/elasticsearch/common-schema

  11. Folmer, E., Verhoosel, J.: State of the art on semantic is standardization, interoperability & quality. J. Biomech. (2011)

    Google Scholar 

  12. Gormley, C., Tong, Z.: Elasticsearch the Definitive Guide: A Distributed Real-Time Search and Analytics Engine. O’Reilly Media, 1 edn. (2015)

    Google Scholar 

  13. Grafana Labs: About Grafana. https://grafana.com/docs/grafana/latest/introduction/

  14. He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE International Conference on Web Services (ICWS), pp. 33–40 (2017). https://doi.org/10.1109/ICWS.2017.13

  15. Kozhukh, D.: An easy look at Grafana architecture (2024). https://www.kozhuhds.com/blog/an-easy-look-at-grafana-architecture/

  16. Kumar, G., Basri, S., Imam, A.A., Khowaja, S.A., Capretz, L.F., Balogun, A.O.: Data harmonization for heterogeneous datasets: a systematic literature review. Appl. Sci. 11(17), 8275 (2021). https://doi.org/10.3390/app11178275

    Article  Google Scholar 

  17. Layer, L., et al.: Automatic log analysis with NLP for the CMS workflow handling. In: EPJ Web of Conferences, vol. 245, p. 03006 (2020). https://doi.org/10.1051/epjconf/202024503006

  18. Le, V., Zhang, H.: Log-based anomaly detection without log parsing. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 492–504 (2021). https://doi.org/10.1109/ASE51524.2021.9678773

  19. Madkan, P.: Empirical study of ERP implementation strategies-filling gaps between the success and failure of ERP implementation process. Int. J. Inf. Comput. Technol. 4(6), 633–642 (2014)

    MATH  Google Scholar 

  20. Mendix: Deploy API (2024). https://docs.mendix.com/apidocs-mxsdk/apidocs/deploy-api/

  21. Meng, W., et al.: A semantic-aware representation framework for online log analysis, pp. 1–7 (2020). https://doi.org/10.1109/ICCCN49398.2020.9209707

  22. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26 (2013)

    Google Scholar 

  23. Mitra, M., Sy, D.: The rise of elastic stack (2016). https://doi.org/10.13140/RG.2.2.17596.03203

  24. Münch, J., Armbrust, O., Kowalczyk, M., Soto, M.: Prescriptive Process Models. In: Münch, J., Armbrust, O., Kowalczyk, M., Soto, M. (eds.) Software Process Definition and Management, pp. 19–77. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24291-5_2

  25. Plotly: Plotly JavaScript Open Source Graphing Library. https://plotly.com/javascript/

  26. Srivastava, D.: An introduction to data visualization tools and techniques in various domains. Int. J. Comput. Trends Technol. 71, 125–130 (2023). https://doi.org/10.14445/22312803/IJCTT-V71I4P116

  27. Trent, B.: Categorize your logs with Elasticsearch categorize_text aggregation (2022). https://www.elastic.co/blog/categorize-your-logs-with-the-new-elasticsearch-categorize-text-search-aggregation

  28. Wang, J., et al.: LogEvent2vec: LogEvent-to-vector based anomaly detection for large-scale logs in internet of things. Sens. (Switz.) 20(9) (2020). https://doi.org/10.3390/s20092451

  29. Wang, J., Zhao, C., He, S., Gu, Y., Alfarraj, O., Abugabah, A.: LogUAD: log unsupervised anomaly detection based on word2Vec. Comput. Syst. Sci. Eng. 41(3), 1207–1222 (2022). https://doi.org/10.32604/csse.2022.022365

  30. Wei, Y., Li, M., Xu, B.: Research on Establish an Efficient Log Analysis System with Kafka and Elastic Search. J. Softw. Eng. Appl. 10(11), 843–853 (2017). https://doi.org/10.4236/jsea.2017.1011047

    Article  MATH  Google Scholar 

  31. Yu, B., et al.: Deep learning or classical machine learning? An empirical study on log-based anomaly detection. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE 2024, Association for Computing Machinery, New York (2024). https://doi.org/10.1145/3597503.3623308

  32. Zamfir, V.A., Carabas, M., Carabas, C., Tapus, N.: Systems monitoring and big data analysis using the elastic search system. In: Proceedings - 2019 22nd International Conference on Control Systems and Computer Science, CSCS 2019, pp. 188–193. Institute of Electrical and Electronics Engineers Inc. (2019). https://doi.org/10.1109/CSCS.2019.00039

  33. Zhao, K., Xia, M.: Forming interoperability through interorganizational systems standards. J. Manag. Inf. Syst. 30(4), 269–298 (2014). https://doi.org/10.2753/MIS0742-1222300410

    Article  MATH  Google Scholar 

  34. Zhou, J., Qian, Y., Zou, Q., Liu, P., Xiang, J.: DeepSyslog: deep anomaly detection on syslog using sentence embedding and metadata. IEEE Trans. Inf. Forensics Secur. (2022). https://doi.org/10.1109/TIFS.2022.3201379

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Twente, Drienerlolaan 5, 7522 NB, Enschede, The Netherlands

    Tim Eichhorn, João Luiz Rebelo Moreira, Luís Ferreira Pires & Lucas Meertens

  2. CAPE Groep, Transportcentrum 16, 7547 RW, Enschede, The Netherlands

    Lucas Meertens

Authors
  1. Tim Eichhorn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. João Luiz Rebelo Moreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luís Ferreira Pires
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lucas Meertens
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Eichhorn .

Editor information

Editors and Affiliations

  1. University of Duisburg-Essen, Essen, Germany

    Monika Kaczmarek-Heß

  2. Niederrhein University of Applied Sciences, Mönchengladbach, Germany

    Kristina Rosenthal

  3. Czech Technical University in Prague, Prague, Czech Republic

    Marek Suchánek

  4. Universidade de Lisboa, Lisbon, Portugal

    Miguel Mira Da Silva

  5. TU Wien, Vienna, Austria

    Henderik A. Proper

  6. TU Wien, Vienna, Austria

    Marianne Schnellmann

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Eichhorn, T., Moreira, J.L.R., Pires, L.F., Meertens, L. (2025). Enhancing Observability: Real-Time Application Health Checks. In: Kaczmarek-Heß, M., Rosenthal, K., Suchánek, M., Da Silva, M.M., Proper, H.A., Schnellmann, M. (eds) Enterprise Design, Operations, and Computing. EDOC 2024 Workshops . EDOC 2024. Lecture Notes in Business Information Processing, vol 537. Springer, Cham. https://doi.org/10.1007/978-3-031-79059-1_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-79059-1_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-79058-4

  • Online ISBN: 978-3-031-79059-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics