research-article

A big data framework for cloud monitoring

Authors:

Marios Fokaefs,

Hamzeh Khazaei,

Xi ZhangAuthors Info & Claims

BIGDSE '16: Proceedings of the 2nd International Workshop on BIG Data Software Engineering

Pages 58 - 64

https://doi.org/10.1145/2896825.2896828

Published: 14 May 2016 Publication History

Abstract

Elasticity is a key component of modern cloud environments and monitoring is an essential part of this process. Monitoring demonstrates several challenges including gathering metrics from a variety of layers (infrastructure, platform, application), the need for fast processing of this data to enable efficient elasticity and the proper management of this data in order to facilitate analysis of current and past data and future predictions. In this work, we classify monitoring as a big data problem and propose appropriate solutions in a layered, pluggable and extendable architecture for a monitoring component. More specifically, we propose the use of NoSQL databases as the back-end and BigQueue as a write buffer to achieve high throughput. Our evaluation shows that our monitoring is capable of achieving response time of a few hundreds of milliseconds for the insertion of hundreds of rows regardless of the underlying NoSQL database.

References

[1]

M. Anand. Cloud monitor: Monitoring applications in cloud. IEEE Cloud Computing for Emerging Markets, CCEM 2012 - Proceedings, pages 58--61, 2012.

[2]

O. Foundation. Ceilometer System Architecture. http://docs.openstack.org/developer/ceilometer/architecture.html, 2015.

[3]

S. I. Gatherer and Reporter. SIGAR. http://sigar.hyperic.com, 2015.

[4]

P. Horn. Autonomic computing: Ibms perspective on the state of information technology. 2001.

[5]

H. Khazaei, M. Fokaefs, S. Zareian, N. Beigi-Mohammadi, B. Ramprasad, M. Shtern, P. Gaikwad, and M. Litoiu. How do I choose the right nosql solution? a comprehensive theoretical and experimental survey. Big Data and Information Analytics (BDIA), (2):1, 2016.

[6]

J. Klein, I. Gorton, N. Ernst, P. Donohoe, K. Pham, and C. Matser. Performance evaluation of nosql databases: A case study. In Proceedings of the 1st Workshop on Performance Analysis of Big Data Systems, PABS '15, pages 5--10, New York, NY, USA, 2015. ACM.

Digital Library

[7]

J. Lin, R. Ravichandiran, H. Bannazadeh, and A. Leon-garcia. Monitoring and Measurement in Software-Defined Infrastructure. In Integrated Network Management (IM), 2015 IFIP/IEEE International Symposium on, pages 742--745, 2015.

[8]

S. Meng, A. K. Iyengar, I. M. Rouvellou, L. Liu, K. Lee, B. Palanisamy, and Y. Tang. Reliable state monitoring in cloud datacenters. Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012, pages 951--958, 2012.

Digital Library

[9]

S. Meng, S. Member, L. Liu, and S. Member. Enhanced Monitoring-as-a-Service for Effective Cloud Management. Journal of IEEE Transactions on Computers, 62(9):1705--1720, 2013.

Digital Library

[10]

Opesntack Foundation. Monasca Architecture. https://wiki.openstack.org/wiki/Monasca{\#}Architecture, 2015.

[11]

SAVI. Cloud platform. http://www.savinetwork.ca, June 2015.

[12]

A. W. Services. Amazon Cloudwatch Architecture. http://docs.aws.amazon.com/AmazonCloudWatch, 2015.

[13]

M. Smit, B. Simmons, and M. Litoiu. Distributed, application-level monitoring for heterogeneous clouds using stream processing. Future Generation Computer Systems, 29(8):2103--2114, 2013.

Digital Library

[14]

H. Yongdnog, W. Jing, Z. Zhuofeng, and H. Yanbo. A Scalable and Integrated Cloud Monitoring Framework Based on Distributed Storage. 2013 10th Web Information System and Application Conference, pages 318--323, 2013.

Digital Library

[15]

S. Zareian, R. Veleda, M. Litoiu, M. Shtern, H. Ghanbari, and M. Garg. K-Feed - a data-oriented approach to application performance management in cloud. In IEEE 8th International Conference on Cloud Computing, pages 1045--1048, June 2015.

Digital Library

Cited By

Xia HLi JMilisavljevic-Syed JSalonitis K(2024)Transformation of the Product Lifecycle Value Chain Towards Industry 5.0IFAC-PapersOnLine10.1016/j.ifacol.2024.08.12058:8(198-203)Online publication date: 2024
https://doi.org/10.1016/j.ifacol.2024.08.120
Bisicchia GForti SColla APisa CBarchiesi ABrogi A(2024)A Distributed Tool for Monitoring and Benchmarking a National Federated CloudCloud Computing and Services Science10.1007/978-3-031-68165-3_5(92-112)Online publication date: 15-Aug-2024
https://doi.org/10.1007/978-3-031-68165-3_5
Sarda KNamrud ZRouf RAhuja HRasolroveicy MLitoiu MShwartz LWatts IOnut IShirani POnut IOnut IBranco P(2023)ADARMA Auto-Detection and Auto-Remediation of Microservice Anomalies by Leveraging Large Language ModelsProceedings of the 33rd Annual International Conference on Computer Science and Software Engineering10.5555/3615924.3615949(200-205)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.5555/3615924.3615949
Show More Cited By

Index Terms

A big data framework for cloud monitoring

Recommendations

Big data analytics in Cloud computing: an overview
Abstract
Big Data and Cloud Computing as two mainstream technologies, are at the center of concern in the IT field. Every day a huge amount of data is produced from different sources. This data is so big in size that traditional processing tools are unable ...
The anatomy of big data computing

Advances in information technology and its widespread growth in several areas of business, engineering, medical, and scientific studies are resulting in information/data explosion. Knowledge discovery and decision-making from such rapidly growing ...
Cloud computing for big data analytics How cloud computing can handle procesing large amounts of data and improve real-time data analytics
Abstract
With the increasing volume, velocity, and variety of data generated by various sources, big data has become a critical challenge for many organizations. Cloud computing provides an efficient and cost-effective solution to address the challenges of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

BIGDSE '16: Proceedings of the 2nd International Workshop on BIG Data Software Engineering

May 2016

92 pages

ISBN:9781450341523

DOI:10.1145/2896825

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
450
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xia HLi JMilisavljevic-Syed JSalonitis K(2024)Transformation of the Product Lifecycle Value Chain Towards Industry 5.0IFAC-PapersOnLine10.1016/j.ifacol.2024.08.12058:8(198-203)Online publication date: 2024
https://doi.org/10.1016/j.ifacol.2024.08.120
Bisicchia GForti SColla APisa CBarchiesi ABrogi A(2024)A Distributed Tool for Monitoring and Benchmarking a National Federated CloudCloud Computing and Services Science10.1007/978-3-031-68165-3_5(92-112)Online publication date: 15-Aug-2024
https://doi.org/10.1007/978-3-031-68165-3_5
Sarda KNamrud ZRouf RAhuja HRasolroveicy MLitoiu MShwartz LWatts IOnut IShirani POnut IOnut IBranco P(2023)ADARMA Auto-Detection and Auto-Remediation of Microservice Anomalies by Leveraging Large Language ModelsProceedings of the 33rd Annual International Conference on Computer Science and Software Engineering10.5555/3615924.3615949(200-205)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.5555/3615924.3615949
Zhou QShen J(2023)Low temperature plasma sterilization intelligent control system design based on PLC and cloud platform2023 3rd International Symposium on Computer Technology and Information Science (ISCTIS)10.1109/ISCTIS58954.2023.10213211(834-838)Online publication date: 7-Jul-2023
https://doi.org/10.1109/ISCTIS58954.2023.10213211
Khanahmadi MShameli‐Sendi AJabbarifar MFournier QDagenais M(2023)Detection of microservice‐based software anomalies based on OpenTracing in cloudSoftware: Practice and Experience10.1002/spe.320853:8(1681-1699)Online publication date: 8-Apr-2023
https://doi.org/10.1002/spe.3208
Khudur APolos R(2021)Gathering Multi-dimensional Data of Scalable Container-based Cluster for Performance Analysis2021 Fourth International Conference on Computational Intelligence and Communication Technologies (CCICT)10.1109/CCICT53244.2021.00037(139-145)Online publication date: Jul-2021
https://doi.org/10.1109/CCICT53244.2021.00037
Gu RQi YWu TWang ZXu XYuan CHuang Y(2021)SparkDQ: Efficient generic big data quality management on distributed data-parallel computationJournal of Parallel and Distributed Computing10.1016/j.jpdc.2021.05.012156(132-147)Online publication date: Oct-2021
https://doi.org/10.1016/j.jpdc.2021.05.012
Ji SLi QCao WZhang PMuccini H(2020)Quality Assurance Technologies of Big Data Applications: A Systematic Literature ReviewApplied Sciences10.3390/app1022805210:22(8052)Online publication date: 13-Nov-2020
https://doi.org/10.3390/app10228052
Großmann MIllig SMatějka CWang B(2019)SensIoTWireless Communications & Mobile Computing10.1155/2019/42603592019Online publication date: 7-Mar-2019
https://dl.acm.org/doi/10.1155/2019/4260359
Milechin LGadepally VKepner J(2018)Database Operations in D4M.jl2018 IEEE High Performance extreme Computing Conference (HPEC)10.1109/HPEC.2018.8547567(1-5)Online publication date: Sep-2018
https://doi.org/10.1109/HPEC.2018.8547567
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten