article

InteMon: continuous mining of sensor data in large-scale self-infrastructures

Authors:

John D. Strunk,

Gregory R. Ganger,

Christos FaloutsosAuthors Info & Claims

ACM SIGOPS Operating Systems Review, Volume 40, Issue 3

Pages 38 - 44

https://doi.org/10.1145/1151374.1151384

Published: 01 July 2006 Publication History

Abstract

Modern data centers have a large number of components that must be monitored, including servers, switches/routers, and environmental control systems. This paper describes InteMon, a prototype monitoring and mining system for data centers. It uses the SNMP protocol to monitor a new data center at Carnegie Mellon. It stores the monitoring data in a MySQL database, allowing visualization of the time-series data using a JSP web-based frontend interface for system administrators. What sets InteMon apart from other cluster monitoring systems is its ability to automatically analyze correlations in the monitoring data in real time and alert administrators of potential anomalies. It uses efficient, state of the art stream mining methods to report broken correlations among input streams. It also uses these methods to intelligently compress historical data and avoid the need for administrators to configure threshold-based monitoring bands.

References

[1]

D. J. Abadi, D. Carney, U. Cetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik. Aurora: a new model and architecture for data stream management. The VLDB Journal, 12(2):120--139, 2003.]]

Digital Library

[2]

A. Arasu, B. Babcock, S. Babu, J. McAlister, and J. Widom. Characterizing memory requirements for queries over continuous data streams. In PODS, 2002.]]

Digital Library

[3]

B. Babcock, S. Babu, M. Datar, and R. Motwani. Chain: Operator scheduling for memory minimization in data stream systems. In SIGMOD, pages 253--264, 2003.]]

Digital Library

[4]

P. Barham, R. Isaacs, R. Mortier, and D. Narayanan. Magpie: Online modelling and performance-aware systems. In HOTOS, pages 79--84. USENIX Association, 2003.]]

Digital Library

[5]

Big Brother. http://www.bb4.org.]]

[6]

G. E. Box and G. M. Jenkins. Time Series Analysis: Forecasting and Control. Holden-Day Inc., San Francisco, revised edition, 1976.]]

Digital Library

[7]

G. E. Box, G. M. Jenkins, and G. C. Reinsel. Time Series Analysis: Forecasting and Control. Prentice Hall, Englewood Cliffs, NJ, 3rd edition, 1994.]]

Digital Library

[8]

R. Buyya. PARMON: a portable and scalable monitoring system for clusters. Software - Practice and Experience, 30(7):723--739, 2000.]]

Digital Library

[9]

D. Carney, U. Cetintemel, A. Rasin, S. B. Zdonik, M. Cherniack, and M. Stonebraker. Operator scheduling in a data stream manager. In VLDB, 2003.]]

Digital Library

[10]

J. Case, M. Fedor, M. Schoffstall, and J. Davin. A simple network management protocol (SNMP). RFC 1157, Network Working Group, 1990.]]

Digital Library

[11]

S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, V. Raman, F. Reiss, and M. A. Shah. Telegraphcq: Continuous dataflow processing for an uncertain world. In CIDR, 2003.]]

[12]

E. Cohen and M. Strauss. Maintaining time-decaying stream aggregates. In SIGMOD, 2003.]]

Digital Library

[13]

C. Cranor, T. Johnson, O. Spataschek, and V. Shkapenyuk. Gigascope: a stream database for network applications. In SIGMOD, 2003.]]

Digital Library

[14]

A. Das, J. Gehrke, and M. Riedewald. Approximate join processing over data streams. In SIGMOD, pages 40--51, 2003.]]

Digital Library

[15]

I. Daubechies. Ten Lectures on Wavelets. Capital City Press, Montpelier, Vermont, 1992. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA.]]

Digital Library

[16]

A. Deshpande, C. Guestrin, S. Madden, and W. Hong. Exploiting correlated attributes in acqusitional query processing. In ICDE, 2005.]]

Digital Library

[17]

K. Fukunaga. Introduction to Statistical Pattern Recognition. Academic Press, 2nd edition, 1990.]]

Digital Library

[18]

HP OpenView. http://www.managementsoftware.hp.com/index.html.]]

[19]

IBM Tivoli. http://www.ibm.com/software/tivoli/.]]

[20]

J. Moore, J. Chase, and P. Ranganathan. Weatherman: Automated, online, and predictive thermal mapping and management for data centers. In International Conference on Autonomic Computing, 2006.]]

Digital Library

[21]

R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Olston, J. Rosenstein, and R. Varma. Query processing, resource management, and approximation in a data stream management system. In CIDR, 2003.]]

[22]

Nagios. http://www.nagios.org.]]

[23]

A. V. Oppenheim and R. W. Schafer. Digital Signal Processing. Prentice-Hall, Englewood Cliffs, N. J., 1975.]]

[24]

S. Papadimitriou, A. Brockwell, and C. Faloutsos. Adaptive, hands-off stream mining. VLDB, Sept. 2003.]]

Digital Library

[25]

S. Papadimitriou, J. Sun, and C. Faloutsos. Streaming pattern discovery in multiple time-series. In VLDB, pages 697--708, 2005.]]

Digital Library

[26]

F. D. Sacerdoti, M. J. Katz, M. L. Massie, and D. E. Culler. Wide area cluster monitoring with ganglia. In CLUSTER, 2003.]]

[27]

M. J. Sottile and R. Minnich. Supermon: A high-speed cluster monitoring system. In CLUSTER, pages 39--46, 2002.]]

Digital Library

[28]

N. Tatbul, U. Cetintemel, S. B. Zdonik, M. Cherniack, and M. Stonebraker. Load shedding in a data stream manager. In VLDB, 2003.]]

Digital Library

[29]

TC9.9 Mission Critical Facilities. Thermal Guidelines for Data Processing Environments. ASHRAE, 2004.]]

Cited By

Singh AStephan EElsethagen TMacDuff MRaju BSchram Mvan Dam KKerbyson DAltintas I(2016)Leveraging large sensor streams for robust cloud control2016 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2016.7840839(2115-2120)Online publication date: Dec-2016
https://doi.org/10.1109/BigData.2016.7840839
Tang LZeng CLi TShwartz LGraharnik G(2015)Tuning up IT Services using Monitoring Configuration AnalyticsMaximizing Management Performance and Quality with Service Analytics10.4018/978-1-4666-8496-6.ch007(179-206)Online publication date: 2015
https://doi.org/10.4018/978-1-4666-8496-6.ch007
Yanhaona MProdhan MGrimshaw A(2015)An agent-based distributed monitoring framework (Extended abstract)2015 International Conference on Networking Systems and Security (NSysS)10.1109/NSysS.2015.7043515(1-10)Online publication date: Jan-2015
https://doi.org/10.1109/NSysS.2015.7043515
Show More Cited By

Index Terms

InteMon: continuous mining of sensor data in large-scale self-infrastructures

Recommendations

InteMon: intelligent system monitoring on large clusters
VLDB '06: Proceedings of the 32nd international conference on Very large data bases

InteMon is a prototype monitoring and mining system for large clusters. Currently, it monitors over 100 hosts of a prototype data center at CMU. It uses the SNMP protocol and it stores the monitoring data in an mySQL database. Then, it allows for ...
CCNA Data Center DCICN 640-911 Official Cert Guide
Cisco Icnd2 Official Cert Guide With Myitcertificationlabs B

Comments

Information & Contributors

Information

Published In

cover image ACM SIGOPS Operating Systems Review

ACM SIGOPS Operating Systems Review Volume 40, Issue 3

July 2006

107 pages

ISSN:0163-5980

DOI:10.1145/1151374

Issue’s Table of Contents

Copyright © 2006 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2006

Published in SIGOPS Volume 40, Issue 3

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Singh AStephan EElsethagen TMacDuff MRaju BSchram Mvan Dam KKerbyson DAltintas I(2016)Leveraging large sensor streams for robust cloud control2016 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2016.7840839(2115-2120)Online publication date: Dec-2016
https://doi.org/10.1109/BigData.2016.7840839
Tang LZeng CLi TShwartz LGraharnik G(2015)Tuning up IT Services using Monitoring Configuration AnalyticsMaximizing Management Performance and Quality with Service Analytics10.4018/978-1-4666-8496-6.ch007(179-206)Online publication date: 2015
https://doi.org/10.4018/978-1-4666-8496-6.ch007
Yanhaona MProdhan MGrimshaw A(2015)An agent-based distributed monitoring framework (Extended abstract)2015 International Conference on Networking Systems and Security (NSysS)10.1109/NSysS.2015.7043515(1-10)Online publication date: Jan-2015
https://doi.org/10.1109/NSysS.2015.7043515
Malik HShakshuki E(2014)Classification of Post-deployment Performance Diagnostic Techniques for Large-scale Software SystemsProcedia Computer Science10.1016/j.procs.2014.08.03637(244-251)Online publication date: 2014
https://doi.org/10.1016/j.procs.2014.08.036
Shafer IRen KBoddeti VAbe YGanger GFaloutsos CYang QAgarwal DPei J(2012)RainMonProceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/2339530.2339711(1158-1166)Online publication date: 12-Aug-2012
https://dl.acm.org/doi/10.1145/2339530.2339711
Wang LLaszewski GHuang FDayal JFrulani TFox G(2011)Task scheduling with ANN-based temperature prediction in a data centerEngineering with Computers10.5555/3225276.322549827:4(381-391)Online publication date: 1-Oct-2011
https://dl.acm.org/doi/10.5555/3225276.3225498
Patnaik DMarwah MSharma RRamakrishnan N(2011)Temporal data mining approaches for sustainable chiller management in data centersACM Transactions on Intelligent Systems and Technology10.1145/1989734.19897382:4(1-29)Online publication date: 15-Jul-2011
https://dl.acm.org/doi/10.1145/1989734.1989738
Wang LKhan SDayal J(2011)Thermal aware workload placement with task-temperature profiles in a data centerThe Journal of Supercomputing10.1007/s11227-011-0635-z61:3(780-803)Online publication date: 7-Jun-2011
https://doi.org/10.1007/s11227-011-0635-z
Wang Lvon Laszewski GHuang FDayal JFrulani TFox G(2011)Task scheduling with ANN-based temperature prediction in a data center: a simulation-based studyEngineering with Computers10.1007/s00366-011-0211-427:4(381-391)Online publication date: 18-Feb-2011
https://doi.org/10.1007/s00366-011-0211-4
Teixeira PClemente RKaiser RVieira DPietzuch PBacon JSventek JCetintemel U(2010)HOLMESProceedings of the Fourth ACM International Conference on Distributed Event-Based Systems10.1145/1827418.1827461(216-221)Online publication date: 12-Jul-2010
https://dl.acm.org/doi/10.1145/1827418.1827461
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents