Abstract
In this study, we combine cloud computing with big data processing techniques to build a real-time energy monitoring system for smart campus. The monitor plat-form collects the electricity usage in campus buildings through smart meters and environmental sensors, and processes the huge amount of data by big data processing techniques. A Hadoop ecosystem is built on top of big data processing architecture to improve the capacity of big data storage and processing ability. Moreover, we compare the performance of Hive and HBase in searching energy data, and the performance of relational database and big data distributed database for data search. We also identify abnormal electrical condition through the MapReduce framework, and compared the difference of performances between Spark and Hadoop in real-time processing. The proposed system has been implemented in Tunghai University campus. It enables administrators to observe the real-time electricity usage and analyze historical data anytime and from anyplace.
Similar content being viewed by others
References
Yang, C.-T., Chen, C.-J., Tsan, Y.-T., Liu, P.-Y., Chan, Y.-W., Chan, W.-C.: An implementation of real-time air quality and influenza-like illness data storage and processing platform. Comput. Hum. Behav. (2018a)
Yang, C.-T., Chen, S.-T., Den, W., Wang, Y.-T., Kristiani, E.: Implementation of an intelligent indoor environmental monitoring and management system in cloud. Fut. Gener. Comput. Syst. (2018b)
Yang, C.-T., Chen, S.-T., Yan, Y.-Z.: The implementation of a cloud city traffic state assessment system using a novel big data architecture. Cluster Comput. 20, 1101–1121 (2017)
Liu, P.-Y., Tsan, Y.-T., Chan, Y.-W., Chan, W.-C., Shi, Z.-Y., Yang, C.-T., Lou, B.-S.: Associations of PM2.5 and Aspergillosis: ambient fine particulate air pollution and population-based big data linkage analyses. J Ambient Intell Humaniz Comput(2018)
Yang, C.-T., Liu, J.-C., Chen, S.-T., Lu, H.-W.: Implementation of a big data accessing and processing platform for medical records in cloud. J. Med. Syst. 41, 149 (2017)
Kambatla, K., Kollias, G., Kumar, V., Grama, A.: Trends in big data analytics. J. Parallel Distrib. Comput. 74, 2561 (2014)
Yang, C.T., Chen, L.T., Chou, W.L., Wang, K.C.: Implementation of a medical image file accessing system on cloud computing. In: Proceedings - 2010 13th IEEE International Conference on Computational Science and Engineering, CSE 2010, pp. 321–326 (2010)
Hassan, Q.: Demystifying cloud computing. J. Def. Softw. Eng. 2011, 16–21 (2011)
Mell, P., Grance, T., et al.: The NIST definition of cloud computing. Natl. Inst. Stand. Technol. 53, 50 (2009)
Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19, 171–209 (2014)
White, T.: Hadoop: the definitive guide. O’Reilly Media Inc, Newton (2012)
Kusnetzky, D.: What is big data?, ZDNet. http://www.zdnet.com/blog/virtualization/what-is-big-data/1708 (2010)
Laney D (2001) 3D data management: controlling data volume, velocity and variety. META Group Research Note, 6, p 70
Skiba, D.J.: The internet of things (iot). Nurs. Educ. Perspect. 34, 63–64 (2013)
CISCO, The Internet of Things, Infographic (2015). http://blogs.cisco.com/news/the-internet-of-things-infographic/
Hadoop (2014). http://hadoop.apache.org/
Borthakur, D.: The hadoop distributed file system: architecture and design (2007)
Azzedin, F.: Towards a scalable HDFS architecture. In: Proceedings of the 2013 International Conference on Collaboration Technologies and Systems, CTS pp. 155–161 (2013)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. HotCloud 10, 10 (2010)
MapReduce (2014). http://en.wikipedia.org/wiki/MapReduce
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, USENIX Association, p. 2
Apache Hive (2015). https://en.wikipedia.org/wiki/Apache_Hive
Venner, J.: Advanced and Alternate MapReduce Techniques. Pro Hadoop, pp. 239–284 (2009)
Lam, C.: Hadoop in Action. Manning Publications Co., New York (2010)
Thusoo, A., Sarma, J.S., Jain, N., Shao, Z., Chakka, P., Zhang, N., Antony, S., Liu, H., Murthy, R.: Hive—A Petabyte Scale Data Warehouse Using Hadoop, Proceedings - International Conference on Data Engineering, pp. 996–1005 (2010)
Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. ACM Trans. Comput. Syst. (TOCS) 26, 4 (2008)
Jin, Y., Deyu, T., Yi, Z.: A distributed storage Model for EHR based on HBase. In: Information Management, Innovation Management and Industrial Engineering (ICIII), 2011 International Conference on, vol. 2, pp. 369–372
Ding, H., Jin, Y., Cui, Y., Yang, T.: Distributed storage of network measurement data on HBase. In: Cloud Computing and Intelligent Systems (CCIS), 2012 IEEE 2nd International Conference on, vol. 02, pp. 716–720 (2012)
Bai, J.: Feasibility analysis of big log data real time search based on Hbase and elastic search. In: Natural Computation (ICNC), 2013 Ninth International Conference on, pp. 1166–1170 (2013 )
Vora, M.N.: Hadoop-HBase for large-scale data. In: Computer Science and Network Technology (ICCSNT), International Conference on, vol. 1, pp. 601–605 (2011)
Cai, L., Huang, S., Chen, L., Zheng, Y.: Performance analysis and testing of HBase based on its architecture. In: Computer and Information Science (ICIS), IEEE/ACIS 12th International Conference on, pp. 353–358 (2013 )
Apache Hbase (2014). http://wiki.apache.org/hadoop/Hbase
George, L.: HBase: The Definitive Guide, O’REILLY (2012)
Rathore, M.M., Ahmad, A., Paul, A., Rho, S.: Urban planning and building smart cities based on the internet of things using big data analytics. Comput. Netw. 101, 63–80 (2016)
Lin, X., Wang, P., Wu, B.: Log analysis in cloud computing environment with Hadoop and Spark. In: 2013 5th IEEE International Conference on Broadband Network & Multimedia Technology, pp. 273–276 (2013)
Yang, C.-T., Yan, Y.-Z., Liu, R.-H., Chen, S.-T.: Cloud city traffic state assessment system using a novel architecture of big data. In: 2015 International Conference on Cloud Computing and Big Data (CCBD) (2015)
Zhang, C., Liu, X.: HBaseMQ: a distributed message queuing system on clouds with HBase. In: INFOCOM, 2013 Proceedings IEEE, pp. 40–44
Hu, Y.W., Xu, Y., Liu, Y., Chen, J., Lin, S.: Qmapper for smart Grid: Migrating SQL-based Application to Hive. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (2015)
Gruenheid, A., Omiecinski, E., Mark, L.: Query optimization using column statistics in Hive. In: ACM International Conference Proceeding Series, pp. 97–105 (2011)
Liu, R.-H., Kuo, C.-F., Yang, C.-T., Chen, S.-T., Liu, J.-C.: On construction of an energy monitoring service using big data technology for smart campus. In: 2016 7th International Conference on Cloud Computing and Big Data (CCBD), pp. 403–410
Rubén Pérez-Chacón, J.C.R., Luna-Romera, José M., Troncoso, Alicia, Martínez-Álvarez, Francisco: Big data analytics for discovering electricity consumption patterns in smart cities. Energies 11, 1–19 (2018)
Adeyemi, O.J., Popoola, S.I., Atayero, A.A., Afolayan, D.G., Ariyo, M., Adetiba, E.: Exploration of daily internet data traffic generated in a smart university campus. Data Brief 20, 30–52 (2018)
Popoola, S.I., Atayero, A.A., Okanlawon, T.T., Omopariola, B.I., Takpor, O.A.: Smart campus: data on energy consumption in an ICT-driven university. Data Brief 16, 780–793 (2018)
Acknowledgements
This work was sponsored by the Ministry of Science and Technology (MOST), Taiwan, under grants number 104-2221-E-029-010-MY3 and 106-2622-E-029-002-CC3.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yang, CT., Chen, ST., Liu, JC. et al. On construction of an energy monitoring service using big data technology for the smart campus. Cluster Comput 23, 265–288 (2020). https://doi.org/10.1007/s10586-019-02921-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-019-02921-5