Skip to main content

Advertisement

Log in

On construction of an energy monitoring service using big data technology for the smart campus

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

In this study, we combine cloud computing with big data processing techniques to build a real-time energy monitoring system for smart campus. The monitor plat-form collects the electricity usage in campus buildings through smart meters and environmental sensors, and processes the huge amount of data by big data processing techniques. A Hadoop ecosystem is built on top of big data processing architecture to improve the capacity of big data storage and processing ability. Moreover, we compare the performance of Hive and HBase in searching energy data, and the performance of relational database and big data distributed database for data search. We also identify abnormal electrical condition through the MapReduce framework, and compared the difference of performances between Spark and Hadoop in real-time processing. The proposed system has been implemented in Tunghai University campus. It enables administrators to observe the real-time electricity usage and analyze historical data anytime and from anyplace.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32

Similar content being viewed by others

References

  1. Yang, C.-T., Chen, C.-J., Tsan, Y.-T., Liu, P.-Y., Chan, Y.-W., Chan, W.-C.: An implementation of real-time air quality and influenza-like illness data storage and processing platform. Comput. Hum. Behav. (2018a)

  2. Yang, C.-T., Chen, S.-T., Den, W., Wang, Y.-T., Kristiani, E.: Implementation of an intelligent indoor environmental monitoring and management system in cloud. Fut. Gener. Comput. Syst. (2018b)

  3. Yang, C.-T., Chen, S.-T., Yan, Y.-Z.: The implementation of a cloud city traffic state assessment system using a novel big data architecture. Cluster Comput. 20, 1101–1121 (2017)

    Article  Google Scholar 

  4. Liu, P.-Y., Tsan, Y.-T., Chan, Y.-W., Chan, W.-C., Shi, Z.-Y., Yang, C.-T., Lou, B.-S.: Associations of PM2.5 and Aspergillosis: ambient fine particulate air pollution and population-based big data linkage analyses. J Ambient Intell Humaniz Comput(2018)

  5. Yang, C.-T., Liu, J.-C., Chen, S.-T., Lu, H.-W.: Implementation of a big data accessing and processing platform for medical records in cloud. J. Med. Syst. 41, 149 (2017)

    Article  Google Scholar 

  6. Kambatla, K., Kollias, G., Kumar, V., Grama, A.: Trends in big data analytics. J. Parallel Distrib. Comput. 74, 2561 (2014)

    Article  Google Scholar 

  7. Yang, C.T., Chen, L.T., Chou, W.L., Wang, K.C.: Implementation of a medical image file accessing system on cloud computing. In: Proceedings - 2010 13th IEEE International Conference on Computational Science and Engineering, CSE 2010, pp. 321–326 (2010)

  8. Hassan, Q.: Demystifying cloud computing. J. Def. Softw. Eng. 2011, 16–21 (2011)

    Google Scholar 

  9. Mell, P., Grance, T., et al.: The NIST definition of cloud computing. Natl. Inst. Stand. Technol. 53, 50 (2009)

    Google Scholar 

  10. Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19, 171–209 (2014)

    Article  Google Scholar 

  11. White, T.: Hadoop: the definitive guide. O’Reilly Media Inc, Newton (2012)

    Google Scholar 

  12. Kusnetzky, D.: What is big data?, ZDNet. http://www.zdnet.com/blog/virtualization/what-is-big-data/1708 (2010)

  13. Laney D (2001) 3D data management: controlling data volume, velocity and variety. META Group Research Note, 6, p 70

  14. Skiba, D.J.: The internet of things (iot). Nurs. Educ. Perspect. 34, 63–64 (2013)

    Article  Google Scholar 

  15. CISCO, The Internet of Things, Infographic (2015). http://blogs.cisco.com/news/the-internet-of-things-infographic/

  16. Hadoop (2014). http://hadoop.apache.org/

  17. Borthakur, D.: The hadoop distributed file system: architecture and design (2007)

  18. Azzedin, F.: Towards a scalable HDFS architecture. In: Proceedings of the 2013 International Conference on Collaboration Technologies and Systems, CTS pp. 155–161 (2013)

  19. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. HotCloud 10, 10 (2010)

    Google Scholar 

  20. MapReduce (2014). http://en.wikipedia.org/wiki/MapReduce

  21. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, USENIX Association, p. 2

  22. Apache Hive (2015). https://en.wikipedia.org/wiki/Apache_Hive

  23. Venner, J.: Advanced and Alternate MapReduce Techniques. Pro Hadoop, pp. 239–284 (2009)

  24. Lam, C.: Hadoop in Action. Manning Publications Co., New York (2010)

    Google Scholar 

  25. Thusoo, A., Sarma, J.S., Jain, N., Shao, Z., Chakka, P., Zhang, N., Antony, S., Liu, H., Murthy, R.: Hive—A Petabyte Scale Data Warehouse Using Hadoop, Proceedings - International Conference on Data Engineering, pp. 996–1005 (2010)

  26. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. ACM Trans. Comput. Syst. (TOCS) 26, 4 (2008)

    Article  Google Scholar 

  27. Jin, Y., Deyu, T., Yi, Z.: A distributed storage Model for EHR based on HBase. In: Information Management, Innovation Management and Industrial Engineering (ICIII), 2011 International Conference on, vol. 2, pp. 369–372

  28. Ding, H., Jin, Y., Cui, Y., Yang, T.: Distributed storage of network measurement data on HBase. In: Cloud Computing and Intelligent Systems (CCIS), 2012 IEEE 2nd International Conference on, vol. 02, pp. 716–720 (2012)

  29. Bai, J.: Feasibility analysis of big log data real time search based on Hbase and elastic search. In: Natural Computation (ICNC), 2013 Ninth International Conference on, pp. 1166–1170 (2013 )

  30. Vora, M.N.: Hadoop-HBase for large-scale data. In: Computer Science and Network Technology (ICCSNT), International Conference on, vol. 1, pp. 601–605 (2011)

  31. Cai, L., Huang, S., Chen, L.,  Zheng, Y.: Performance analysis and testing of HBase based on its architecture. In: Computer and Information Science (ICIS), IEEE/ACIS 12th International Conference on, pp. 353–358 (2013 )

  32. Apache Hbase (2014). http://wiki.apache.org/hadoop/Hbase

  33. George, L.: HBase: The Definitive Guide, O’REILLY (2012)

  34. Rathore, M.M., Ahmad, A., Paul, A., Rho, S.: Urban planning and building smart cities based on the internet of things using big data analytics. Comput. Netw. 101, 63–80 (2016)

    Article  Google Scholar 

  35. Lin, X., Wang, P., Wu, B.: Log analysis in cloud computing environment with Hadoop and Spark. In: 2013 5th IEEE International Conference on Broadband Network & Multimedia Technology, pp. 273–276 (2013)

  36. Yang, C.-T., Yan, Y.-Z., Liu, R.-H., Chen, S.-T.: Cloud city traffic state assessment system using a novel architecture of big data. In: 2015 International Conference on Cloud Computing and Big Data (CCBD) (2015)

  37. Zhang, C., Liu, X.: HBaseMQ: a distributed message queuing system on clouds with HBase. In: INFOCOM, 2013 Proceedings IEEE, pp. 40–44

  38. Hu, Y.W., Xu, Y., Liu, Y., Chen, J., Lin, S.: Qmapper for smart Grid: Migrating SQL-based Application to Hive. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (2015)

  39. Gruenheid, A., Omiecinski, E., Mark, L.: Query optimization using column statistics in Hive. In: ACM International Conference Proceeding Series, pp. 97–105 (2011)

  40. Liu, R.-H., Kuo, C.-F., Yang, C.-T., Chen, S.-T., Liu, J.-C.: On construction of an energy monitoring service using big data technology for smart campus. In: 2016 7th International Conference on Cloud Computing and Big Data (CCBD), pp. 403–410

  41. Rubén Pérez-Chacón, J.C.R., Luna-Romera, José M., Troncoso, Alicia, Martínez-Álvarez, Francisco: Big data analytics for discovering electricity consumption patterns in smart cities. Energies 11, 1–19 (2018)

    Google Scholar 

  42. Adeyemi, O.J., Popoola, S.I., Atayero, A.A., Afolayan, D.G., Ariyo, M., Adetiba, E.: Exploration of daily internet data traffic generated in a smart university campus. Data Brief 20, 30–52 (2018)

    Article  Google Scholar 

  43. Popoola, S.I., Atayero, A.A., Okanlawon, T.T., Omopariola, B.I., Takpor, O.A.: Smart campus: data on energy consumption in an ICT-driven university. Data Brief 16, 780–793 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

This work was sponsored by the Ministry of Science and Technology (MOST), Taiwan, under grants number 104-2221-E-029-010-MY3 and 106-2622-E-029-002-CC3.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chao-Tung Yang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, CT., Chen, ST., Liu, JC. et al. On construction of an energy monitoring service using big data technology for the smart campus. Cluster Comput 23, 265–288 (2020). https://doi.org/10.1007/s10586-019-02921-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-019-02921-5

Keywords

Navigation