Skip to main content

Advertisement

SpringerLink
Log in

A new outlier rejection methodology for supporting load forecasting in smart grids based on big data

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Internet of things (IoT) enables smart electrical grids (SEGs) to solve its problems and to support a lot of tasks. These tasks include power monitoring, demand-side energy management coordination of distributed storage, and the integration of renewable energy generators. Sending the complete captured data directly to the cloud would lead to resource wastage. Hence, 2-tier architecture is replaced by 3-tier one in order to include a fog computing tier. Fog tier acts as a bridge in the middle between IoT devices embedded in SEG and cloud tier to overcome the cloud challenges. The main actions of the added fog tier are collecting, computing, and storing smart meters data before transmitting it to the cloud. In this paper, a new electrical load forecasting (ELF) strategy has been proposed based on the pre-mentioned 3-tier architecture. ELF consists of two main phases, which are; (i) data pre-processing phase (DP2) and (ii) load prediction phase (LP2). Both phases are executed at cloud servers (CSs) on the collected data, which is received from all fogs connected to the entire cloud. The main objective of DP2 is to; (i) select the meaningful features and (ii) eliminate outlier items from the collected data. The main contribution of this paper lied on outlier rejection phase. The paper introduces a new outlier rejection methodology called hybrid outlier rejection methodology (HORM). HORM try to eliminate all outliers from the training dataset before start learning the prediction model during LP2. HORM involves two stages which are; (i) a new statistical based outlier rejection stage, which is called fast outlier rejection (FOR) and (ii) an accurate outlier rejection (AOR) stage using genetic algorithm (GA). Then, the filtered data is used to give fast and accurate load prediction decisions. Experimental results have shown that the proposed HORM outperforms recent outlier rejection methods in terms of accuracy, precision, recall, and F1-measure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

References

  1. Rabie, A.S., Abo-Al-Ez, K.: A new strategy of load forecasting technique for smart grids. Int. J. Modern Trends Eng. Res. (IJMTER) 2(12), 332–341 (2015)

    Google Scholar 

  2. Saleh, A.R., Abo-Al-Ezb, K.: A data mining based load forecasting strategy for smart electrical grids. Adv. Eng. Inform. 30(3), 422–448 (2016)

    Article  Google Scholar 

  3. Ozger, M., Cetinkaya, O., Akan, O.: Energy harvesting cognitive radio networking for iot-enabled smart grid. Mob. Netw. Appl. 23(4), 956–966 (2018)

    Article  Google Scholar 

  4. Mahajan, V., Patil, P.: Internet of things based residential power load forecasting. Int. Res. J. Eng. Technol. (IRJET) 3(7), 1362–1364 (2016)

    Google Scholar 

  5. Atlam, H., Walters, R., Wills, G.: Fog computing and the internet of things: a review. Big Data Cognit. Comput. 2(10), 1–18 (2018)

    Google Scholar 

  6. Jaradat, M., Jarrah, M., Bousselham, A., Jararweh, Y., Al-Ayyouba, M.: The internet of energy: smart sensor networks and big data management for smart grid. Procedia Comput. Sci. 56, 592–597 (2015)

    Article  Google Scholar 

  7. Ghanbari, Z., Navimipour, N., Hosseinzadeh, M., Darwesh, A.: Resource allocation mechanisms and approaches on the internet of things. Comput Clust (2019). https://doi.org/10.1007/s10586-019-02910-8

    Article  Google Scholar 

  8. Yang, S.C., Liu, J., Liu, R., Chang, C.: On construction of an energy monitoring service using big data technology for the smart campus. Comput Clust (2019). https://doi.org/10.1007/s10586-019-02921-5

    Article  Google Scholar 

  9. Rabie, S.A., Ali, H., Saleh, A.: A fog based load forecasting strategy for smart grids using big electrical data. Clust. Comput. 22(1), 241–270 (2019)

    Article  Google Scholar 

  10. Madhusudhanan, P.S., Karpagam, N., Mahesh, A., Suhi, P.: An hybrid metaheuristic approach for efficient feature selection. Comput Clust (2018). https://doi.org/10.1007/s10586-018-2337-2

    Article  Google Scholar 

  11. Manoj, R., Praveena, M., Vijayakumar, K.: An ACO–ANN based feature selection algorithm for big data. Clust. Comput. (2018). https://doi.org/10.1007/s10586-018-2550-z

    Article  Google Scholar 

  12. Mao, J., Wang, T., Jin, C., Zhou, A.: Feature grouping-based outlier detection upon streaming trajectories. IEEE Trans. Knowl. Data Eng. 29(12), 2696–2709 (2017)

    Article  Google Scholar 

  13. Rahmani, M., Atia, G.: Randomized robust subspace recovery and outlier detection for high dimensional data matrices. IEEE Trans. Signal Process. 65(6), 1580–1594 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  14. Vasconcelos, R.V., Olivieri, B., Roriz, M., Endler, M., Junior, M.: Smartphone-based outlier detection: a complex event processing approach for driving behavior detection. J. Internet Serv. Appl. 8(13), 1–30 (2017)

    Google Scholar 

  15. Venkatesh, G., Arunesh, K.: Map Reduce for big data processing based on traffic aware partition and aggregation. Clust. Comput. (2018). https://doi.org/10.1007/s10586-018-1799-6

    Article  Google Scholar 

  16. VeeraManickam, M., Mohanapriya, M., Pandey, B., Akhade, S., Kale, S., Patil, R., Vigneshwar, M.: Map-Reduce framework based cluster architecture for academic student’s performance prediction using cumulative dragonfly based neural network. Clust. Comput. (2018). https://doi.org/10.1007/s10586-017-1553-5

    Article  Google Scholar 

  17. Tellis V, Souza D (2018) Detecting Anomalies in Data Stream Using Efficient Techniques: A Review. In: Proceedings of the 2018 International Conference on Control, Power, Communication and Computing Technologies (ICCPCCT), Kannur, India, pp. 296–298

  18. Park, C.H.: Outlier and anomaly pattern detection on data streams. J. Supercomput. (2018). https://doi.org/10.1007/s11227-018-2674-1

    Article  Google Scholar 

  19. Shou, Z., Li, S.: Large dataset summarization with automatic parameter optimization and parallel processing for local outlier detection. Concurr. Comput. Pract. Exp. 30(23), 1–13 (2018)

    Article  Google Scholar 

  20. Chomatek L, Duraj A (2019) Efficient Genetic Algorithm for Breast Cancer Diagnosis. In: Proceedings of the International Conference on Information Technologies in Biomedicine, ITIB 2018: Advances in Intelligent Systems and Computing, Springer, Cham, vol. 762, pp. 64–76

  21. Saneja, B., Rani, R.: A scalable correlation-based approach for outlier detection in wireless body sensor networks. Int. J. Commun Syst (2019). https://doi.org/10.1002/dac.3918

    Article  Google Scholar 

  22. Yan Y, Cao L, Rundensteiner E (2017) Distributed Top-N Local Outlier Detection in Big Data. In: Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, USA, pp. 827–836

  23. Liu, M.W., Newell, G.: Detecting outliers in species distribution data. J. Biogeogr. 45(1), 164–176 (2018)

    Article  Google Scholar 

  24. Liu, X., Zhou, Y., Chen, X.: Mining outlier data in mobile internet-based large real-time databases. Complex. Hindawi (2018). https://doi.org/10.1155/2018/9702304

    Article  Google Scholar 

  25. Okay F, Ozdemir S (2016) A fog Computing Based Smart Grid Model. In: Proceedings of the 2016 International Symposium on Networks, Computers and Communications (ISNCC), Yasmine Hammamet, Tunisia, pp. 1–6

  26. Yu, W., Liang, F., He, X., Hatcher, W., Lu, C., Lin, J., Yang, X.: A survey on the edge computing for the internet of things. IEEE Access IEEE 6, 6900–6919 (2018)

    Article  Google Scholar 

  27. Aiyad, S., Saleh, A., Labib, L.: A new distributed feature selection technique for classifying gene expression data. Int. J. Biomath. (2019). https://doi.org/10.1142/S1793524519500396

    Article  MathSciNet  MATH  Google Scholar 

  28. Posio, K.L., Ruuska, J., Ruha, P.: Outlier detection for 2D temperature data. IFAC Proc. 41(2), 1958–1963 (2008)

    Article  Google Scholar 

  29. Raja, P., Bhaskara, V.: An effective genetic algorithm for outlier detection. Int. J. Comput. Appl. 38(6), 30–33 (2012)

    Google Scholar 

  30. Afzal, M., Ashraf, S.: Genetic algorithm for outlier detection. Int. J. Comput. Sci. Inf. Technol. (IJCSIT) 7(2), 833–835 (2016)

    Google Scholar 

  31. Zhang, Y., Meratnia, N., Havinga, P.: Outlier detection techniques for wireless sensor networks: a survey. IEEE Commun. Surv. Tutor. 12(2), 159–170 (2010)

    Article  Google Scholar 

  32. Yu, N., Zhang, L., Ren, Y.: A novel D-S based secure localization algorithm for wireless sensor networks. Sec. Commun. Netw. 7(11), 1945–1954 (2014)

    Article  Google Scholar 

  33. Revathi, L., Appandiraj, A.: Hadoop based parallel framework for feature subset selection in big data. Int. J. Innov. Res. Sci. Eng. Technol. 4(5), 3530–3534 (2015)

    Google Scholar 

  34. Feng, X., Li, S., Yuan, C., Zeng, P., Sun, Y.: Prediction of slope stability using naive Bayes classifier. KSCE J. Civil Eng. 22(3), 941–950 (2018)

    Article  Google Scholar 

  35. European Network on Intelligent Technologies for Smart Adaptive Systems. Available at: http://www.eunite.org/. The competition page is: http://neuron.tuke.sk/competition/

  36. Zhang, P., Wu, X., Wang, X., Bi, S.: Short-term load forecasting based on big data technologies. CSEE J. Power Energy Syst. 1(3), 59–67 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computers and Systems Department, Faculty of Engineering, Mansoura University, Po. Box: 35516, Mansoura, Egypt

    Asmaa H. Rabie, Ahmed I. Saleh & Hesham A. Ali

  2. Communications & Electronics Engineering Department, Delta Higher Institute for Engineering & Technology, Mansoura, Egypt

    Shereen H. Ali

Authors
  1. Asmaa H. Rabie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shereen H. Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed I. Saleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hesham A. Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Asmaa H. Rabie.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rabie, A.H., Ali, S.H., Saleh, A.I. et al. A new outlier rejection methodology for supporting load forecasting in smart grids based on big data. Cluster Comput 23, 509–535 (2020). https://doi.org/10.1007/s10586-019-02942-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-019-02942-0

Keywords

Search

Navigation