Skip to main content
SpringerLink
Log in

Evaluating the Boundaries of Big Data Environments for Machine Learning

  • Conference paper
  • First Online:
AI 2019: Advances in Artificial Intelligence (AI 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11919))

Included in the following conference series:

  • 2129 Accesses

Abstract

Hadoop and Spark are popular open-source Apache projects in the Big Data ecosystem. Due to shortcomings associated with Hadoop MapReduce (Hadoop) Apache Spark had gained prominence in the Big Data environment. However, there is little work aimed at evaluating these two Big Data frameworks to provide understanding for when they could be of most utility for machine learning, for example for when frequently querying large-scale data for input to recommendation systems. To explore the possible best use cases of each platform an experimental analysis between Hadoop and Spark was done and assessed using four criteria in terms of performance, storage, reliability and architecture. Different test environments were created varying memory, cache and volumes of data throughout the experiment, where Impala and Hive were used as query engines on the Hadoop file system against the native Spark query engine. We then conducted analyses along two dimensions. Our outcomes show that Spark performs best with large volumes of data processing compared with other query engines such as Apache Impala and Apache Hive. Findings here suggest that each platform have particular strengths given particular contexts, however, Spark seems to demonstrate most utility overall.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.forbes.com/sites/gilpress/2016/08/05/iot-mid-year-update-from-idc-and-other-research-firms. Retrieved 25 June 2019.

References

  1. Armbrust, M., et al.: Scaling spark in the real world: performance and usability. Proc. VLDB Endow. 8(12), 1840–1843 (2015)

    Article  Google Scholar 

  2. Bende, S., Shedge, R.: Dealing with small files problem in hadoop distributed file system. Procedia Comput. Sci. 79, 1001–1012 (2016). Proceedings International Conference on Communication, Computing and Virtualization (ICCCV) 2016

    Article  Google Scholar 

  3. Bu, Y., Borkar, V., Jia, J., Carey, M.J., Condie, T.: Pregelix: Big(Ger) graph analytics on a dataflow engine. Proc. VLDB Endow. 8(2), 161–172 (2014)

    Article  Google Scholar 

  4. Cai, Z., Gao, Z.J., Luo, S., Perez, L.L., Vagena, Z., Jermaine, C.: A comparison of platforms for implementing and running very large scale machine learning algorithms. In: Proceedings 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD’14, pp. 1371–1382. ACM, New York (2014)

    Google Scholar 

  5. Chang, B.R., Tsai, H., Wang, Y., Huang, C.: Resilient distributed computing platforms for big data analysis using Spark and Hadoop. In: Proceedings 2016 International Conference on Applied System Innovation (ICASI), pp. 1–4, May 2016

    Google Scholar 

  6. Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19(2), 171–209 (2014)

    Article  Google Scholar 

  7. Chodorow, K., Dirolf, M.: MongoDB - The Definitive Guide: Powerful and Scalable Data Storage. O’Reilly, Sebastopol (2010)

    Google Scholar 

  8. Dias, J., Ogasawara, E., de Oliveira, D., Porto, F., Valduriez, P., Mattoso, M.: Algebraic dataflows for big data analysis. In: Proceedings 2013 IEEE International Conference on Big Data, pp. 150–155. IEEE Press, October 2013

    Google Scholar 

  9. Floratou, A., Minhas, U.F., Özcan, F.: SQL-on-Hadoop: full circle back to shared-nothing database architectures. Proc. VLDB Endow. 7(12), 1295–1306 (2014)

    Article  Google Scholar 

  10. Frampton, M.: Mastering Apache Spark, 1st edn. Packt Publishing, Birmingham (2015)

    Google Scholar 

  11. Fuad, A., Erwin, A., Ipung, H.P.: Processing performance on apache pig, apache hive and MySQL cluster. In: Proceedings International Conference on Information, Communication Technology and System (ICTS) 2014, pp. 297–302. IEEE Press, September 2014

    Google Scholar 

  12. George, L.: Hbase: The Definitive Guide, 1st edn. O’Reilly Media Inc., Sebastopol (2011)

    Google Scholar 

  13. Gopalani, S., Arora, R.: Article: comparing apache spark and map reduce with performance analysis using K-Means. Int. J. Comput. Appl. 113(1), 8–11 (2015)

    Google Scholar 

  14. Jain, A.: Mastering Apache Storm, 1st edn. Packt Publishing, Birmingham (2017)

    Google Scholar 

  15. Kalavri, V.: Performance optimization techniques and tools for data-intensive computation platforms: an overview of performance limitations in big data systems and proposed optimizations (2014). qC 20140605

    Google Scholar 

  16. Khan, N., et al.: Big data: survey, technologies, opportunities, and challenges. Sci. World J. 2014, 1–18 (2014)

    Google Scholar 

  17. Kupisz, B., Unold, O.: Collaborative filtering recommendation algorithm based on Hadoop and Spark. In: Proceedings 2015 IEEE International Conference on Industrial Technology (ICIT), pp. 1510–1514. IEEE Press, March 2015

    Google Scholar 

  18. Landset, S., Khoshgoftaar, T.M., Richter, A.N., Hasanin, T.: A survey of open source tools for machine learning with big data in the Hadoop ecosystem. J. Big Data 2(1), 24 (2015)

    Article  Google Scholar 

  19. L’Heureux, A., Grolinger, K., Elyamany, H.F., Capretz, M.A.M.: Machine learning with big data: challenges and approaches. IEEE Access 5, 7776–7797 (2017)

    Article  Google Scholar 

  20. Ousterhout, K., Rasti, R., Ratnasamy, S., Shenker, S., Chun, B.G.: Making sense of performance in data analytics frameworks. In: Proceedings 12th USENIX Conference on Networked Systems Design and Implementation, NSDI’15, pp. 293–307. USENIX Association, Berkeley, CA, USA (2015)

    Google Scholar 

  21. Russell, J.: Getting Started with Impala, 1st edn. O’Reilly Media Inc., Sebastopol (2014)

    Google Scholar 

  22. Salloum, S., Dautov, R., Chen, X., Peng, P.X., Huang, J.Z.: Big data analytics on Apache Spark. Int. J. Data Sci. Anal. 1(3), 145–164 (2016)

    Article  Google Scholar 

  23. Shahrivari, S.: Beyond batch processing: towards real-time and streaming big data. CoRR abs/1403.3375 (2014). http://arxiv.org/abs/1403.3375

  24. Shi, J., et al.: Clash of the titans: MapReduce vs. spark for large scale data analytics. Proc. VLDB Endow. 8(13), 2110–2121 (2015)

    Article  Google Scholar 

  25. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop distributed file system. In: Proceedings 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST’10, pp. 1–10. IEEE Computer Society (2010)

    Google Scholar 

  26. Singh, R., Kaur, P.J.: Analyzing performance of Apache Tez and MapReduce with hadoop multinode cluster on Amazon cloud. J. Big Data 3(1), 19 (2016)

    Article  MathSciNet  Google Scholar 

  27. Su, T., Dy, J.: A deterministic method for initializing K-means clustering. In: Proceedings 16th IEEE International Conference on Tools with Artificial Intelligence, pp. 784–786. IEEE Press, November 2004

    Google Scholar 

  28. Taneja, R., Krishnamurthy, R., Liu, G.: Optimization of machine learning on Apache Spark. In: Proceedings 2016 International Conference on Parallel and Distributed Processing Techniques and Applications, pp. 163–167. CSREA Press (2016)

    Google Scholar 

  29. Thusoo, A., et al.: Hive - a warehousing solution over a map-reduce framework. Proc. VLDB Endow. 2, 1626–1629 (2009)

    Article  Google Scholar 

  30. Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings 9th USENIX Conference on Networked Systems Design and Implementation, NSDI’12, p. 2. USENIX Association, Berkeley, CA, USA (2012)

    Google Scholar 

  31. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10, p. 10. USENIX Association, Berkeley, CA, USA (2010)

    Google Scholar 

  32. Zhang, Y., Gao, Q., Gao, L., Wang, C.: iMapReduce: a distributed computing framework for iterative computation. J. Grid Comput. 10(1), 47–68 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Science, University of Otago, Dunedin, New Zealand

    Fathima Nuzla Ismail, Brendon J. Woodford & Sherlock A. Licorish

Authors
  1. Fathima Nuzla Ismail
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brendon J. Woodford
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sherlock A. Licorish
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brendon J. Woodford .

Editor information

Editors and Affiliations

  1. University of South Australia, Adelaide, SA, Australia

    Jixue Liu

  2. The University of Melbourne, Melbourne, VIC, Australia

    James Bailey

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ismail, F.N., Woodford, B.J., Licorish, S.A. (2019). Evaluating the Boundaries of Big Data Environments for Machine Learning. In: Liu, J., Bailey, J. (eds) AI 2019: Advances in Artificial Intelligence. AI 2019. Lecture Notes in Computer Science(), vol 11919. Springer, Cham. https://doi.org/10.1007/978-3-030-35288-2_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35288-2_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35287-5

  • Online ISBN: 978-3-030-35288-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation