Skip to main content
SpringerLink
Log in

Chapter 6 Big Data and FAIR Data for Data Science

  • Chapter
  • First Online:
Resilience in the Digital Age

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12660))

Abstract

The article is devoted to the review of such modern phenomena in the field of data storage and processing as Big Data and FAIR data. For Big Data, you will find an overview of the technologies used to work with them. And for FAIR data, their definition is given, and the current state of their development is described, including the Internet of FAIR Data & Services (IFDS).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lohr, S.: The Origins of ‘Big Data': An Etymological Detective Story. The New York Times (2013). https://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story/

  2. Snijders, C., Matzat, U., Reips, U.-D.: “Big Data”: big gaps of knowledge in the field of internet science. Int. J. Internet Sci. 7, 1–5 (2012)

    Google Scholar 

  3. Dedić, N., Stanier, C.: Towards differentiating business intelligence, big data, data analytics and knowledge discovery. In: Piazolo, F., Geist, V., Brehm, L., Schmidt, R. (eds.) ERP Future 2016. LNBIP, vol. 285, pp. 114–122. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58801-8_10

    Chapter  Google Scholar 

  4. Hashem, I.A.T., Yaqoob, I., Anuar, N.B., Mokhtar, S., Gani, A., Khan, S.U.: The rise of “big data” on cloud computing: review and open research issues. Inf. Syst. 47, 98–115 (2015). https://doi.org/10.1016/j.is.2014.07.006

    Article  Google Scholar 

  5. Grimes, S.: Big Data: Avoid ‘Wanna V’ Confusion. InformationWeek (2013). https://www.informationweek.com/big-data/big-data-analytics/big-data-avoid-wanna-v-confusion/d/d-id/1111077

  6. Fox, C.: Data Science for Transport. Springer Textbooks in Earth Sciences, Geography and Environment. Springer, Cham (2018). doi: https://doi.org/10.1007/978-3-319-72953-4

  7. Onay, C., Öztürk, E.: A review of credit scoring research in the age of Big Data. J. Financ. Regul. Compliance. 26, 382–405 (2018). https://doi.org/10.1108/JFRC-06-2017-0054

    Article  Google Scholar 

  8. Kitchin, R., McArdle, G.: What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets. Big Data Soc. 3(1), 2053951716631130 (2016). https://doi.org/10.1177/2053951716631130

    Article  Google Scholar 

  9. NIST Big Data Interoperability Framework, vol. 1, Definitions. Version 3. NIST Special Publication 1500–1r2 (2019). https://doi.org/10.6028/NIST.SP.1500-1r2

  10. Usha, D., Aps, J.A.: A survey of Big Data processing in perspective of Hadoop and MapReduce. Int. J. Curr. Eng. Technol. 4, 602–606 (2014)

    Google Scholar 

  11. White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., United States (2012)

    Google Scholar 

  12. Mall, N.N., Rana, S.: Overview of Big Data and Hadoop. Imperial J. Interdisc. Res. 2, 1399–1406 (2016)

    Google Scholar 

  13. Prasad, B.R., Agarwal, S.: Comparative study of Big Data computing and storage tools: a review. Int. J. Database Theory Appl. 9, 45–66 (2016)

    Article  Google Scholar 

  14. Dimiduk, N., Khurana, A., Ryan, M.H., Stack, M.: HBase in Action. Manning, Shelter Island (2013)

    Google Scholar 

  15. Hashem, I.A.T., Anuar, N.B., Gani, A., Yaqoob, I., Xia, F., Khan, S.U.: MapReduce: review and open challenges. Scientometrics 109(1), 389–422 (2016). https://doi.org/10.1007/s11192-016-1945-y

    Article  Google Scholar 

  16. Chen, X., Hu, L., Liu, L., Chang, J., Bone, D.L.: Breaking down Hadoop distributed file systems data analytics tools: apache Hive vs. Apache Pig vs. pivotal HWAQ. In: 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), pp. 794–797. IEEE (2017)

    Google Scholar 

  17. Swarna, C., Ansari, Z.: Apache Pig - a data flow framework based on Hadoop Map Reduce. Int. J. Eng. Trends Technol. 50, 271–275 (2017)

    Article  Google Scholar 

  18. Gates, A., Dai, D.: Programming Pig: Dataflow Scripting with Hadoop. O’Reilly Media Inc., United States (2016)

    Google Scholar 

  19. Singh, N., Agrawal, S.: A performance analysis of high-level MapReduce query languages in Big Data. In: Proceedings of the International Congress on Information and Communication Technology, pp. 551–558. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0767-5_57

  20. Camacho-Rodríguez, J., et al.: Apache Hive: From Mapreduce to enterprise-grade Big Data warehousing. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1773–1786 (2019)

    Google Scholar 

  21. Pen, H.D., Dsilva, P., Mascarnes, S.: Comparing HiveQL and MapReduce methods to process fact data in a data warehouse. In: 2017 2nd International Conference on Communication Systems, Computing and IT Applications (CSCITA), pp. 201–206. IEEE (2017)

    Google Scholar 

  22. Vohra, D.: Using apache sqoop. In: Pro Docker, pp. 151–183. Apress, Berkeley, CA (2016)

    Google Scholar 

  23. Lydia, E.L., Swarup, M.B.: Analysis of Big Data through Hadoop ecosystem components like flume mapreduce, pig and hive. Int. J. Comput. Sci. Eng. 5, 21–29 (2016)

    Google Scholar 

  24. Mehta, S., Mehta, V.: Hadoop ecosystem: an introduction. IJSR. 5, 557–562 (2016)

    Google Scholar 

  25. Jain, A.: Mastering Apache Storm: Real-time Big Data Streaming using Kafka. Packt Publishing Ltd., Hbase and Redis (2017)

    Google Scholar 

  26. Zaharia, M., et al.: Apache Spark: a unified engine for Big Data processing. Commun. ACM 59, 56–65 (2016)

    Article  Google Scholar 

  27. Jayaratne, M., Alahakoon, D., De Silva, D., Yu, X.: Apache Spark based distributed self-organizing map algorithm for sensor data analysis. In: IECON 2017–43rd Annual Conference of the IEEE Industrial Electronics Society, pp. 8343–8349. IEEE (2017)

    Google Scholar 

  28. Luu, H.: Beginning Apache Spark 2: With Resilient Distributed Datasets, Spark Sql, Structured Streaming, and Spark Machine Learning Library. Apress, Berkeley (2018)

    Book  Google Scholar 

  29. Vaddeman, B.: HCatalog. In: Beginning Apache Pig, pp. 103–113. Apress, Berkeley, CA (2016). https://doi.org/10.1007/978-1-4842-2337-6_7

    Chapter  Google Scholar 

  30. Lyubimov, D., Palumbo, A.: Apache Mahout: Beyond MapReduce. CreateSpace Independent Publishing Platform, United States (2016)

    Google Scholar 

  31. Schmidt, D., Chen, W.C., Matheson, M.A., Ostrouchov, G.: Programming with BIG data in R: scaling analytics from one to thousands of nodes. Big Data Res. 8, 1–1 (2017)

    Article  Google Scholar 

  32. Elshawi, R., Sakr, S., Talia, D., Trunfio, P.: Big data systems meet machine learning challenges: towards Big Data science as a service. Big data Res. 14, 1–1 (2018)

    Article  Google Scholar 

  33. Haloi, S.: Apache Zookeeper Essentials. Packt Publishing Ltd., United Kingdom (2015)

    Google Scholar 

  34. Vohra, D.: Apache Avro. In: Practical Hadoop Ecosystem, pp. 303–323. Apress, Berkeley, CA (2016). https://doi.org/10.1007/978-1-4842-2199-0_7

    Chapter  Google Scholar 

  35. Islam, M.K., Srinivasan, A.: Apache Oozie: The Workflow Scheduler for Hadoop. O’Reilly Media Inc., United States (2015)

    Google Scholar 

  36. Wadkar, S., Siddalingaiah, M.: Apache Ambari. In: Pro Apache Hadoop, pp. 399–401. Apress, Berkeley, CA (2014). https://doi.org/10.1007/978-1-4302-4864-4_20

    Chapter  Google Scholar 

  37. Saxena, A., Singh, S., Shakya, C.: Concepts of HBase archetypes in Big Data engineering. In: Roy, S.S., Samui, P., Deo, R., Ntalampiras, S. (eds.) Big Data in Engineering Applications. SBD, vol. 44, pp. 83–111. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-8476-8_5

    Chapter  Google Scholar 

  38. Sirisha, N., Kiran, K.V.D.: Stock exchange analysis using Hadoop user experience (Hue). In: 2017 International Conference on Intelligent Sustainable Systems (ICISS), pp. 1141–1144. IEEE (2017)

    Google Scholar 

  39. Ofli, F., et al.: Combining human computing and machine learning to make sense of big (aerial) data for disaster response. Big Data. 4, 47–59 (2016). https://doi.org/10.1089/big.2014.0064

    Article  Google Scholar 

  40. Chen, D., Liu, Z., Wang, L., Dou, M., Chen, J., Li, H.: Natural disaster monitoring with wireless sensor networks: a case study of data-intensive applications upon low-cost scalable systems. Mob. Netw. Appl. 18(5), 651–663 (2013). https://doi.org/10.1007/s11036-013-0456-9

    Article  Google Scholar 

  41. MacLachlan, C., et al.: Global seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Q. J. R. Meteorol. Soc. 141, 1072–1084 (2015). https://doi.org/10.1002/qj.2396

    Article  Google Scholar 

  42. Poblet, M., García-Cuesta, E., Casanovas, P.: Crowdsourcing tools for disaster management: a review of platforms and methods. In: Casanovas, P., Pagallo, U., Palmirani, M., Sartor, G. (eds.) AI Approaches to the Complexity of Legal Systems, pp. 261–274. Springer, Heidelberg (2014)

    Chapter  Google Scholar 

  43. Nativi, S., Mazzetti, P., Craglia, M.: A view-based model of data-cube to support big earth data systems interoperability. Big Earth Data. 1, 75–99 (2017). https://doi.org/10.1080/20964471.2017.1404232

    Article  Google Scholar 

  44. USGS Earth Explorer online portal. https://earthexplorer.usgs.gov/

  45. Copernicus Sentinel Hub. https://scihub.copernicus.eu/

  46. GEOSS portal. https://www.geoportal.org/

  47. Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, D., Moore, R.: GoogleEarth engine: planetary-scale geospatial analysis for everyone. Remote Sens. Environ. 202(Suppl C), 18–27 (2017). https://doi.org/10.1016/j.rse.2017.06.031

    Article  Google Scholar 

  48. Baumann, P., et al.: Big Data analytics for earth sciences: the earthserver approach. Int. J. Digit. Earth. 9, 3–29 (2016). https://doi.org/10.1080/17538947.2014.1003106

    Article  Google Scholar 

  49. Wilkinson, M., Dumontier, M., Aalbersberg, I., et al.: The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data. 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18

    Article  Google Scholar 

  50. GO FAIR initiative. https://www.go-fair.org/

Download references

Acknowledgments

This work was conducted in the framework of budgetary funding of the Geophysical Center of RAS, adopted by the Ministry of Science and Higher Education of the Russian Federation.

Author information

Authors and Affiliations

  1. Geophysical Center of the Russian Academy of Sciences, Molodezhnaya St. 3, 119296, Moscow, Russia

    Alexei Gvishiani, Michael Dobrovolsky & Alena Rybkina

  2. Schmidt Institute of Physics of the Earth of the Russian Academy of Sciences, Bolshaya Gruzinskaya Street 10-1, 123242, Moscow, Russia

    Alexei Gvishiani & Alena Rybkina

Authors
  1. Alexei Gvishiani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Dobrovolsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alena Rybkina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Dobrovolsky .

Editor information

Editors and Affiliations

  1. DIMACS Center, Rutgers University, Piscataway, NJ, USA

    Fred S. Roberts

  2. Russian Foundation for Basic Research, Moscow, Russia

    Igor A. Sheremet

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gvishiani, A., Dobrovolsky, M., Rybkina, A. (2021). Chapter 6 Big Data and FAIR Data for Data Science. In: Roberts, F.S., Sheremet, I.A. (eds) Resilience in the Digital Age. Lecture Notes in Computer Science(), vol 12660. Springer, Cham. https://doi.org/10.1007/978-3-030-70370-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-70370-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-70369-1

  • Online ISBN: 978-3-030-70370-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation