Skip to main content

Advertisement

Springer Nature Link
Log in

The big data system, components, tools, and technologies: a survey

  • Survey Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets. Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches. However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data. This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data. In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process. Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models. The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML. Further, we classify analytics based on the type of data, domain, and application. We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment. Furthermore, we systematically investigate big data tools and technologies (Hadoop 3.0, Spark 2.3) including distributed/cloud-based stream processing tools in a comparative approach. Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters. Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data. Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. Institute of Electrical and Electronics Engineers.

  2. https://db-engines.com/en/system/Terrastore.

  3. http://couchdb.apache.org/.

  4. http://infogrid.org/trac/.

  5. https://neo4j.com/.

  6. https://spark.apache.org/.

  7. https://flink.apache.org/.

  8. https://www.h2o.ai/.

  9. https://azure.microsoft.com/en-in/.

  10. http://scikit-learn.org/stable/documentation.html.

  11. https://www.netflix.co.

  12. http://www.ebay.com/.

  13. https://azure.microsoft.com/en-in/solutions/data-lake/.

  14. https://www.alibabacloud.com/product/oss.

  15. https://samoa.incubator.apache.org/.

  16. https://zeppelin.apache.org/.

  17. https://aws.amazon.com/ec2/.

  18. https://aws.amazon.com/kinesis/data-firehose/.

  19. https://www.ibm.com/watson/.

  20. https://aws.amazon.com/docker/.

References

  1. The size of the world wide web (the internet). http://worldwidewebsize.com/

  2. Mattmann CA (2013) Computing: a vision for data science. Nature 493(7433):473–475

    Article  Google Scholar 

  3. National Aeronautics and Space Administration. https://www.nasa.gov/

  4. Clavin W (2013) Managing the deluge of ‘big data’ from space. NASA Jet Propulsion Labratory

  5. Atzori L, Iera A, Morabito G (2010) The internet of things: a survey. Comput Netw 54(15):2787–2805

    Article  MATH  Google Scholar 

  6. SCB Intelligence (2008) Six technologies with potential impacts on us interests out to 2025. National Intelligent Concil, Tech. Rep

  7. Yu S, Liu M, Dou W, Liu X, Zhou S (2017) Networking for big data: a survey. IEEE Commun Surv Tutor 19(1):531–549

    Article  Google Scholar 

  8. Pouyanfar S, Yang Y, Chen S-C, Shyu M-L, Iyengar SS (2018) Multimedia big data analytics: a survey. ACM Comput Surv 51(1):10

    Article  Google Scholar 

  9. Alaba FA, Othman M, Hashem IAT, Alotaibi F (2017) Internet of things security: a survey. J Netw Comput Appl 88:10–28

    Article  Google Scholar 

  10. Zikopoulos P, Eaton C, et al (2011) Understanding big data: analytics for enterprise class hadoop and streaming data. ISBN: 0071790535

  11. Chen M, Mao S, Liu Y (2014) Big data: a survey. Mob Netw Appl 19(2):171–209

    Article  Google Scholar 

  12. Hashem IAT, Yaqoob I, Anuar NB, Mokhtar S, Gani A, Khan SU (2015) The rise of big data on cloud computing: review and open research issues. Inf Syst 47:98–115

    Article  Google Scholar 

  13. Ma C, Zhang HH, Wang X (2014) Machine learning for big data analytics in plants. Trends Plant Sci 19(12):798–808

    Article  Google Scholar 

  14. Laney D (2013) 3d data management: controlling data volume, velocity and variety. META Group Research Note 6(70), 1

  15. Fan W, Bifet A (2013) Mining big data: current status, and forecast to the future. ACM sIGKDD Explor Newsl 14(2):1–5

    Article  Google Scholar 

  16. Demchenko Y, De Laat C, Membrey P (2014) Defining architecture components of the big data ecosystem. In: Collaboration technologies and systems (CTS), 2014 international conference on, pp 104–112

  17. Fernández A, del Río S, López V, Bawakid A, del Jesus MJ, Benítez JM, Herrera F (2014) Big data with cloud computing: an insight on the computing environment, mapreduce, and programming frameworks. Wiley Interdiscip Rev: Data Min Knowl Discov 4(5):380–409

    Google Scholar 

  18. Assunção MD, Calheiros RN, Bianchi S, Netto MAS, Buyya R (2015) Big data computing and clouds: trends and future directions. J Parallel Distrib Comput 79:3–15

    Article  Google Scholar 

  19. Emani CK, Cullot N, Nicolle C (2015) Understandable big data: a survey. Comput Sci Rev 17:70–81

    Article  MathSciNet  Google Scholar 

  20. Schuelke-Leech B-A, Barry B, Muratori M, Yurkovich BJ (2015) Big data issues and opportunities for electric utilities. Renew Sustain Energy Rev 52:937–947

    Article  Google Scholar 

  21. O’Leary DE (2015) Big data and privacy: emerging issues. IEEE Intell Syst 30(6):92–96

    Article  Google Scholar 

  22. Kune R, Konugurthi PK, Agarwal A, Chillarige RR, Buyya R (2016) The anatomy of big data computing. Softw: Pract Exp 46(1):79–105

    Google Scholar 

  23. Bello-Orgaz G, Jung JJ, Camacho D (2016) Social big data: recent achievements and new challenges. Inf Fusion 28:45–59

    Article  Google Scholar 

  24. Bajaber F, Elshawi R, Batarfi O, Altalhi A, Barnawi A, Sakr S (2016) Big data 2.0 processing systems: taxonomy and open challenges. J Grid Comput 14(3):379–405

    Article  Google Scholar 

  25. Nadal S, Herrero V, Romero O, Abell A, Franch X, Vansummeren S, Valerio D (2017) A software reference architecture for semantic-aware big data systems. Inf Softw Technol 90:75–92

    Article  Google Scholar 

  26. Big data and veracity challenges. https://www.isical.ac.in/~acmsc/TMW2014/LVS.pdf

  27. Gandomi A, Haider M (2015) Beyond the hype: big data concepts, methods, and analytics. Int J Inf Manag 35(2):137–144

    Article  Google Scholar 

  28. Lee I (2017) Big data: dimensions, evolution, impacts, and challenges. Bus Horiz 60(3):293–303

    Article  Google Scholar 

  29. Kung S-Y (2015) Visualization of big data. In: Cognitive informatics and cognitive computing (ICCI* CC), 2015 IEEE 14th international conference on, pp 447–448

  30. Strohbach M, Ziekow H, Gazis V, Akiva N (2015) Towards a big data analytics framework for IoT and smart city applications. In: Modeling and processing for next-generation big-data technologies. pp 257–282. ISBN: 14-9783319385006

  31. Wu X, Zhu X, Wu G-Q, Ding W (2014) Data mining with big data. IEEE Trans Knowl Data Eng 26(1):97–107

    Article  Google Scholar 

  32. Wu X, Chen H, Wu G, Liu J, Zheng Q, He X, Zhou A, Zhao Z-Q, Wei B, Ming G (2015) Knowledge engineering with big data. IEEE Intell Syst 30(5):46–55

    Article  Google Scholar 

  33. Wu X, Chen H, Liu J, Gongqing W, Ruqian L, Zheng N (2017) Knowledge engineering with big data (bigke): a 54-month, 45-million rmb, 15-institution national grand project. IEEE Access 5:12696–12701

    Article  Google Scholar 

  34. Venner J, Wadkar S, Siddalingaiah M (2014) Pro apache hadoop. ISBN-13: 9781430248637

  35. Pavlo A, Paulson E, Rasin A, Abadi DJ, DeWitt DJ, Madden S, Stonebraker M (2009) A comparison of approaches to large-scale data analysis. In: Proceedings of the 2009 ACM SIGMOD international conference on management of data, pp 165–178

  36. Teradata. http://www.teradata.com/Press-Releases/2016/Teradata-Announces-the-World%E2%80%99s-Most-Powerful

  37. Chang L, Wang Z, Ma T, Jian L, Ma L, Goldshuv A, Lonergan L, Cohen J, Welton C, Sherry G et al (2014) HAWQ: a massively parallel processing SQL engine in hadoop. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data, pp 1223–1234

  38. Greenplum architecture. http://greenplum.org/gpdb-sandbox-tutorials/ introduction-greenplum-database-architecture/

  39. Ibm netezza. https://www-01.ibm.com/software/data/netezza/

  40. Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Article  Google Scholar 

  41. Valiant LG (1990) A bridging model for parallel computation. Commun ACM 33(8):103–111

    Article  Google Scholar 

  42. Lenharth A, Nguyen D, Pingali K (2016) Parallel graph analytics. Commun ACM 59(5):78–87

    Article  Google Scholar 

  43. Apache hama project. https://hama.apache.org/

  44. Malewicz G, Austern MH, Bik AJC, Dehnert JC, Horn I, Leiser N, Czajkowski G (2010) Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD international conference on management of data, pp 135–146

  45. Apache giraph project. http://giraph.apache.org/

  46. Zhang H, Chen G, Ooi BC, Tan K-L, Zhang M (2015) In-memory big data management and processing: a survey. IEEE Trans Knowl Data Eng 27(7):1920–1948

    Article  Google Scholar 

  47. Cai Q, Zhang H, Guo W, Chen G, Ooi BC, Tan K-L, Wong WF (2018) Memepic: towards a unified in-memory big data management system. IEEE Trans Big Data

  48. Lim H, Han D, Andersen DG, Kaminsky M (2014) Mica: a holistic approach to fast in-memory key-value storage. USENIX, pp 429–444

  49. Kuznetsov SD, Poskonin AV (2014) Nosql data management systems. Program Comput Softw 40(6):323–332

    Article  Google Scholar 

  50. In-memory storage engine. https://docs.mongodb.com/manual/core/inmemory/

  51. Chen CLP, Zhang C-Y (2014) Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf Sci 275:314–347

    Article  Google Scholar 

  52. Mazón J-N, Lechtenbörger J, Trujillo J (2009) A survey on summarizability issues in multidimensional modeling. Data Knowl Eng 68(12):1452–1469

    Article  Google Scholar 

  53. Hu H, Wen Y, Chua T-S, Li X (2014) Toward scalable systems for big data analytics: a technology tutorial. IEEE Access 2:652–687

    Article  Google Scholar 

  54. Gantz J, Reinsel D (2011) Extracting value from chaos. IDC iview 1142:1–12

    Google Scholar 

  55. Kouzes RT, Anderson GA, Elbert ST, Gorton I, Gracio DK (2009) The changing paradigm of data-intensive computing. IEEE Comput 42(1):26–34

    Article  Google Scholar 

  56. Labrinidis A, Jagadish HV (2012) Challenges and opportunities with big data. Proc VLDB Endow 5(12):2032–2033

    Article  Google Scholar 

  57. UN Global Pulse (2012) Big data for development: challenges and opportunities. UN Global Pulse, New York

  58. Kambatla K, Kollias G, Kumar V, Grama A (2014) Trends in big data analytics. J Parallel Distrib Comput 74(7):2561–2573

    Article  Google Scholar 

  59. Chen Y, Qin X, Bian H, Chen J, Dong Z, Du X, Gao Y, Liu D, Lu J, Zhang H (2014) A study of SQL-on-hadoop systems. In: Workshop on big data benchmarks, performance optimization, and emerging hardware, pp 154–166

  60. Mohammed EA, Far BH, Naugler C (2014) Applications of the mapreduce programming framework to clinical big data analysis: current landscape and future trends. BioData Min 7(1):1

    Article  Google Scholar 

  61. Yang C, Huang Q, Li Z, Liu K, Hu F (2017) Big data and cloud computing: innovation opportunities and challenges. Int J Digit Earth 10(1):13–53

    Article  Google Scholar 

  62. Oussous A, Benjelloun F-Z, Lahcen AA, Belfkih S (2017) Big data technologies: a survey. J King Saud Univ-Comput Inf Sci

  63. Salloum S, Dautov R, Chen X, Peng PX, Huang JZ (2016) Big data analytics on apache spark. Int J Data Sci Anal, pp 1–20

  64. de Assuncao MD, da Silva Veith A, Buyya R (2018) Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. J Netw Comput Appl 103:1–17

    Article  Google Scholar 

  65. Krumm J, Davies N, Narayanaswami C (2008) User-generated content. IEEE Pervasive Comput 4(7):10–11

    Article  Google Scholar 

  66. White paper: How machine data supports gdpr compliance. https://www.splunk.com/pdfs/white-papers/splunk-how-machine-data-dupports-gdpr-compliance.pdf

  67. Shameer K, Badgeley MA, Miotto R, Glicksberg BS, Morgan JW, Dudley JT (2016) Translational bioinformatics in the era of real-time biomedical, health care and wellness data streams. Briefings in Bioinformatics, bbv118

  68. Marx V (2013) Biology: the big challenges of big data. Nature 498(7453):255–260

    Article  Google Scholar 

  69. Cook CE, Bergman MT, Cochrane G, Apweiler R, Birney E (2017) The european bioinformatics institute in 2017: data coordination and integration. Nucleic Acids Res 46(D1):D21–D29

    Article  Google Scholar 

  70. Akter S, Wamba SF (2016) Big data analytics in e-commerce: a systematic review and agenda for future research. Electron Mark 26(2):173–194

    Article  Google Scholar 

  71. Aws: streaming data. https://aws.amazon.com/streaming-data/

  72. Groenfeldt T, At nyse, the data deluge overwhelms traditional databases. https://www.forbes.com/sites/tomgroenfeldt/2013/02/14/at-nyse-the-data-deluge-overwhelms-traditional-databases/#25cda10f5aab

  73. Sun J, Reddy CK (2013) Big data analytics for healthcare. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 1525–1525

  74. Ranjan R, Georgakopoulos D, Wang L (2016) A note on software tools and technologies for delivering smart media-optimized big data applications in the cloud. Computing 98(1–2):1–5

    Article  MathSciNet  MATH  Google Scholar 

  75. Lloyd MD, Minor B. Harnessing the power of data in health. https://med.stanford.edu/content/dam/sm/sm-news/documents/StanfordMedicineHealthTrendsWhitePaper2017.pdf

  76. Twitter statistics and facts. https://www.statista.com/topics/737/twitter/

  77. Twitter by the numbers: stats, demographics and fun facts. https://www.omnicoreagency.com/twitter-statistics/

  78. Number of monthly active facebook users worldwide as of 4th quarter 2017. https://www.statista.com/statistics/264810/number-of-monthly-active-facebook-users-worldwide/

  79. Rob Kitchin (2017) Big data. The International Encyclopedia of Geography

  80. Gudivada VN, Baeza-Yates RA, Raghavan VV (2017) Big data: promises and problems. IEEE Comput 48(3):20–23

    Article  Google Scholar 

  81. Al-Fuqaha A, Guizani M, Mohammadi M, Aledhari M, Ayyash M (2015) Internet of things: a survey on enabling technologies, protocols, and applications. IEEE Commun Surv Tutor 17(4):2347–2376

    Article  Google Scholar 

  82. Raun J, Ahas R, Tiru M (2016) Measuring tourism destinations using mobile tracking data. Tour Manag 57:202–212

    Article  Google Scholar 

  83. Kitchin R (2014) The data revolution: Big data, open data, data infrastructures and their consequences. Sage, ISBN: 13-9781446287484

  84. Abiteboul S, Manolescu I, Rigaux P, Rousset M-C, Senellart P (2011) Web data management. Cambridge University Press, ISBN-13: 9781107012431

  85. Ghemawat S, Gobioff H, Leung S-T (2003) The google file system. In: ACM SIGOPS operating systems review, vol 37, pp 29–43

  86. Doctorow C (2008) Big data: welcome to the petacenre. Nat News 455(7209):16–21

    Article  Google Scholar 

  87. Ovsiannikov M, Rus S, Reeves D, Sutter P, Rao S, Kelly J (2013) The quantcast file system. Proc VLDB Endow 6(11):1092–1101

    Article  Google Scholar 

  88. Guerraoui R, Schiper A (1996) Fault-tolerance by replication in distributed systems. In: International conference on reliable software technologies, pp 38–57

  89. Wiesmann M, Pedone F, Schiper A, Kemme B, Alonso G (2000) Understanding replication in databases and distributed systems. In: Distributed computing systems, 2000. Proceedings of 20th international conference on, pp 464–474

  90. Shvachko K, Kuang H, Radia S, Chansler R (2010) The hadoop distributed file system. In: 2010 IEEE 26th symposium on mass storage systems and technologies (MSST), pp 1–10

  91. Hdfs architecture. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html

  92. Schmuck FB, Haskin RL (2002) Gpfs: a shared-disk file system for large computing clusters. In: FAST, vol 2, pp 231–244

  93. Jones T, Koniges AE, Yates RK (2000) Performance of the IBM general parallel file system. In: IPDPS, pp 673–681

  94. Limitations: The IBM SONAS system. https://www.ibm.com/support/knowledgecenter/en/STAV45/com.ibm.sonas.doc/adm_limitations.h

  95. Thanh TD, Mohan S, Choi E, Kim SB, Kim P (2008) A taxonomy and survey on distributed file systems. In: Networked computing and advanced information management, 2008. NCM’08. Fourth international conference on 1, pp 144–149

  96. Beaver D, Kumar S, Li HC, Sobel J, Vajgel P (2010) Finding a needle in haystack: facebook’s photo storage. OSDI 10:1–8

    Google Scholar 

  97. Fetterly D, Haridasan M, Isard M, Sundararaman S (2011) Tidyfs: a simple and small distributed file system. In: USENIX annual technical conference, pp 34–34

  98. Quantcast file system. https://www.quantcast.com/wp-content/uploads/2012/09/QC-QFS-One-Pager2.pdf

  99. Mapr file system. https://maprdocs.mapr.com/52/MapROverview/c_maprfs.html

  100. Brewer E (2010) A certain freedom: thoughts on the cap theorem. In: Proceedings of the 29th ACM SIGACT-SIGOPS symposium on principles of distributed computing, pp 335–335

  101. Lourenço JR, Cabral B, Carreiro P, Vieira M, Bernardino J (2015) Choosing the right nosql database for the job: a quality attribute evaluation. J Big Data 2(1):1–26

    Article  Google Scholar 

  102. Buyya R, Calheiros RN, Dastjerdi AV (2016) Big data: principles and paradigms. Morgan Kaufmann, ISBN-13: 9780128053942

  103. Abadi D, Boncz P, Harizopoulos S, Idreos S, Madden S et al (2013) The design and implementation of modern column-oriented database systems. Now 5(3):197–280

    Google Scholar 

  104. Matei G, Bank RC (2010) Column-oriented databases, an alternative for analytical environment. Database Syst J 1(2):3–16

  105. Floratou A, Patel JM, Shekita EJ, Tata S (2011) Column-oriented storage techniques for mapreduce. Proc VLDB Endow 4(7):419–429

    Article  Google Scholar 

  106. Chang F, Dean J, Ghemawat S, Hsieh WC, Wallach DA, Burrows M, Chandra T, Fikes A, Gruber RE (2008) Bigtable: a distributed storage system for structured data. ACM Trans Comput Syst 26(2):1–26

    Article  Google Scholar 

  107. Lakshman A, Malik P (2010) Cassandra: a decentralized structured storage system. ACM SIGOPS Oper Syst Rev 44(2):35–40

    Article  Google Scholar 

  108. Stonebraker M, Abadi DJ, Batkin A, Chen X, Cherniack M, Ferreira M, Lau E, Lin A, Madden S, O’Neil E et al. (2005) C-store: a column-oriented DBMS. In: Proceedings of the 31st international conference on very large data bases, pp 553–564

  109. Boncz PA, Zukowski M, Nes N (2005) Monetdb/x100: hyper-pipelining query execution. CIDR 5:225–237

    Google Scholar 

  110. Idreos S, Groffen F, Nes N, Manegold S, Mullender S, Kersten M (2012) Monetdb: two decades of research in column-oriented database architectures. Bull IEEE Comput Soc Tech Comm Data Eng 35(1):40–45

    Google Scholar 

  111. Sciore E (2007) Simpledb: a simple java-based multiuser syst for teaching database internals. ACM SIGCSE Bull 39(1):561–565

    Article  Google Scholar 

  112. Zukowski M, Boncz P (2012) Vectorwise: beyond column stores. IEEE Data Eng Bull 35(1):21–27

    Google Scholar 

  113. Edward SG, Sabharwal N (2015) Mongodb limitations. In: Practical MongoDB, pp 227–232

  114. Ravendb project. https://ravendb.net/docs/article-page/3.0/csharp

  115. Cross datacenter replication. http://docs.couchbase.com/admin/admin/XDCR/xdcr-intro.html

  116. DeCandia G, Hastorun D, Jampani M, Kakulapati G, Lakshman A, Pilchin A, Sivasubramanian S, Vosshall P, Vogels W (2007) Dynamo: amazon’s highly available key-value store. ACM SIGOPS Oper Syst Rev 41(6):205–220

    Article  Google Scholar 

  117. Basho products-riak products. http://basho.com/products/

  118. Sumbaly R, Kreps J, Gao L, Feinberg A, Soman C, Shah S (2012) Serving large-scale batch computed data with project voldemort. In: Proceedings of the 10th USENIX conference on file and storage technologies, pp 18–18

  119. Gudivada VN, Rao D, Raghavan VV (2014) NoSQL systems for big data management. In: 2014 IEEE World congress on services, pp 190–197

  120. Allegrograph. https://franz.com/agraph/allegrograph/

  121. Hypergraphdb. http://www.hypergraphdb.org/

  122. Infinitegraph. http://www.objectivity.com/products/infinitegraph/

  123. Moniruzzaman ABM, Hossain SA (2013) Nosql database: new era of databases for big data analytics-classification, characteristics and comparison. arXiv preprint arXiv:1307.0191

  124. Apache hbase reference guide. https://hbase.apache.org/apache_hbase_reference_guide.pdf

  125. Transparent data encryption. http://docs.datastax.com/en/archived/datastax_enterprise/4.0/datastax_enterprise/sec/secTDE.html

  126. Khetrapal A, Ganesh V (2006) Hbase and hypertable for large scale distributed storage systems. Dept. of Computer Science, Purdue University, pp 22–28

  127. Apache accumulo project. https://accumulo.apache.org/

  128. Ghaffari Amir, Chechina Natalia, Trinder Phil, Meredith Jon (2013) Scalable persistent storage for Erlang: theory and practice. In: Proceedings of the twelfth ACM SIGPLAN workshop on Erlang, pp 73–74

  129. Vogels W (2009) Eventually consistent. Commun ACM 52(1):40–44

    Article  Google Scholar 

  130. Apache hbase project. https://blogs.apache.org/hbase/entry/hbase_cell_security

  131. Mongodb mannual. https://docs.mongodb.org/manual/core/security-encryption-at-rest

  132. Redis project. https://redis.io/

  133. Random notes on improving the Redis LRU algorithm. http://antirez.com/news/109

  134. Redis4.0. https://redislabs.com/blog/redis-4-0-0-released/

  135. Redis cluster specification. https://redis.io/topics/cluster-spec

  136. In-memory storage engine. http://learnmongodbthehardway.com/schema/wiredtiger/

  137. The apache mahout project. https://mahout.apache.org/

  138. Spark 2.3-mllib guide. https://spark.apache.org/releases/spark-release-2-3-0.html#mllib

  139. Flinkml: Machine learning for flink. https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/ml/

  140. Mllib guide. https://spark.apache.org/docs/1.6.2/mllib-guide.html

  141. Meng X, Bradley J, Yuvaz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S et al (2016) Mllib: Machine learning in apache spark. JMLR 17(34):1–7

    MathSciNet  MATH  Google Scholar 

  142. Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, Meng X, Rosen J, Venkataraman S, Franklin MJ (2016) Apache spark: a unified engine for big data processing. Commun ACM 59(11):56–65

    Article  Google Scholar 

  143. Machine learning library (mllib) guide. https://spark.apache.org/docs/latest/ml-guide.html

  144. Different default regparam values in als. https://issues.apache.org/jira/browse/SPARK-19787

  145. Spark 2.3, mllib guide. https://spark.apache.org/docs/2.3.0/ml-guide.html

  146. Carbone P, Ewen S, Haridi S, Katsifodimos A, Markl V, Tzoumas K (2015) Apache flink: stream and batch processing in a single engine. Data Eng 38:28–38

    Google Scholar 

  147. Introducing Neo4j Bloom: Graph Data Visualization for Everyone. https://neo4j.com/blog/introducing-neo4j-bloom-graph-data-visualization-for-everyone/

  148. Orange documentation https://orange.biolab.si/docs/

  149. Raghavan UN, Réka A, Kumara S (2007) Near linear time algorithm to detect community structures in large-scale networks. Phys Rev E 76(3):036106

    Article  Google Scholar 

  150. Chappell D (2015) Introducing azure machine learning. A guide for technical professionals, sponsored by microsoft corporation

  151. Overview diagram of azure machine learning studio capabilities. https://docs.microsoft.com/en-in/azure/machine-learning/studio/studio-overview-diagram

  152. Azure capabilities, limitations and support. https://docs.microsoft.com/en-us/azure/machine-learning/studio/faq

  153. Ibm cloud/machine learning. https://console.bluemix.net/docs/services/PredictiveModeling/index.html#WMLgettingstarted

  154. Amazon machine learning. https://aws.amazon.com/aml/

  155. Amazon sagemaker features. https://aws.amazon.com/sagemaker/features/

  156. Netflix’s recommendation ml pipeline using apache spark. https://www.dbtsai.com/assets/pdf/2017-netflixs-recommendation-ml-pipeline-using-apache-spark.pdf

  157. Role of spark in transforming ebay’s enterprise data platform. https://databricks.com/session/role-of-spark-in-transforming-ebays-enterprise-data-platform

  158. Number of full-time employees at alibaba from 2012 to 2017. https://www.statista.com/statistics/226794/number-of-employees-at-alibabacom/

  159. Number of active consumers across alibaba’s online shopping. https://www.statista.com/statistics/226927/alibaba-cumulative-active-online-buyers-taobao-tmall/

  160. Huang L, Hu G, Lu X (2009) E-business ecosystem and its evolutionary path: the case of the alibaba group in china. Pacific Asia J Assoc Inf Syst 1(4)

  161. A year of blink at alibaba: apache flink in large scale production. http://www.dataversity.net/year-blink-alibaba/

  162. Gupta P, Sharma A, Jindal R (2016) Scalable machine-learning algorithms for big data analytics: a comprehensive review. Wiley Interdiscip Rev: Data Min Knowl Discov 6(6):194–214

    Google Scholar 

  163. Alibaba Blink: Real-time computing for big-time gains. https://medium.com/@alitech_2017/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26

  164. Ji X, Chun SA, Cappellari P, Geller J (2017) Linking and using social media data for enhancing public health analytics. J Inf Sci 43(2):221–245

    Article  Google Scholar 

  165. Kanaujia PKM, Pandey M, Rautaray SS (2017) Real time financial analysis using big data technologies. In: I-SMAC (IoT in social, mobile, analytics and cloud)(I-SMAC), 2017 international conference on, pp 131–136

  166. Moe WW, Schweidel DA (2017) Opportunities for innovation in social media analytics. J Prod Innov Manag 34(5):697–702

    Article  Google Scholar 

  167. Psyllidis A, Bozzon A, Bocconi S, Bolivar CT (2015) A platform for urban analytics and semantic data integration in city planning. In: International conference on computer-aided architectural design futures, pp 21–36

  168. Gust G, Flath C, Brandt T, Ströhle P, Neumann D (2016) Bringing analytics into practice: evidence from the power sector

  169. Nguyen D, Lenharth A, Pingali K (2013) A lightweight infrastructure for graph analytics. In: Proceedings of the twenty-fourth ACM symposium on operating systems principles, pp 456–471

  170. Baesens B, Van Vlasselaer V, Verbeke W (2015) Fraud analytics: a broader perspective. Fraud analytics using descriptive, predictive, and social network techniques: a guide to data science for fraud detection, pp 313–346

  171. Xu Z, Mei L, Chuanping H, Liu Y (2016) The big data analytics and applications of the surveillance system using video structured description technology. Cluster Comput 19(3):1283–1292

    Article  Google Scholar 

  172. Bisias D, Flood M, Lo AW, Valavanis S (2012) A survey of systemic risk analytics. Annu Rev Financ Econ 4(1):255–296

    Article  Google Scholar 

  173. Sagiroglu S, Sinanc D (2013) Big data: a review. In: Collaboration technologies and systems (CTS), 2013 international conference on, pp 42–47

  174. Rabkin A, Arye M, Sen S, Pai VS, Freedman MJ (2014) Aggregation and degradation in JetStream: streaming analytics in the wide area. In: NSDI vol 14, 275–288

  175. Zhang L, Stoffel A, Behrisch M, Mittelstadt S, Schreck T, Pompl R, Weber S, Last H, Keim D (2012) Visual analytics for the big data era comparative review of state-of-the-art commercial systems. In: Visual analytics science and technology (VAST), 2012 IEEE conference on, pp 173–182

  176. Waller MA, Fawcett SE (2013) Data science, predictive analytics, and big data: a revolution that will transform supply chain design and management. J Bus Logist 34(2):77–84

    Article  Google Scholar 

  177. Chen H, Chiang RHL, Storey VC (2012) Business intelligence and analytics: from big data to big impact. MIS Q 36(4):1165–1188

    Article  Google Scholar 

  178. Raghupathi W, Raghupathi V (2013) An overview of health analytics. J Health Med Inform 4(3):1–11

    Google Scholar 

  179. Cook DJ, Holder LB (2006) Mining graph data. Wiley, London

    Book  MATH  Google Scholar 

  180. Fortunato S (2010) Community detection in graphs. Phys Rep 486(3):75–174

    Article  MathSciNet  Google Scholar 

  181. Xin RS, Gonzalez JE, Franklin MJ, Stoica I (2013) Graphx: a resilient distributed graph system on spark. In: First international workshop on graph data management experiences and systems 2(1–2):6

  182. Low Y, Gonzalez J, Kyrola A, Bickson D, Guestrin C (2011) Graphlab: A distributed framework for machine learning in the cloud. arXiv preprint arXiv:1107.0922

  183. Introducing gelly: Graph processing with apache flink. https://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html

  184. Liu B (2007) Web data mining: exploring hyperlinks, contents, and usage data. Springer, Berlin. ISBN-13: 9783642194597

  185. Wesley R, Eldridge M, Terlecki PT (2011) An analytic data engine for visualization in tableau. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data, pp 1185–1194

  186. García M, Harmsen B (2012) Qlikview 11 for developers. Packt Publishing Ltd

  187. JMP https://www.jmp.com/en_us/home.html

  188. Microstrategy enterprise analytics and mobility. http://www.microstrategy.com/us/capabilities/visualizations

  189. Tibco spotfire. http://spotfire.tibco.com/

  190. Abousalh-Neto NA, Kazgan S (2012) Big data exploration through visual analytics. In: Visual analytics science and technology (VAST), 2012 IEEE conference on, pp 285–286

  191. Sas. http://www.sas.com/en_in/home.html

  192. Advizor. http://www.advizorsolutions.com/

  193. Smoot ME, Ono K, Ruscheinski J, Wang P-L, Ideker T (2011) Cytoscape 2.8: new features for data integration and network visualization. Bioinformatics 27(3):431–432

    Article  Google Scholar 

  194. Batagelj V, Mrvar A (1998) Pajek-program for large network analysis. Connections 21(2):47–57

    MATH  Google Scholar 

  195. Smith MA, Shneiderman B, Milic-Frayling N, Mendes Rodrigues E, Barash V, Dunne C, Capone T, Perer A, Gleave E (2009) Analyzing (social media) networks with NodeXL. In: Proceedings of the fourth international conference on communities and technologies, pp 255–264

  196. Bastian M, Heymann S, Jacomy M et al (2009) Gephi: an open source software for exploring and manipulating networks. ICWSM 8:361–362

    Google Scholar 

  197. Csardi G, Nepusz T (2006) The igraph software package for complex network research. Int J Complex Syst 1695(5):1–9

    Google Scholar 

  198. Apache hadoop project. http://hadoop.apache.org

  199. Sakr S, Liu A, Fayoumi AG (2013) The family of mapreduce and large-scale data processing systems. ACM Comput Surv 46(1):11

    Article  Google Scholar 

  200. Lee K-H, Lee Y-J, Choi H, Chung YD, Moon B (2012) Parallel data processing with mapreduce: a survey. AcM sIGMoD Rec 40(4):11–20

    Article  Google Scholar 

  201. Chen Y, Kreulen J, Campbell M, Abrams C (2011) Analytics ecosystem transformation: a force for business model innovation. In: 2011 Annual SRII global conference, pp 11–20

  202. Venner J, Wadkar S, Siddalingaiah M (2014) Pro apache Hadoop. ISBN: 9781430248637

  203. Apache hadoop project. http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

  204. Hdfs high availability using the quorum journal manager. https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

  205. Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe Jason, Shah Hitesh, Seth Siddharth et al (2013) Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th annual symposium on cloud computing, pp 5:1–16

  206. HDFS Erasure Coding. http://hadoop.apache.org/docs/r3.0.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html

  207. Apache Hadoop 3.0.1. http://hadoop.apache.org/docs/r3.0.1/

  208. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. HotCloud 10:10–10

    Google Scholar 

  209. Marcu O-C, Costan A, Antoniu G, Pérez-Hernández MS (2016) Spark versus flink: understanding performance in big data analytics frameworks. In: Cluster computing (CLUSTER), 2016 IEEE international conference on, pp 433–442

  210. Kubernetes concepts. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

  211. Rensin DK (2015) Kubernetes-scheduling the future at cloud scale

  212. Thusoo A, Sarma JS, Jain N, Shao Z, Chakka P, Zhang N, Antony S, Liu H, Murthy R (2010) Hive-a petabyte scale data warehouse using hadoop. In: 2010 IEEE 26th international conference on data engineering (ICDE 2010), pp 996–1005

  213. Impala project. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html

  214. Armbrust M, Xin RS, Lian C, Huai Y, Liu D, Bradley JK, Meng X, Kaftan T, Franklin MJ, Ghodsi A, et al (2015) Spark SQL: relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 1383–1394

  215. Traverso M (2013) Presto: interacting with petabytes of data at facebook. Retrieved February 4:2014

  216. Hausenblas M, Nadeau J (2013) Apache drill: interactive ad-hoc analysis at scale. Big Data 1(2):100–104

    Article  Google Scholar 

  217. Apache kylin. http://kylin.apache.org/docs

  218. Ho L-Y, Li T-H, Wu J-J, Liu P (2013) Kylin: an efficient and scalable graph data processing system. In: Big data, 2013 IEEE international conference on, pp 193–198

  219. Lamb A, Fuller M, Varadarajan R, Tran N, Vandiver B, Doshi L, Bear C (2012) The vertica analytic database: C-store 7 years later. Proc VLDB Endow 5(12):1790–1801

    Article  Google Scholar 

  220. Chattopadhyay B, Lin L, Liu W, Mittal S, Aragonda P, Lychagina V, Kwon Y, Wong M (2011) Tenzing a SQL implementation on the mapreduce framework

  221. Floratou A, Minhas UF, Özcan F (2014) Sql-on-hadoop: full circle back to shared-nothing database architectures. Proc VLDB Endow 7(12):1295–1306

    Article  Google Scholar 

  222. Nasir MAU (2016) Fault tolerance for stream processing engines. arXiv preprint arXiv:1605.00928

  223. Apache storm. http://storm.apache.org/

  224. Apache storm. http://storm.apache.org/releases/current/Concepts.html

  225. van der Veen JS, van der Waaij B, Lazovik E, Wijbrandi W, Meijer RJ (2015) Dynamically scaling apache storm for the analysis of streaming data. In: Big data computing service and applications (BigDataService), 2015 IEEE first international conference on, pp 154–161

  226. Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel JM, Kulkarni S, Jackson J, Gade K, Fu M, Donham J et al (2014) Storm@ twitter. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data, pp 147–156

  227. Apache strom 1.2.1. http://storm.apache.org/releases/current/Fault-tolerance.html

  228. Storm 1.2.0. http://storm.apache.org/2018/02/15/storm120-released.html

  229. Samza documentation. https://samza.apache.org/learn/documentation/0.14/comparisons/spark-streaming.html

  230. Bockermann C (2014) A survey of the stream processing landscape. Lehrstuhl fork unstliche Intelligenz Technische Universit. at Dortmund

  231. Neumeyer L, Robbins B, Nair A, Kesari A (2010) S4: distributed stream computing platform. In: Data mining workshops (ICDMW), 2010 IEEE international conference on, pp 170–177

  232. Zaharia M, Das T, Li H, Shenker S, Stoica I (2012) Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. HotCloud 12:10–10

    Google Scholar 

  233. Zaharia M, Das T, Li H, Hunter T, Shenker S, Stoica I (2013) Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the twenty-fourth ACM symposium on operating systems principles, pp 423–438

  234. Spark streaming programming guide. https://spark.apache.org/docs/2.2.0/streaming-programming -guide.html#discretized-streams-dstreams

  235. Improved fault-tolerance and zero data loss in apache spark streaming. https://databricks.com/blog/2015/01/15/improved-driver-fault-tolerance-and-zero-data-loss-in-spark-streaming.html

  236. Apache spark 2.3. https://spark.apache.org/releases/spark-release-2-3-0.html

  237. Chandy KM, Lamport L (1985) Distributed snapshots: determining global states of distributed systems. ACM Trans Comput Syst 3(1):63–75

    Article  Google Scholar 

  238. Apache spark 2.3. https://databricks.com/blog/2018/02/28/introducing-apache-spark-2-3.html

  239. Alexandrov A, Bergmann R, Ewen S, Freytag J-C, Hueske F, Heise A, Kao O, Leich M, Leser U, Markl V (2014) The stratosphere platform for big data analytics. VLDB J 23(6):939–964

    Article  Google Scholar 

  240. Apache flink 1.4. https://ci.apache.org/projects/flink/flink-docs-release-1.4/concepts/runtime.html

  241. Flink checkpointing. https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/checkpointing.html

  242. Exactly-once processing in samza. https://cwiki.apache.org/confluence/display/SAMZA/SEP-10+Exactly-once+Processing+in+Samza

  243. De Morales GF, Bifet A (2015) Samoa: scalable advanced massive online analysis. J Mach Learn Res 16(1):149–153

    Google Scholar 

  244. Samoa project. https://samoa.incubator.apache.org/documentation/SAMOA-Topology.html

  245. Apache samoa documentation. https://samoa.incubator.apache.org/documentation/Home.html

  246. Akidau T, Balikov A, Bekiroğlu K, Chernyak S, Haberman J, Lax R, McVeety S, Mills D, Nordstrom P, Whittle S (2013) Millwheel: fault-tolerant stream processing at internet scale. Proc VLDB Endow 6(11):1033–1044

    Article  Google Scholar 

  247. Kulkarni S, Bhagat N, Fu M, Kedigehalli V, Kellogg C, Mittal S, Patel JM, Ramasamy K, Taneja S (2015) Twitter heron: stream processing at scale. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 239–250

  248. Abadi D, Carney D, Cetintemel U, Cherniack M, Convey C, Erwin C, Galvez E, Hatoun M, Maskey A, Rasin A et al (2003) Aurora: a data stream management system. In: Proceedings of the 2003 ACM SIGMOD international conference on management of data, pp 666–666

  249. Heron project. https://twitter.github.io/heron/docs/concepts/architecture/#metrics-manager

  250. Structured streaming programming guide. https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

  251. Flink streaming. https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html

  252. Fu M, Agrawal A, Floratou A, Graham B, Jorgensen A, Li M, Lu N, Ramasamy K, Rao S, Wang C (2017) Twitter heron: towards extensible streaming engines. In: Data engineering (ICDE), 2017 IEEE 33rd international conference on, pp 1165–1172

  253. Amazon kinesis data streams. https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

  254. Azure stream analytics. https://docs.microsoft.com/en-us/azure/stream-analytics/ stream-analytics-introduction#how-does-stream-analytics-work

  255. Ibm streaming analytics. https://www.ibm.com/cloud/streaming-analytics

  256. Samza-storm. https://samza.apache.org/learn/documentation/0.7.0/comparisons/storm.html

  257. Apache storm 2.0. http://storm.apache.org/releases/2.0.0-SNAPSHOT/index.html

  258. Shukla A, Chaturvedi S, Simmhan Y (2017) Riotbench: a real-time iot benchmark for distributed stream processing platforms. arXiv preprint arXiv:1701.08530

  259. Dreissig F, Pollner N (2017) A data center infrastructure monitoring platform based on storm and trident. Datenbanksysteme für Business, Technologie und Web (BTW 2017)-Workshopband

  260. Saha B, Shah H, Seth S, Vijayaraghavan G, Murthy A, Curino C (2015) Apache tez: a unifying framework for modeling and building data processing applications. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 1357–1369

  261. Tpc-h is a decision support benchmark. http://www.tpc.org/

  262. Hortonworks data platform-apache hive performance tuning. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_hive-performance-tuning/bk_hive-performance-tuning.pdf

  263. Aws-containers. https://aws.amazon.com/what-are-containers/

  264. Apache mesos. http://mesos.apache.org/documentation/latest/

  265. Sebastio S, Ghosh R, Mukherjee T (2018) An availability analysis approach for deployment configurations of containers. IEEE Trans Serv Comput

  266. Medel V, Rana O, Bañares JÁ, Arronategui Unai (2016) Modelling performance and resource management in kubernetes. In: Utility and cloud computing (UCC), 2016 IEEE/ACM 9th international conference on, pp 257–262

  267. Hindman B, Konwinski A, Zaharia M, Ghodsi A, Joseph AD, Katz RH, Shenker S, Stoica I (2011) Mesos: a platform for fine-grained resource sharing in the data center. In: NSDI, vol 11, pp 295–308

  268. Amazon web services. https://aws.amazon.com/docker/

  269. Kreps J, Narkhede N, Rao J et al (2011) Kafka: a distributed messaging system for log processing. In: Proceedings of the NetDB, pp 1–7

  270. Rabbitmq. https://www.rabbitmq.com/

  271. Activemq. http://activemq.apache.org/

  272. AmazonmQ. https://aws.amazon.com/amazon-mq/

  273. Lampesberger H (2016) Technologies for web and cloud service interaction: a survey. Serv Oriented Comput Appl 10(2):71–110

    Article  Google Scholar 

  274. Dobbelaere P, Esmaili KS (2017) Kafka versus RabbitMQ. arXiv preprint arXiv:1709.00333

  275. Sangat P, Indrawan-Santiago M, Taniar D (2018) Sensor data management in the cloud: data storage, data ingestion, and data retrieval. Concurr Comput: Pract Exp 30(1)

  276. Hoffman S (2013) Apache flume: distributed log collection for hadoop. Packt Publishing Ltd

  277. Ting K, Cecho JJ (2013) Apache Sqoop Cookbook. O’Reilly Media, Inc

  278. Rabkin A, Katz RH (2010) Chukwa: a system for reliable large-scale log collection. LISA 10:1–15

    Google Scholar 

  279. Apach sqoop-overview. https://blogs.apache.org/sqoop/entry/apache_sqoop_overview

  280. Low Y, Gonzalez J, Kyrola A, Bickson D, Guestrin C, Hellerstein JM (2010) Graphlab: a new framework for parallel machine learning. arxiv preprint. arXiv preprint arXiv:1006.4990

  281. Aver C (2011) Giraph: large-scale graph processing infrastructure on hadoop. In: Proceedings of the Hadoop summit. Santa Clara 11(3), 5–9

  282. Gonzalez JE, Low Y, Haijie G, Bickson D, Guestrin C (2012) Powergraph: distributed graph-parallel computation on natural graphs. OSDI 12(1):2–2

    Google Scholar 

  283. Salihoglu S, Widom J (2013) Gps: a graph processing system. In: Proceedings of the 25th international conference on scientific and statistical database management 22, pp 1–12

  284. Gonzalez JE, Xin RS, Dave A, Crankshaw D, Franklin MJ, Stoica I (2014) Graphx: graph processing in a distributed dataflow framework. OSDI 14:599–613

    Google Scholar 

  285. Xin RS, Crankshaw D, Dave A, Gonzalez JE, Franklin MJ, Stoica I (2014) Graphx: unifying data-parallel and graph-parallel analytics. arXiv preprint arXiv:1402.2394

  286. Graphx programming guide. https://spark.apache.org/docs/latest/graphx-programming-guide.html

  287. Junghanns M, Petermann A, Gómez K, Rahm E (2015) Gradoop: scalable graph data management and analytics with hadoop. arXiv preprint arXiv:1506.00548

  288. Hunt P, Konar M, Junqueira FP, Reed B (2010) Zookeeper: Wait-free coordination for internet-scale systems. In: USENIX annual technical conference 8(9)

  289. Myriad home. https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home

  290. Apache avro. https://avro.apache.org/docs/current/

  291. Hu W, Qu Y (2008) Falcon-AO: a practical ontology matching system. Web Semant: Sci Serv Agents World Wide Web 6(3):237–239

    Article  Google Scholar 

  292. Apache nifi project. https://nifi.apache.org/

  293. Islam M, Huang AK, Battisha M, Chiang M, Srinivasan S, Peters C, Neumann A, Abdelnur A (2012) Oozie: towards a scalable workflow management system for hadoop. In: Proceedings of the 1st ACM SIGMOD workshop on scalable workflow execution engines and technologies 4:1–4:10

Download references

Author information

Authors and Affiliations

  1. Theoretical Computer Science Group, Department of Mathematics, Indian Institute of Technology Kharagpur, Kharagpur, 721302, India

    T. Ramalingeswara Rao & A. Goswami

  2. Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, 721302, India

    Pabitra Mitra

  3. Department of Computer Science and Engineering, Jaypee University of Information Technology, Waknaghat, 173234, India

    Ravindara Bhatt

Authors
  1. T. Ramalingeswara Rao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Pabitra Mitra
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ravindara Bhatt
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. A. Goswami
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to T. Ramalingeswara Rao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rao, T.R., Mitra, P., Bhatt, R. et al. The big data system, components, tools, and technologies: a survey. Knowl Inf Syst 60, 1165–1245 (2019). https://doi.org/10.1007/s10115-018-1248-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1248-0

Keywords