research-article

Free access

Big data and its technical challenges

Authors:

H. V. Jagadish,

Johannes Gehrke,

Alexandros Labrinidis,

Yannis Papakonstantinou,

Jignesh M. Patel,

Raghu Ramakrishnan,

Cyrus ShahabiAuthors Info & Claims

Communications of the ACM, Volume 57, Issue 7

Pages 86 - 94

https://doi.org/10.1145/2611567

Published: 01 July 2014 Publication History

All formats PDF

Abstract

Exploring the inherent technical challenges in realizing the potential of Big Data.

References

[1]

Computing Community Consortium. Advancing Discovery in Science and Engineering. Spring 2011.

Google Scholar

[2]

Computing Community Consortium. Advancing Personalized Education. Spring 2011.

Google Scholar

[3]

Computing Community Consortium. Smart Health and Wellbeing. Spring 2011.

Google Scholar

[4]

Computing Community Consortium. A Sustainable Future. Summer 2011.

Google Scholar

[5]

Computer Research Association. Challenges and Opportunities with Big Data. Community white paper available at http://cra.org/ccc/docs/init/bigdatawhitepaper.pdf

Google Scholar

[6]

Dobbie, W. and Fryer, Jr. R.G. Getting Beneath the Veil of Effective Schools: Evidence from New York City. NBER Working Paper No. 17632. Issued Dec. 2011.

Google Scholar

[7]

Economist. Drowning in numbers: Digital data will flood the planet---and help us understand it better. (Nov 18, 2011); http://www.economist.com/blogs/dailychart/2011/11/big-data-0

Google Scholar

[8]

Flood, M., Jagadish, H.V., Kyle, A., Olken, F. and Raschid, L. Using data for systemic financial risk management. In Proc. 5^th Biennial Conf. Innovative Data Systems Research (Jan. 2011).

Google Scholar

[9]

Forbes. Data-driven: Improving business and society through data. (Feb. 10, 2012); http://www.forbes.com/special-report/data-driven.html

Google Scholar

[10]

Gartner Group. Pattern-Based Strategy: Getting Value from Big Data. (July 2011 press release); http://www.gartner.com/it/page.jsp?id=1731916

Google Scholar

[11]

González, M.C., Hidalgo, C.A. and Barabási, A-L. Understanding individual human mobility patterns. Nature 453, (June 5, 2008), 779--782.

Crossref

Google Scholar

[12]

Hey, T., Tansley, S. and Tolle, K., eds. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, 2009.

Google Scholar

[13]

Kahn, S.D. On the future of genomic data. Science 331, 6018 (Feb. 11, 2011), 728--729.

Crossref

Google Scholar

[14]

Lazar, D. et al. Computational social science. Science 323, 5915 (Feb. 6, 2009), 721--723.

Google Scholar

[15]

Lohr, A. The age of Big Data. New York Times (Feb. 11, 2012); http://www.nytimes.com/2012/02/12/sunday-review/big-datas-impact-in-the-world.html

Google Scholar

[16]

Lohr, S. How Big Data became so big. New York Times (Aug. 11, 2012); http://www.nytimes.com/2012/08/12/business/how-big-data-became-so-big-unboxed.html

Google Scholar

[17]

Manyika, J. et al. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute. May 2011.

Google Scholar

[18]

National Science and Technology Council. Materials Genome Initiative for Global Competitiveness. June 2011.

Google Scholar

[19]

Noguchi, Y. Following the Breadcrumbs to Big Data Gold. National Public Radio (Nov. 29, 2011); http://www.npr.org/2011/11/29/142521910/the-digital-breadcrumbs-that-lead-to-big-data

Google Scholar

[20]

Noguchi, Y. The Search for Analysts to Make Sense of Big Data. National Public Radio, (Nov. 30, 2011); http://www.npr.org/2011/11/30/142893065/the-search-for-analysts-to-make-sense-of-big-data

Google Scholar

[21]

Pariser, E. The Filter Bubble: What the Internet Is Hiding From You. Penguin Press, May 2011.

Digital Library

Google Scholar

[22]

PCAST Report. Designing a Digital Future: Federally Funded Research and Development in Networking and Information Technology (Dec. 2010); http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf

Google Scholar

[23]

SDSS-III: Massive Spectroscopic Surveys of the Distant Universe, the Milky Way Galaxy, and Extra-Solar Planetary Systems (Jan. 2008); http://www.sdss3.org/collaboration/description.pdf/

Google Scholar

Cited By

View all

Anandaram HTaseen RYaseen HBhange AM. S. N(2025)Data Analytics and Visualization Tools and ApplicationsRevolutionizing Data Science and Analytics for Industry Transformation10.4018/979-8-3693-7868-7.ch009(225-242)Online publication date: 14-Feb-2025
https://doi.org/10.4018/979-8-3693-7868-7.ch009
Seethala SSeethala S(2025)AI-Powered Data Warehouse Modernization for Utilities: Achieving Real-Time Insights and Efficiency SSRN Electronic Journal10.2139/ssrn.5113231Online publication date: 2025
https://doi.org/10.2139/ssrn.5113231
Seethala S(2025)AI-Driven Data Warehousing for Financial Institutions: Future Proofing Against Market Volatility SSRN Electronic Journal10.2139/ssrn.5113169Online publication date: 2025
https://doi.org/10.2139/ssrn.5113169
Show More Cited By

Index Terms

Big data and its technical challenges
1. Information systems
  1. Data management systems
    1. Database management system engines
  2. Information systems applications

Recommendations

Challenges for MapReduce in Big Data
SERVICES '14: Proceedings of the 2014 IEEE World Congress on Services

In the Big Data community, MapReduce has been seen as one of the key enabling approaches for meeting continuously increasing demands on computing resources imposed by massive data sets. The reason for this is the high scalability of the MapReduce ...
From Big Data to Big Data Mining: Challenges, Issues, and Opportunities
Proceedings of the 18th International Conference on Database Systems for Advanced Applications - Volume 7827

While "big data" has become a highlighted buzzword since last year, "big data mining", i.e., mining from big data, has almost immediately followed up as an emerging, interrelated research area. This paper provides an overview of big data mining and ...
Hadoop Essentials - Tackling the Challenges of Big Data with Hadoop

Comments

Information & Contributors

Information

Published In

Communications of the ACM Volume 57, Issue 7

July 2014

98 pages

ISSN:0001-0782

EISSN:1557-7317

DOI:10.1145/2622628

Editor:
Moshe Y. Vardi
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2014

Published in CACM Volume 57, Issue 7

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Popular
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

780
Total Citations
View Citations
28,856
Total Downloads

Downloads (Last 12 months)2,647
Downloads (Last 6 weeks)280

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Anandaram HTaseen RYaseen HBhange AM. S. N(2025)Data Analytics and Visualization Tools and ApplicationsRevolutionizing Data Science and Analytics for Industry Transformation10.4018/979-8-3693-7868-7.ch009(225-242)Online publication date: 14-Feb-2025
https://doi.org/10.4018/979-8-3693-7868-7.ch009
Seethala SSeethala S(2025)AI-Powered Data Warehouse Modernization for Utilities: Achieving Real-Time Insights and Efficiency SSRN Electronic Journal10.2139/ssrn.5113231Online publication date: 2025
https://doi.org/10.2139/ssrn.5113231
Seethala S(2025)AI-Driven Data Warehousing for Financial Institutions: Future Proofing Against Market Volatility SSRN Electronic Journal10.2139/ssrn.5113169Online publication date: 2025
https://doi.org/10.2139/ssrn.5113169
Yoon AKim JDonaldson D(2025)Big data curation frameworkJournal of Information Science10.1177/0165551522113352851:1(205-223)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1177/01655515221133528
Pan ZCai FLiu XChen H(2025)Distance-Aware Learning for Inductive Link Prediction on Temporal NetworksIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.332892436:1(978-990)Online publication date: Jan-2025
https://doi.org/10.1109/TNNLS.2023.3328924
Chen LZhao QLi GZhou MDai CFeng YLiu XLi J(2025)A Sparse Cross Attention-Based Graph Convolution Network With Auxiliary Information Awareness for Traffic Flow PredictionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2025.353356026:3(3210-3222)Online publication date: Mar-2025
https://doi.org/10.1109/TITS.2025.3533560
Yuan QWang JHan YLiu ZLiu W(2025)DAGCAN: Decoupled Adaptive Graph Convolution Attention Network for Traffic ForecastingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2025.353166526:3(3513-3526)Online publication date: Mar-2025
https://doi.org/10.1109/TITS.2025.3531665
Ouyang NAo LCai QWan WRen XHe XSheng K(2025)Graph Transformer-Based Dynamic Edge Interaction Encoding for Traffic PredictionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.351332526:3(4066-4079)Online publication date: Mar-2025
https://doi.org/10.1109/TITS.2024.3513325
Lu KSong LXie Y(2025)Enhancing Fault Tolerance in Distributed Storage Systems through an Extended EVENODD CodeITM Web of Conferences10.1051/itmconf/2025730301773(03017)Online publication date: 17-Feb-2025
https://doi.org/10.1051/itmconf/20257303017
Fang HWang HGao YZhang YBu JHan BLin H(2025)InsGNN: Interpretable spatio-temporal graph neural networks via information bottleneckInformation Fusion10.1016/j.inffus.2025.102997119(102997)Online publication date: Jul-2025
https://doi.org/10.1016/j.inffus.2025.102997
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF Chinese translation

eReader

View online with eReader.

eReader

Digital Edition

View this article in digital edition.

Digital Edition

Magazine Site

View this article on the magazine site (external)

Magazine Site

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Challenges for MapReduce in Big Data

From Big Data to Big Data Mining: Challenges, Issues, and Opportunities

Hadoop Essentials - Tackling the Challenges of Big Data with Hadoop

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Digital Edition

Magazine Site

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations