Abstract
Organizations today are gathering and storing more and more data in the belief that it is necessary for compliance, legal reasons or that it may be necessary in the future. Most of this data is considered dark as it is unstructured, uncatalogued, unmanaged, and unanalyzed. Big data consists of structured data (business critical and redundant obsolete and trivial (ROT) data) and unstructured data being dark data. This dark data can be in data silos isolated to specific departments or sectors in an organization unable to be accessed and analyzed by other departments in the organization. Organizations waste time and operating budgets searching for this data and storing the data. Data management practices, policies and procedures need to be reviewed by organizations. The creation of a position solely to be responsible for the storage, curation, and general good health of data should be considered. Dark data can have inherent security risks for organizations that can damage reputations, harm revenue, and leave the organization vulnerable to cybersecurity threats and risks such as personal data breaches or stolen data. Data governance principles need to be established and implemented in all organizations. The three main components of data governance are people (roles, responsibilities, working groups and committees), processes, and tools and technology. This paper presents a brief review of the various aspects of dark data and their implications for organisations.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ackoff, R.L.: From data to wisdom. J. Appl. Syst. Anal. 16, 3–9 (1989)
Allen, G.D.: Hierarchy of knowledge – from data to wisdom. Int. J. Curr. Res. Multidisc. (IJCRM) 2(1), 15–23 (2017)
Turban, E., Rainer, R.K., Potter, R.E.: Introduction to Information Technology, 3rd edn. Wiley, New York (2005)
Snyder, H.: Literature review as a research methodology - An overview and guidelines (2019)
Braun, V., Clarke, V.: Using thematic analysis in psychology (2006)
Ashbel, A.: Dark data: a challenge enterprise data management can’t ignore. https://bluexp.netapp.com/blog/cds-blg-dark-data-a-challenge-enterprise-data-management-cant-ignore. Accessed 11 Nov 2022
Splunk: The state of dark data (2019). Accessed 11 Nov 2022
Goetz, T.: Freeing the dark data of failed scientific experiment. Wired Mag. 15(10), 7–12 (2007). http://www.wired.com/science/discoveries/magazine/15-10/st_essay. Accessed 11 Nov 2022
Grimm, D.J.: The dark data quandary. Am. Univ. Law Rev. 68(3), 768 (2019)
Martin, E.J.: Dark Data: Analyzing Unused and Ignored Information (2016)
Heidorn, P.B.: Shedding light on the dark data in the long tail of science. Libr. Trends 57(2), 280–299 (2008)
Heidorn, P.B., Stahlman, G.R., Steffen, J.: The astrolabe project: identifying and curating astronomical ‘dark data’ through development of cyberinfrastructure resources. Astrophys. J. Suppl. Ser. 236(1), 3 (2018). https://doi.org/10.1051/epjconf/201818603003,lastaccessed2022/11/11
Schembera, B., Durán, J.M.: Dark data as the new challenge for big data science and the introduction of the scientific data officer (2019)
Ajis, A.F.M., Zakaria, S., Ahmad, A.R.: Demystifying dark data characteristics in small and medium enterprises: a Malaysian experience (2022)
Gartner Inc.: Innovation Insight: File Analysis Innovation Delivers an Understanding of Unstructured Dark Data, Alan Dayley, March (2013)
Cadariu, S.: Dark Data at the Enterprise Level: What is it and What Risks Does it Pose? https://www.aitimejournal.com/dark-data-at-the-enterprise-level-what-is-is-and-what-risks-does-it-pose. Accessed 11 Nov 2022
Cadariu, S.: Data Fabric and Cloud Computing as Enterprise Technologies, https://www.aitimejournal.com/data-fabric-and-cloud-computing-as-enterprise-technologies, last accessed 2022/11/11
Veritas: The databerg report - see what others don’t (2015)
Dimitrov, W., Siarova, S., Petkova, L.: Types of dark data and hidden cybersecurity risks (2018)
Jackson, T.W., Hodgkinson, I.R.: Keeping a lower profile: how firms can reduce their digital carbon footprints (2022)
Intel: A vision for big data
Zikopoulos, P., Eaton, C., Dirk, D., Deutsch, T., Lapis, G.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, 1st edn. Mcgraw-Hill, New York (2012)
McKinsey: Big Data: The Next Frontier For Innovation, Competition and Creativity (2011)
Schniederjans, M.J., Schniederjans, D.G., Starkey, C.M.: Business Analytics Principles, Concepts and Applications, 1st edn. Gill Editorial Services, Pearson Education, Inc., Upper Saddle River (2014)
Imdad, M., et al.: Dark Data: Opportunities and Challenges (2020)
Gimpel, G.: Dark data: the invisible resource that can drive performance now (2021)
CommVault: 5 ways to illuminate your dark data (2014)
Ryan, S.: Illuminating Dark Data (2014)
Hand, D.J.: Dark data: why what you don’t know matters (2020)
Ajis, A.F.M., Ishak, I., Harun, Q.N.: Modelling dark data management framework - a grounded theory (2022)
Bertino, E.: Data protection from insider threats. Synthesis Lect. Data Manage. 4(4), 1–91 (2012). https://doi.org/10.2200/S00431ED1V01Y201207DTM028
CommVault.: Turning dark data into smart data (2014)
Kevin, N.M., Wanyaga, F.M., Kibaara, D., Dinda, W.A., Ngatia, J.K.: Dark data: business analytical tools and facilities for illuminating dark data (2016)
The Data Governance Institute: Definitions of Data Governance (2015). https://datagovernance.com/defining-data-governance/. Accessed 08 Dec 2022
Newman, D., Logan, D.: Governance is an essential building block for enterprise information system. Gartner Research (2006). https://www.gartner.com/en/documents/492444. Accessed 08 Dec 2022
Almeida, B.: Data governance challenges just got easier to solve (2021). https://bluexp.netapp.com/blog/clc-blg-data-governance-just-got-easier-to-solve. Accessed 11 Nov 2022
Henderson, D.: DAMA-DMBOK-Data-Management-Body-of-Knowledge, 2nd edn. (2017)
Wilkinson, M.D., et al.: The FAIR guiding principles for scientific data management and stewardship. Sci. Data 3 (2016). https://www.nature.com/articles/sdata201618. Accessed 11 Nov 2022
Panian, Z.: Some practical experiences in data governance (2010)
Koltay, T.: Data governance, data literacy and the management of data quality. IFLA J. 42(4), 303–312 (2016)
Tallon, P.P., Ramirez, R.V., Short, J.E.: The information artifact in it governance: toward a theory of information governance. J. Manage. Inf. Syst. 30(3), 141–177 (2014)
Donnelley Financial Solutions: Understanding Risk: The Dark Side of Data (2022). https://www.dfinsolutions.com/sites/default/files/documents/2022-10/DealMaker_Meter_Security_Report. Accessed 08 Dec 2022
Al-Ruithe, M., Benkhelifa, E., Hameed, K.: Systematic literature review of data governance & cloud data governance (2019)
Mahanti, R.: Data governance components and framework. In: Mahanti, R. (ed.) Data Governance Success, pp. 127–166. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-5086-4_5, https://doi.org/10.1007/978-981-16-3583-0_4. Accessed 08 Dec 2022
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chant, G.G. (2023). Dealing with Dark Data – Shining a Light. In: Uden, L., Ting, IH. (eds) Knowledge Management in Organisations. KMO 2023. Communications in Computer and Information Science, vol 1825. Springer, Cham. https://doi.org/10.1007/978-3-031-34045-1_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-34045-1_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34044-4
Online ISBN: 978-3-031-34045-1
eBook Packages: Computer ScienceComputer Science (R0)