A Survey of Data Challenges Across a Modernizing Bureaucracy: A New Perspective on Examining Old Government Problems

Abstract

The introduction and increasing popularity of artificial intelligence (AI) and machine learning (ML) technologies allow organizations to gain valuable insights from their copious amounts of data. However, legacy organizations often struggle to overcome outdated data management practices and unleash the potential of AI and ML on their data. There is simply too much data to sift through manually. Therefore, a data science tool is required to locate relevant information effectively within an organizations’ data lake. This paper presents a survey of challenges government organizations face related to this data discovery issue. The challenges are ubiquitous across mission sets, covering human resources and personnel management, logistics and supply chains, fraud and predatory business detection, government procurement, and civil litigation. This paper introduces the Data Discovery by Example (DICE) system to alleviate this problem. Unlike traditional data discovery techniques to find data of interest within a data lake, DICE alleviates the need to write queries and does not require users to manually inspect the lake to find their data of interest. Lastly, we walk through a DICE example, where we apply the tool on data from complaints of predatory business practices. This example highlights the challenges of acquiring, accessing, and interpreting data across multiple agencies and functions. There is an opportunity for innovative, ML-enabled data discovery solutions, such as DICE, to help unlock the value of data and augment development of a modern government organization.