Skip to main content
SpringerLink
Log in

Computational Intelligence Techniques for Assessing Data Quality: Towards Knowledge-Driven Processing

  • Conference paper
  • First Online:
Computational Science – ICCS 2021 (ICCS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12744))

Included in the following conference series:

Abstract

Since the right decision is made from the correct data, assessing data quality is an important process in computational science when working in a data-driven environment. Appropriate data quality ensures the validity of decisions made by any decision-maker. A very promising area to overcome common data quality issues is computational intelligence. This paper examines from past to current intelligence techniques used for assessing data quality, reflecting the trend for the last two decades. Results of a bibliometric analysis are derived and summarized based on the embedded clustered themes in the data quality field. In addition, a network visualization map and strategic diagrams based on keyword co-occurrence are presented. These reports demonstrate that computational intelligence, such as machine and deep learning, fuzzy set theory, evolutionary computing is essential for uncovering and solving data quality issues.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. de Aquino, G.R.C., de Farias, C.M., Pirmez, L.: Hygieia: data quality assessment for smart sensor network. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC 2019, pp. 889–891. Association for Computing Machinery, New York (2019)

    Google Scholar 

  2. BoesBoes, L., et al.: Evaluation of the German surveillance system for hepatitis B regarding timeliness, data quality, and simplicity, from 2005 to 2014. Public Health 180, 141 (2020)

    Article  Google Scholar 

  3. Bouhamed, S.A., Dardouri, H., Kallel, I.K., Bossé, E., Solaiman, B.: Data and information quality assessment in a possibilistic framework based on the Choquet Integral. In: 5th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pp. 1–6 (2020)

    Google Scholar 

  4. Cai, L., Zhu, Y.: The challenges of data quality and data quality assessment in the big data era. Data Sci. J. 14, 1–10 (2015)

    Article  Google Scholar 

  5. Chu, X., Ilyas, I.F., Krishnan, S., Wang, J.: Data cleaning: overview and emerging challenges. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 2201–2206 (2016)

    Google Scholar 

  6. Cichy, C., Rass, S.: An overview of data quality frameworks. IEEE Access 7, 24634–24648 (2019)

    Article  Google Scholar 

  7. Cobo, M.J., López-Herrera, A.G., Herrera-Viedma, E., Herrera, F.: An approach for detecting, quantifying, and visualizing the evolution of a research field: a practical application to the fuzzy sets theory field. J. Inf. 5(1), 146–166 (2011)

    Google Scholar 

  8. Corrales, D.C., Corrales, J.C., Ledezma, A.: How to address the data quality issues in regression models: a guided process for data cleaning. Symmetry 10(4), 99 (2018)

    Article  Google Scholar 

  9. Das, S., Saha, B.: Data quality mining using genetic algorithm. Int. J. Comput. Sci. Secur. IJCSS 3(2), 105–112 (2009)

    Google Scholar 

  10. van Eck, N.J., Waltman, L.: Software survey: VOS viewer, a computer program for bibliometric mapping. Scientometrics 84(2), 523–538 (2010)

    Article  Google Scholar 

  11. Heinrich, B., Klier, M.: Metric-based data quality assessment - developing and evaluating a probability-based currency metric. Decis. Support Syst. 72, 82–96 (2015)

    Article  Google Scholar 

  12. Heinrich, B., Klier, M., Schiller, A., Wagner, G.: Assessing data quality - a probability-based metric for semantic consistency. Decis. Support Syst. 110, 95–106 (2018)

    Article  Google Scholar 

  13. Hermans, K., et al.: Novel approaches to assess the quality of fertility data stored in dairy herd management software. J. Dairy Sci. 100(5), 4078–4089 (2017)

    Article  Google Scholar 

  14. Jarwar, M.A., Chong, I.: Web objects based contextual data quality assessment model for semantic data application. Appl. Sci. 10(6), 2181 (2020)

    Article  Google Scholar 

  15. Larson, S., et al.: Outlier detection for improved data quality and diversity in dialog systems. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 517–527 (2019)

    Google Scholar 

  16. Li, A., et al.: TODQA: efficient task-oriented data quality assessment. In: 15th International Conference on Mobile Ad-Hoc and Sensor Networks, pp. 81–88. IEEE (2019)

    Google Scholar 

  17. Li, J.S., Hamann, A., Beaubien, E.: Outlier detection methods to improve the quality of citizen science data. Int. J. Biometeorol. 64, 1825–1833 (2020)

    Article  Google Scholar 

  18. Liberati, A., et al.: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration, J. Clin. Epidemiol. 62(10), e1–e34 (2009)

    Google Scholar 

  19. Liu, H., Ashwin Kumar, T.K., Thomas, J.P.: Cleaning framework for big data - object identification and linkage. In: 2015 IEEE International Congress on Big Data, pp. 215–221 (2015)

    Google Scholar 

  20. Liu, H., Wang, X., Lei, S., Zhang, X., Liu, W., Qin, M.: A rule based data quality assessment architecture and application for electrical data. In: Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing. Association for Computing Machinery (2019)

    Google Scholar 

  21. Malar Vizhi, J.: Data quality measurement on categorical data using genetic algorithm. Int. J. Data Min. Knowl. Manage. Process. 2(1), 33–42 (2012)

    Article  Google Scholar 

  22. Nagle, T., Redman, T., Sammon, D.: Assessing data quality: A managerial call to action. Bus. Horiz. 63(3), 325–337 (2020)

    Article  Google Scholar 

  23. Pezoulas, V.C., et al.: Medical data quality assessment: on the development of an automated framework for medical data curation. Comput. Biol. Med. 107, 270–283 (2019)

    Article  Google Scholar 

  24. Pietro Biancone, P., Secinaro, S., Brescia, V., Calandra, D.: Data quality methods and applications in health care system: a systematic literature review. Int. J. Bus. Manage. 14(4), 35 (2019)

    Article  Google Scholar 

  25. Plotkin, D.: Important Roles of Data Stewards. In: Data Stewardship, pp. 127–162. Morgan Kaufmann (2014)

    Google Scholar 

  26. Pucher, S., Król, D.: A quality assessment tool for Koblenz datasets using metrics-driven approach. In: Fujita, H., Fournier-Viger, P., Ali, M., Sasaki, J. (eds.) IEA/AIE 2020. LNCS (LNAI), vol. 12144, pp. 747–758. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55789-8_64

    Chapter  Google Scholar 

  27. Rafi, S., Yu, W., Akbar, M.A., Alsanad, A., Gumaei, A.: Multicriteria based decision making of DevOps data quality assessment challenges using fuzzy TOPSIS. IEEE Access 8, 46958–46980 (2020)

    Article  Google Scholar 

  28. Rajan, N.S., Gouripeddi, R., Mo, P., Madsen, R.K., Facelli, J.C.: Towards a content agnostic computable knowledge repository for data quality assessment. Comput. Methods Programs Biomed. 177, 193–201 (2019)

    Article  Google Scholar 

  29. Simard, V., Rönnqvist, M., Lebel, L., Lehoux, N.: A general framework for data uncertainty and quality classification. IFAC PapersOnLine 52(13), 277–282 (2019)

    Article  Google Scholar 

  30. Symoens, S.H., et al.: QUANTIS: data quality assessment tool by clustering analysis. Int. J. Chem. Kinet. 51(11), 872–885 (2019)

    Article  Google Scholar 

  31. Taleb, I., Serhani, M.A., Dssouli, R.: Big Data Quality Assessment Model for Unstructured Data. In: Proceedings of the 13th International Conference on Innovations in Information Technology, pp. 69–74 (2018)

    Google Scholar 

  32. Terry, A.L., Stewart, M., Cejic, S., et al.: A basic model for assessing primary health care electronic medical record data quality. BMC Med. Inform. Decis. Mak. 19, 30 (2019)

    Article  Google Scholar 

  33. Udeshi, S., Jiang, X., Chattopadhyay, S.: Callisto: entropy-based test generation and data quality assessment for machine learning systems. In: 13th International Conference on Software Testing, Verification and Validation, pp. 448–453 (2020)

    Google Scholar 

  34. Valencia-Parra, Á., Parody, L., Varela-Vaca, Á.J., Caballero, I., Gómez-López, M.T.: DMN4DQ: when data quality meets DMN. Decis. Support Syst. 141, 113450 (2021)

    Article  Google Scholar 

  35. Zan, S., Zhang, X.: Medical data quality assessment model based on credibility analysis. In: 4th Information Technology and Mechatronics Engineering Conference, pp. 940–944 (2018)

    Google Scholar 

  36. Zimek, A., Schubert, E.: Outlier detection. In: Liu, L., Öszu, T. (eds.) Encyclopedia of Database Systems. Springer, Cham (2017) https://doi.org/10.1007/978-1-4899-7993-3_80719-1

Download references

Acknowledgments

Part of the work presented in this paper was received financial support from the statutory funds at the Wrocław University of Science and Technology.

Author information

Authors and Affiliations

  1. Wroclaw University of Science and Technology, Wroclaw, Poland

    Nunik Afriliana & Dariusz Król

  2. Universitas Mutimedia Nusantara, Jakarta, Indonesia

    Nunik Afriliana

  3. Computer Science Department, Bina Nusantara University, Jakarta, Indonesia

    Ford Lumban Gaol

Authors
  1. Nunik Afriliana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dariusz Król
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ford Lumban Gaol
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nunik Afriliana .

Editor information

Editors and Affiliations

  1. AGH University of Science and Technology, Krakow, Poland

    Maciej Paszynski

  2. Ludwig-Maximilians-Universität München, Munich, Germany

    Dieter Kranzlmüller

  3. University of Amsterdam, Amsterdam, The Netherlands

    Valeria V. Krzhizhanovskaya

  4. University of Tennessee at Knoxville, Knoxville, TN, USA

    Jack J. Dongarra

  5. University of Amsterdam, Amsterdam, The Netherlands

    Peter M.A. Sloot

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Afriliana, N., Król, D., Gaol, F.L. (2021). Computational Intelligence Techniques for Assessing Data Quality: Towards Knowledge-Driven Processing. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2021. ICCS 2021. Lecture Notes in Computer Science(), vol 12744. Springer, Cham. https://doi.org/10.1007/978-3-030-77967-2_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77967-2_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77966-5

  • Online ISBN: 978-3-030-77967-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation