Abstract
Data is getting more and more ubiquitous while its importance rises. The quality and outcome of business decisions is directly related to the accuracy of data used in predictions. Thus, a high data quality in database systems being used for business decisions is of high importance. Otherwise bad consequences in the form of commercial loss or even legal implications loom.
In this paper we focus on automating advanced data quality monitoring, and especially the aspect of expressing and evaluating rules for good data quality. We present a domain specific language (DSL) called RADAR for data quality rules, that fulfills our main requirements: reusability of check logic, separation of concerns for different user groups, support for heterogeneous data sources as well as advanced data quality rules such as time series rules. Also, it provides the option to automatically suggest potential rules based on historic data analysis. Furthermore, we show initial optimization approaches for the execution of rules on large data sets and evaluate our language based on these optimizations.
All in all the language presents a novel approach for a flexible and powerful management of data quality in practical applications while meeting the needs of actual data quality managers in being pragmatic and efficient.
The project IQM4HD has been funded by the German Federal Ministry of Education and Research under grant no. 01IS15053A. We would also like to thank our partners SHS Viveon/mVise for implementing the prototype and CTS Eventim for providing important requirements and reviewing practical applicability of the prototype and concepts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Rule Language for Automated Data Quality Assessment and Reporting.
- 2.
- 3.
- 4.
ftp://ftp.fu-berlin.de/pub/misc/movies/database/.
- 5.
The errors were introduced by updating values to NULL or to non-existing FK values for a random subset of the records.
- 6.
- 7.
References
Atzeni, P., Bugiotti, F., Rossi, L.: SOS (Save Our Systems): a uniform programming interface for non-relational systems. In: Proceedings of the 15th International Conference on Extending Database Technology, pp. 582–585. ACM, New York (2012)
Binnig, C., Rehrmann, R., Faerber, F., Riewe, R.: FunSQL: it is time to make SQL functional. In: Proceedings of the 2012 Joint EDBT/ICDT Workshops, pp. 41–46. ACM, New York (2012)
Buneman, P., Fernandez, M., Suciu, D.: UnQL: a query language and algebra for semistructured data based on structural recursion. VLDB J. 9(1), 76–110 (2000)
Cheney, J., Lindley, S., Wadler, P.: A practical theory of language-integrated query. In: Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming, ICFP 2013, pp. 403–416. ACM, New York (2013)
Ehrlinger, L., Haunschmid, V., Palazzini, D., Lettner, C.: A DaQL to monitor data quality in machine learning applications. In: Hartmann, S., Küng, J., Chakravarthy, S., Anderst-Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DEXA 2019. LNCS, vol. 11706, pp. 227–237. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27615-7_17
Ehrlinger, L., Rusz, E., Wöß, W.: A survey of data quality measurement and monitoring tools. arXiv preprint 1907.08138 (2019)
Ehrlinger, L., Wöß, W.: Automated data quality monitoring. In: MIT International Conference on Information Quality, vol. 22, pp. 19-1-19-8 (2017)
Endler, G., Schwab, P.K., Wahl, A.M., Tenschert, J., Lenz, R.: An architecture for continuous data quality monitoring in medical centers. In: MEDINFO (2015)
Feuerstein, S., Pribyl, B.: Oracle PL/SQL Programming. O’Reilly Media, Inc., Sebastopol (2005)
Garcia-Molina, H., et al.: The TSIMMIS approach to mediation: data models and languages. J. Intell. Inf. Syst. 8(2), 117–132 (1997)
Heine, F., Kleiner, C., Koschel, A., Westermayer, J.: The Data Checking Engine: Complex Rules for Data Quality Monitoring. Int. J. Adv. Software 7(12), 171–181 (2014)
Heine, F., Kleiner, C., Oelsner, T.: Automated detection and monitoring of advanced data quality rules. In: Hartmann, S., Küng, J., Chakravarthy, S., Anderst-Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DEXA 2019. LNCS, vol. 11706, pp. 238–247. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27615-7_18
Oelsner, T., Heine, F., Kleiner, C.: IQM4HD Concepts. Technical report, University of Applied Sciences and Arts Hannover, Germany (2018). http://iqm4hd.wp.hs-hannover.de/ConceptsIQM4HD_v09.pdf
Pipino, L.L., Lee, Y.W., Wang, R.Y.: Data quality assessment. Commun. ACM 45(4), 211–218 (2002). https://doi.org/10.1145/505248.506010
Rith, J., Lehmayr, P.S., Meyer-Wegener, K.: Speaking in tongues: SQL Access to NoSQL systems. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC 2014, pp. 855–857. ACM, New York (2014)
Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., Grafberger, A.: Automating large-scale data quality verification. Proc. VLDB Endowment 11(12), 1781–1794 (2018)
Sobieski, S., Zieliński, B.: Using maude rewriting system to modularize and extend SQL. In: Proceedings of the 28th Annual ACM Symposium on Applied Computing, SAC 2013, pp. 853–858. ACM, New York (2013)
Zhang, C., Xu, J.: A unified SQL middleware for NoSQL databases. In: Proceedings of the 2018 International Conference on Big Data and Computing, ICBDC 2018, pp. 14–19. ACM, New York (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Heine, F., Kleiner, C., Oelsner, T. (2020). A DSL for Automated Data Quality Monitoring. In: Hartmann, S., Küng, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2020. Lecture Notes in Computer Science(), vol 12391. Springer, Cham. https://doi.org/10.1007/978-3-030-59003-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-59003-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59002-4
Online ISBN: 978-3-030-59003-1
eBook Packages: Computer ScienceComputer Science (R0)