A comparative study of existing quality measures for process discovery
Introduction
In recent times, organizations possess a tremendous amount of data concerning their customers and products. Many activities which take place in their operational processes are being recorded in event logs [28]. Techniques from the process mining field, which has grown steadily over the last decades, can be applied to gain insights in these event data [27]. In recent years, a lot of attention has been given to the discovery of process models from event logs [11], [18], [29], [35], and subsequently, the quality measurement of these models [1], [2], [6], [25], [26]. Assessing the quality of discovered models is essential in order to find out whether it constitutes an appropriate representation of the process. The quality of discovered process models has been broken down in four dimensions: fitness, precision, generalization and simplicity [22]. For each of the dimensions, several metrics have been implemented, an overview of which can be found in [32].
Although the existing metrics have been used to compare the performance of process discovery algorithms [9], little research has been done concerning the evaluation and comparison of the metrics itself. Until now, it is unclear what the differences are between metrics within the same dimension: do they judge discovered process models in a similar way, or do they qualify models differently? Are some metrics more optimistic or pessimistic than others? Furthermore, there is ongoing debate about the precise definition of certain dimensions, and the relationships between the dimensions. Nevertheless, it is essential to know which quality dimensions to take into account given a specific use case and which measures are most suitable to be used.
In this paper, we conduct an empirical study, incorporating the state-of-the-art quality metrics, with the aim to statistically analyze the relationships between metrics within and among dimensions. The results of the experiments indicate:
- •
the feasibility of the metrics, in terms of CPU-time and memory,
- •
whether metrics measuring the same dimension agree with each other or not,
- •
whether the dimensions are related to each other, or independent from one another,
- •
to which extent some metrics are more optimistic about process model quality compared to others,
- •
to which extent some metrics are more sensitive to differences in process models quality compared to others.
The next section further introduces the different dimensions and the related metrics which are subject of the analysis. Section 3 discusses the experimental set up. The results of the experiment are reported and discussed in Section 4. Section 5 concludes the paper.
Section snippets
Related work
In this section, the quality dimensions are further introduced and discussed. Subsequently, an overview is given of different metrics that have been implemented. Finally, related empirical work is discussed.
Methodology
The methodology used in this paper is based on the framework for comparing process mining algorithms presented in [33]. In particular, the experiment encompasses the steps listed below. Each of these is discussed in more detail in the remainder of this section. The summary of the experiment can be found in Table 3 and a schematic overview is given in Fig. 1.1
Results
The median log size of the generated event logs was 10,704 events, with an overall minimum of 247 and a maximum of 530,203. Each log contained 3649 cases on average, while the average number of distinct activity sequences was 545. It should be noted that for the logs generated from systems with a moderate complexity, the median log size was only 5670 events, while for logs with a high complexity, this was 13,904 events.
Conclusions and future work
In the context of process discovery, being able to evaluate the quality of obtained process models as a representation of the process at hand is essential. In order to do this, different quality dimensions were introduced and for each of the dimensions several metrics were implemented. However, only limited empirical evidence exists on the behavior of these metrics and their relationships both within and across different quality dimensions. Nonetheless, the feasibility, validity and sensitivity
Acknowledgments
The computational resources and services used in this work for both process discovery and process conformance tasks were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government.
Gert Janssenswillen received the M.Sc. degree Business and Information Systems Engineering at Hasselt University, Belgium. He is currently a Ph.D. candidate at the Business Informatics research group at Hasselt University, where his focus lies in the field of business process management. In particular, his main interest goes to the quality measurement of discovered process models.
References (35)
Genetic Process Mining
(2006)- et al.
Robust process discovery with artificial negative events
J. Mach. Learn. Res.
(2009) - et al.
Discovering expressive process models by clustering log traces
Knowl. Data Eng. IEEE Trans.
(2006) - et al.
Multivariate Data Analysis
(2013) - et al.
Workflow mining: discovering process models from event logs
Knowl. Data Eng. IEEE Trans.
(2004) - et al.
Uncovering the relationship between event log characteristics and process discovery techniques
Business Process Management Workshops
(2014) - et al.
A framework for comparing process mining algorithms
GCC Conference and Exhibition
(2011) - et al.
Alignment based precision checking
- et al.
Measuring precision of modeled behavior
Inf. Syst. e-Bus. Manage.
(2015) Flexible Evolutionary Algorithms for Mining Structured Process Models
(2014)
LOWESS: a program for smoothing scatterplots by robust locally weighted regression
Am. Stat.
The weighted varimax rotation and the promax rotation
Psychometrika
An alignment-based framework to check the conformance of declarative process models and to preprocess event-log data
Inf. Syst.
A critical evaluation study of model-log metrics in process discovery
Business Process Management Workshops
A multi-dimensional quality assessment of state-of-the-art process discovery algorithms using real-life event logs
Inf. Syst.
Simplifying discovered process models in a controlled manner
Inf. Syst.
Controlled automated discovery of collections of business process models
Inf. Syst.
Cited by (22)
How active learning and process mining can act as Continuous Auditing catalyst
2019, International Journal of Accounting Information SystemsCitation Excerpt :It is important to note that, depending on which algorithm you use, you might get different results. Despite the improvement steps over the last decade(s), a study by Janssenswillen et al. (2017) reports on the quality issues of existing discovery techniques. These issues are important for an application domain to take into account when possibly adopting new techniques.
Perfect Process Models?! A Process-Discovery Technique Optimizing Over Quality Measures
2023, CEUR Workshop ProceedingsSemi-Automated Approach for Building Event Logs for Process Mining from Relational Database
2022, Applied Sciences (Switzerland)Measuring Fitness and Precision of Automatically Discovered Process Models: A Principled and Scalable Approach
2022, IEEE Transactions on Knowledge and Data EngineeringStochastic Process Model-Log Quality Dimensions: An Experimental Study
2022, Proceedings - 2022 4th International Conference on Process Mining, ICPM 2022Bootstrapping Generalization of Process Models Discovered from Event Data
2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Gert Janssenswillen received the M.Sc. degree Business and Information Systems Engineering at Hasselt University, Belgium. He is currently a Ph.D. candidate at the Business Informatics research group at Hasselt University, where his focus lies in the field of business process management. In particular, his main interest goes to the quality measurement of discovered process models.
Niels Donders received the M.Sc. degree Business and Information Systems Engineering at Hasselt University, Belgium. He is currently a Business Intelligence Consultant at Acumen Consulting were he focusses on enhancing business performance of its clients by streamlining processes, reducing organizational risk and leveraging the global sourcing/outsourcing organizational model.
Toon Jouck received the M.Sc. degree Business and Information Systems Engineering at Hasselt University, Belgium. He is currently a Ph.D. candidate in the Department of Business Informatics, Hasselt University, and his current research interests include business process mining and management, data analytics and experimental design.
Benoît Depaire is an Associate Professor at Hasselt University, Belgium. He has authored and co-authored numerous papers published in international journals and conference proceedings. His current research interests include business process management, experimental design, process simulation and IT-business models. His aim is to use gain empirically-validated insights from process-related data to understand and improve business processes.