Abstract
User Interface (UI) logs are crucial in capturing and analyzing user behavior, enabling a comprehensive understanding of business processes and eventual process automation with Robotic Process Automation (RPA). However, extracting meaningful insights from UI logs becomes challenging, especially when dealing with complex and information-dense graphical user interfaces. This paper presents a novel approach that leverages eye-tracking technology to address this challenge. The proposed solution incorporates gaze fixation (i.e., where the user pays attention to the user interfaces) into the UI log, which is then used to filter irrelevant information from it. Two gaze-based filtering methods are presented and evaluated using synthetic and real-life screenshots. Preliminary results demonstrate that the method effectively reduces the irrelevant UI elements by an average of 76% while keeping meaningful information on the screen.
This publication is part of the projects PID2019-105455GB-C31 and PID2022-137646OB-C31, funded by MCIN/ AEI/10.13039/501100011033/ and by the “European Union”; the FPU scholarship program, granted by the Spanish Ministry of Education and Vocational Training (FPU20/05984) and its mobility grants (EST23/00732).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The set of problems is available at: https://doi.org/10.5281/zenodo.8009445.
- 2.
iMotions Eye Tracking System: https://imotions.com/eye-tracking/#Solution.
References
van der Aalst, W.M.P.: Process mining: data science in action, 2nd edn. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49851-4
Abb, L., Rehse, J.R.: A reference data model for process-related user interaction logs. In: Di Ciccio, C., Dijkman, R., del Rio Ortega, A., Rinderle-Ma, S. (eds.) Business Process Management. BPM 2022. LNCS, vol. 13420, pp. 57–74. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16103-2_7
Abbad Andaloussi, A., Slaats, T., Burattin, A., Hildebrandt, T.T., Weber, B.: Evaluating the understandability of hybrid process model representations using eye tracking: first insights. In: Daniel, F., Sheng, Q.Z., Motahari, H. (eds.) BPM 2018. LNBIP, vol. 342, pp. 475–481. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11641-5_37
Brunyé, T.T., Drew, T., Weaver, D.L., Elmore, J.G.: A review of eye tracking for understanding and improving diagnostic interpretation. Cogn. Res. Princ. Implic. 4(1), 1–16 (2019). https://doi.org/10.1186/s41235-019-0159-2
Feit, A.M., Vordemann, L., Park, S., Berube, C., Hilliges, O.: Detecting relevance during decision-making from eye movements for UI adaptation. In: ACM Symposium on Eye Tracking Research and Applications, pp. 1–11 (2020)
Gwizdka, J.: Characterizing relevance with eye-tracking measures. In: Proceedings of the 5th Information Interaction in Context Symposium, pp. 58–67 (2014)
Ioannou, C., Nurdiani, I., Burattin, A., Weber, B.: Mining reading patterns from eye-tracking data: method and demonstration. Softw. Syst. Model. 19, 345–369 (2020)
Jimenez-Ramirez, A., Reijers, H.A., Barba, I., Del Valle, C.: A method to improve the early stages of the robotic process automation lifecycle. In: Giorgini, P., Weber, B. (eds.) CAiSE 2019. LNCS, vol. 11483, pp. 446–461. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21290-2_28
Leno, V., Polyvyanyy, A., Dumas, M., La Rosa, M., Maggi, F.M.: Robotic process mining: vision and challenges. Busin. Inf. Syst. Eng. 63, 301–314 (2021)
Martínez-Rojas, A., Jiménez-Ramírez, A., Enríquez, J.G., Reijers, H.A.: Analyzing variable human actions for robotic process automation. In: Di Ciccio, C., Dijkman, R., del Rio Ortega, A., Rinderle-Ma, S. (eds.) Business Process Management. BPM 2022. LNCS, vol. 13420, pp. 75–90. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16103-2_8
Martínez-Rojas., A., Jiménez-Ramírez., A., Enríquez., J.G., Lizcano-Casas., D.: Incorporating the user attention in user interface logs. In: Proceedings of the 18th International Conference on Web Information Systems and Technologies - WEBIST, pp. 415–421. INSTICC, SciTePress (2022). https://doi.org/10.5220/0011568000003318
Moran, K., Bernal-Cárdenas, C., Curcio, M., Bonett, R., Poshyvanyk, D.: Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans. Software Eng. 46(2), 196–221 (2018)
Petrusel, R.: Integrating click-through and eye-tracking logs for decision-making process mining. Inform. Econ. 18(1), 56 (2014)
Petrusel, R., Mendling, J., Reijers, H.A.: Task-specific visual cues for improving process model understanding. Inf. Softw. Technol. 79, 63–78 (2016)
Petrusel, R., Mendling, J., Reijers, H.A.: How visual cognition influences process model comprehension. Decis. Support Syst. 96, 1–16 (2017)
Reinkemeyer., L.: Process mining in action. principles, use cases and outlook, 1st Edn. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-40172-6
Simko, J., Vrba, J.: Screen recording segmentation to scenes for eye-tracking analysis. Multimedia Tools Appl. 78, 2401–2425 (2019)
Weber, B., Gulden, J., Burattin, A.: Designing visual decision making support with the help of eye-tracking. In: RADAR+ EMISA@ CAiSE, pp. 47–54 (2017)
Winter, M., Neumann, H., Pryss, R., Probst, T., Reichert, M.: Defining gaze patterns for process model literacy-exploring visual routines in process models with diverse mappings. Expert Syst. Appl. 213, 119217 (2023)
Xie, M., Feng, S., Xing, Z., Chen, J., Chen, C.: UIED: a hybrid tool for GUI element detection. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1655–1659 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Martínez-Rojas, A., Reijers, H.A., Jiménez-Ramírez, A., Enríquez, J.G. (2023). What Are You Gazing At? An Approach to Use Eye-Tracking for Robotic Process Automation. In: Köpke, J., et al. Business Process Management: Blockchain, Robotic Process Automation and Educators Forum. BPM 2023. Lecture Notes in Business Information Processing, vol 491. Springer, Cham. https://doi.org/10.1007/978-3-031-43433-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-43433-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43432-7
Online ISBN: 978-3-031-43433-4
eBook Packages: Computer ScienceComputer Science (R0)