From Screenshots to Process Models: Improving Activity Identification Through Screen Text

Abstract

The analysis for executing Robotic Process Automation (RPA) projects increasingly relies on monitoring user activities through Robotic Process Mining (RPM) techniques. Traditional approaches capture direct information using loggers that capture UI logs, i.e., sequences of events that include data from (1) the keyboard, (2) the mouse, and (3) the application elements, such as its name, the Excel cell, the clicked button, etc. Although the latter is highly relevant for identifying the activity that is being performed, this information is not accessible in virtualized environments; only screenshot data is available. This limitation necessitates activity identification based on screenshots alone. A significant challenge with this method is its sensitivity to minor interface changes, such as different zoom levels or notifications, which can cause detection failures. To address this, we propose a novel approach that, first, integrates embeddings from both screenshots and screen text obtained through OCR and, second, clusters the UI log events using these combined features to identify the activity. Our results show that this method enhances activity identification, outperforming current state-of-the-art techniques, and demonstrates promising improvements in accuracy and reliability.

This research was supported by the EQUAVEL project PID2022-137646OB-C31, funded by MICIU/AEI/10.13039/501100011033 and by FEDER, UE; the DISCOVERY project (2021/C005/00148631), funded by Unión Europea NextGeneration EU and “Plan de Recuperación, Transformación y Resiliencia” of the Ministry of Economic and Digital Transformation; and the grant FPU20/05984 funded by MICIU/AEI/10.13039/501100011033 and by FSE+.