1 Introduction

In recent years, many governments and industrial associations in the world introduced various initiatives oriented towards the new industrial revolution, including Industry 4.0. The vision of these initiatives is incorporating technological changes and advancements in different industrial applications, which subsequently will induce changes in the nature of the tasks and the demands for the human workers. Each individual worker in context of Industry 4.0 will face variety of challenges and problems to solve, mostly related to high cognitive activities [1]. To address this problem, one of the visions is to ensure that the machines and the humans interact and work collaboratively, such that the machines assist the humans. In terms of the Human-Machine Interaction, the User Interface (UI) plays the most important role as a mediator between the human and a certain machine. Traditional industrial UIs are usually designed to handle a unimodal input and to present visual feedback. However, in a dynamically context-changing environment, the possibility for having a unimodal interaction represents a big constraint for the human workers. Therefore, there is an emerging need for providing multi-modal UIs which adapt to the current context, personal abilities and preferences of the human workers. In order to tackle this problem, in our paper, we introduce a solution approach for providing multi-modal adaptive UIs for assistance systems in context of industrial manufacturing processes. For this purpose, we extend our existing model-based reference framework for adaptive UIs [2] by incorporation of a Digital Twin of a human in order to define a structure which models in a systematic and fine-grained way specific human abilities, characteristics and preferences. We use this structure as a source for providing personalized information, based on which the UIs of assistance systems can adapt. The rest of the paper is structured in the following way: in Sect. 2 we elaborate two example scenarios which describe the problem space. Section 3 presents the related work in respect to topics related to adaptive UIs and the concept of a digital twin. Our solution approach is elaborated in Sect. 4 and in Sect. 5 we evaluate our approach based on two case studies. Section 6 concludes this paper and provides outlook for the future work.

2 Example Scenarios

In this section, we present two example scenarios in order to illustrate the problem in respect to assistance systems in manufacturing processes. The first example scenario is related to manual assembly of an Electrical Cabinet (E-cabinet), while the second scenario represents the process of manual assembly of a concrete product in a Smart Factory. With these example scenarios we intend to illustrate more precisely the problem domain and the current challenges. These example scenarios are derived from the results of our interdisciplinary study, realized by conduction of semi-structured interviews with experts from different research fields: Psychology, Sociology, Didactic, Economics, Computer Science, Electrical and Mechanical Engineering. The main objective of the study was investigation of the requirements and needs of the human workers in the industrial sector and development of a solution to meet these requirements.

2.1 Example Scenario 1: Manual Assembly of E-Cabinet

Figure 1 illustrates an example scenario by emphasizing the challenges which the human worker (electrician) is facing by performing an assembly task for an E-cabinet with help of an assistance system - tablet. As we can observe in Fig. 1, there are six different situations depicting the current challenges. According to Situation 2, the electrician faces problems due to the brightness of the tablet and has to stop assembling in order to re-configure it. In Situation 3, he has to perform actions in the back side of the E-Cabinet and he can’t carry the tablet with him. He has to go several times back and forth to finish connecting some cables. Situation 4 shows that even in vocal mode, the adaptation is not appropriately personalized - the interfering noise from other machines around and the speed of the voice instructions are too fast for a 60 year old worker.

Fig. 1.
figure 1

Example Scenario 1 - manual assembly of an E-Cabinet

Full size image

In Situation 5, we can observe that he has difficulties in terms of the navigation and dealing with the information presentation on the graphical UI, since he doesn’t have advanced user experience. Furthermore, as presented in Situation 6, when attempting to perform an action from a longer distance, the electrician faces problems in recognition of the layout and the size of the UI elements.

2.2 Example Scenario 2: Manual Assembly of a Product in a Smart Factory

This scenario demonstrates the challenges which the human worker faces during manual assembly of a product (LEGO car) in a Smart Factory. As Fig. 2 depicts, the working environment consists of an Assembly Station, a Tool Kit and an Assistance System - Tablet (on the left side, fixed). According to Situation 1, the worker opens the interactive guide and starts with the assembly instructions, which are visually presented on the screen. In Situation 2, the worker has to use gloves for operating with a tool and therefore he can not give commands by touch. It takes him a lot of time to stop and set back the instructions while removing the gloves several times.

Fig. 2.
figure 2

Example Scenario 2 - manual assembly of a product in Smart Factory

Full size image

In Situation 3, the worker switches to the next assembly step, for which he has to use a specific tool and it is practically impossible to follow the instructions while searching for the tool in the Tool Kit placed on the right side. This situation requires that the worker again moves several times until he identifies the desired tool. In Situation 4, the worker finds difficulties in recognizing a concrete assembly piece of the product. He is confused, because he sees many details on the screen, which de-focus him. He needs more precise details about the concrete piece, since the assembly piece looks very similar to some of the previous ones. For this, the worker invests much time in navigating and orienting on the graphical UI.

From these two scenarios, we can observe that both human workers experience several challenges during interaction with an assistance system due to context-changing conditions, as well as their personal abilities and characteristics. Based on the concrete example scenarios, we can identify several main challenges for supporting the development of smart assistance systems:

  • C1: Monitoring the context of use. This aspect refers to uncontrollable contextual and environmental changes during the working process and domain-specific constraints, such as change in the lighting condition, noise interference or distance and location variability.

  • C2: Lack of UI adaptation features in existing assistance systems. This aspect addresses the problem that existing assistance systems lack capabilities to adapt the UI based on the current contextual conditions, such as switching to vocal mode, adapting the navigation or adapting the presentation.

  • C3: User Experience improvement through analysis of interaction patterns. This aspect targets the need for personalization and knowledge from past user experiences. Currently, there is no approach in the context of assistance systems for manufacturing processes which includes these factors in the UI adaptation process. From the example scenarios, we can observe the necessity for the assistance systems to obtain knowledge about the workers’ personal characteristics (e.g. age, vision impairments, cognitive capabilities, etc.) and the past user experiences (e.g. usage of certain devices, apps, interaction styles, etc.)

3 Related Work

In this section, we provide an overview of the related work in respect to the concepts associated to adaptive UI and digital twin. In terms of assistance systems, the study of Gorecky [1] provides an overview of the possibilities for human-machine interaction in Industry 4.0, with indication that the interaction in the industrial environments in the future will be mostly based on intelligent mobile assistance systems, such as tablets and mobile phones.

3.1 Adaptive UI

Currently, various research approaches address specific aspects of the UI adaptation process. An overview of the variety of adaptation techniques is presented in the studies of Oppermann [3] and Brusilovsky [4]. Furthermore, in our existing work [2, 5,6,7], we have established a model-driven engineering approach for generating context-specific adaptive UIs which support the modeling, transformation and execution of adaptive UIs. Our model-driven engineering approach was applied for different application domains like cross-channel banking applications or library management applications. In addition, we elaborate the aspect of model-driven context management in our work presented in [8], with focus on contextual parameters which effect the adaptation process of the UIs. However, these approaches do not focus on UI adaptation in terms of incorporating a fine-grained human model for accurate personalization of the adaptive UIs.

3.2 Digital Twin

Nowadays, the Digital Twin framework is frequently mentioned as one of the main enablers for digital transformation. The term Digital Twin was first introduced in 2002 by Grieves [9] in context of product lifecycle management. In the context of Industry 4.0, the Digital Twin is introduced as a framework for mirroring certain aspects of the underlying physical entities in the manufacturing processes. The high-level goal of using a Digital Twin framework is to ensure certain type of process optimization [10,11,12,13]. However, in respect of integrating the human factor, still, one of the biggest challenges requires establishing a relevant knowledge and comprehensive information model of the human worker. This knowledge can be used for various purposes such as for UI adaptation processes or in recommender systems.

4 Digital Twin-Based Multi-modal UI Adaptation Framework

In order to tackle the defined challenges, we have developed a Digital Twin-Based Multi-Modal UI Adaptation Framework for assistance systems in the context of manufacturing processes. As depicted in Fig. 3, our solution extends the existing MAPE-K paradigm for self-adaptive software systems [14] and it consists of three main components: Context Manager, Adaptation Manager and Self-adaptive UI.

The Context Manager addresses challenge C1 and is responsible for storing context of use data and providing contextual information to the Adaptation Manager. The Context Manager consists of three models: Platform Model, Environment Model and a Digital Twin of a Human Model.

The Platform Model defines the characteristics of the underlying interaction platform. It includes the Operating System and various properties of the underlying Device, such as: Battery Level, Network Connection, Screen, as well as various types of Sensors. In addition, the Context Manager contains the Environment Model which specifies the environmental characteristics and include the current Time, Date, Noise Level, Movement Status, Weather and Location. Furthermore, the Digital Twin of a Human Model provides a fine grained representation of the human worker, by considering numerous human characteristics including: Personal and Demographic Data, Physical Abilities, Sensory Abilities, Communication Preferences, Skills, Social and Personal Characteristics, Health Data and Cognitive Abilities. The development of the Digital Twin of a Human Model is based on an interdisciplinary study by conduction of semi-structured interviews with experts engaged in various research fields. By incorporation of the Digital Twin of a Human Model we include the personalization factor as an additional context information for adaptation of the UIs.

Fig. 3.
figure 3

Digital Twin-Based Multi-modal UI Adaptation Framework

Full size image

The Adaptation Manager follows the MAPE-K paradigm [14] and it is responsible for Monitoring the context of use provided by the Context Manager, Analyzing and Planning which adaptation rules will be triggered and finally Executing the adaption operations on the Self-adaptive UI. The Adaptation Manager supports four different layers of adaptation rules for the adaptive UIs: Navigation, Layout, Task-Feature Set Minimization/Maximization and Modality adaptation which are explained in the following paragraph. The adaptation rules are specified based on the ECA (Event-Condition-Action) paradigm and the Context Model. By establishing a Knowledge component in the Adaptation Manager we address challenge C3 since it is possible to store log data about previous context information and adaptation changes. This is achieved due to the Self-adaptive UI which informs the Context Manager about previously executed adaptation information.

The Self-adaptive UI component addresses the challenge C2 and provides a solution for this requirement by establishment of adaptation features which allow the UIs to be adapted. The adaptation features include: Navigation, Layout, Task-Feature Set Minimization/Maximization and Modality adaptation. The Navigation adaptation deals with changes in the navigation flow and adapting the navigation links, while the Layout adaptation ensures proper adaptation of the Font, Font Size, Color Theme, etc. Furthermore, the Task-Feature Set Minimization/Maximization adaptation allows changes based on the current context where the amount of information presented to the user is minimized or maximized. The Modality adaptation allows change in the interaction modality with the user. When certain conditions are detected, the graphical UI can change to a vocal UI and the user can interact by providing and receiving voice commands.

5 Case Studies

In this section, we demonstrate and apply our solution approach for two case studies related to the example scenarios presented in Sect. 2. We describe the solution based on the Digital Twin-Based Multi-Modal UI Adaptation Framework. Besides the variety of sensors which are monitoring the environment, the framework incorporates a fine-grained Digital Twin of a Human Model for providing personalized adaption of the UI.

5.1 Case Study 1: Solution Approach for Manual Assembly of E-Cabinet

In this case study, we apply our model-based solution framework for Example Scenario 1, which is depicted in Fig. 1. In order to tackle the challenge in Situation 2, the Digital Twin provides information about Vision Impairment of the worker. In addition, the Ambient Light Sensor detects the Brightness decrease in the room and based on these two parameters, the brightness of the tablet is adapted appropriately. In respect to Situation 3, we can refer to the excerpt from the Object Diagram of the Context Model presented in Fig. 4. The Proximity Sensor detects that, despite the Movement Status, the Distance between the worker and the tablet is too big (80 cm) and this is the condition which triggers adaptation of the UI Modality into vocal. Figure 5, depicts this concrete adaptation rule and the underlying parameters for adapting the modality from visual to vocal.

However, in Situation 4, the speed and the sound level of the voice instructions are too high for the worker. Based on the personalized information from the Digital Twin, such as Usage Time, Age and previous Interaction Data, the UI is adapted, such that the Speed and the Sound Volume of the instructions fit to the personal abilities of the human worker. In Fig. 6, we can observe the underlying parameters for triggering the adaptation rule for Situation 4.

Fig. 4.
figure 4

Excerpt from the Object Diagram of the Context Model for Example Scenario 1 - Situation 3

Full size image

In Situation 5, the worker faces a problem with the navigation and the content presentation. The Digital Twin provides information that the worker has used the app only 2 days. Based on additional information from the Digital Twin the navigation is adapted to a grid navigation and the amount of content presented on the UI is decreased. The worker can now navigate more intuitively though the app. In Situation 6, the distance among the worker and the tablet increases again.

Fig. 5.
figure 5

Adaptation rule for adapting to vocal modality (Example Scenario 1 - Situation 3)

Full size image
Fig. 6.
figure 6

Adaptation rule for adapting vocal preferences (Example Scenario 1 - Situation 4)

Full size image

In this case, the Digital Twin provides information about his preferences for Font, Font Size and Theme Color when he works on longer distance. Therefore, the Font Size increases and the Color Theme changes, such that the worker can clearly follow the visual content presentation.

5.2 Case Study 2: Solution Approach for Manual Assembly of a Product in a Smart Factory

In the following, we elaborate the solution approach for Example Scenario 2 presented in this paper. As we can observe from Fig. 2, the worker starts to use the interactive visual guide for instructions. However, in Situation 2 he has to use gloves for operating a tool and the visual UI modality is not appropriate anymore.

Similarly as in Example Scenario 1, the Context Model provides information from the past user experience about preferred interaction modality when using a concrete tool and the UI modality is switched to vocal. Furthermore, the Digital Twin provides personalized information about the vocal interaction preferences of the user - for the adaptation process in Situation 2.

Fig. 7.
figure 7

Excerpt from the Object Diagram of the Context Model for Example Scenario 2 - Situation 4

Full size image

After the UI is adapted back to visual modality, the current distance among the assistance system and the worker increases (1.2 m) and this condition triggers again vocal adaptation of the UI - Situation 3.

Furthermore, Situation 4 illustrates a challenge where the worker is having difficulties in recognizing a concrete piece needed for assembling the product. The Context Manager detects that the worker needs much time regarding a concrete instruction (5 min), but at the same time there is no Movement Status detected. This event triggers the UI adaptation process and for this purpose several conditions are checked. The Digital Twin informs that the Mood of the worker changed from normal to contempt. In addition, the Usage Time, Current Distance, Navigation Type, Navigation Preferences, Information Processing Preferences and Task Feature Set are evaluated. As a result of the conditions check, the layout of the graphical UI is adapted. This includes adaptation to a grid navigation and reduction of the amount of information shown on the screen. In addition, the Font Size is increased and the underlying Color Theme is adapted. Figure 7 presents an excerpt from the Object Diagram of the Context Model for Example Scenario 2 in respect to Situation 4, while Fig. 8 depicts the adaptation rule for Situation 4.

Fig. 8.
figure 8

Adaptation rule for adapting the graphical UI layout (Example Scenario 2 - Situation 4)

Full size image

6 Conclusion and Outlook

With our solution approach, we provide a novel reference model for multi-modal adaptive UIs in the context of manufacturing processes. We have demonstrated the benefit and application of our Digital Twin-Based Multi-Modal UI Adaptation Framework based on two case studies in the context of smart manufacturing. Our future work focuses on further development of the framework in terms of refinement of the Digital Twin of a Human Model, development of novel adaptation features and adaptation rules to fit into the domain of smart manufacturing.