Keywords

1 Introduction

Having a look on today’s occupations, there are lots of reasons why people work. The primary objective as well as one of the most important ones is earning a living for themselves and their families [1, 2]. People work for a wage. They are eager to use it to shape their current lives including the time away from work (e.g. leisure or vacations) as well as their future (e.g. retirement). Further objectives include to be part of the society, to create an own identity and to keep one’s own dignity. Work is vital for everyone and for the society as a whole. It allows people to participate within a community, to find contacts and to communicate. The content and the outcome of their work is important for them, too. They become able to use their talents, to learn and to develop their skills. In addition, recognition, reward, knowledge as well as sharing ideas are features people are longing for at work [3], issues that are closely related to user experience [2]. If people take pride in their work, they may find it a very enjoyable part of their lives and they may improve the quality of their work likewise.

Looking upon the topics that shape the future of work, we can observe five big trends: New behaviors, millennials in the workplace, mobility, globalization, and technologies [4]. Today, employees are eager to live more public lives. That means we build online communities, share ideas, and collaborate on a common knowledge base. These new behaviors lead to new requirements the organizations we are working for need to fulfill. In addition, this aspect will be reinforced by the huge number of millennials – the “Generation Y” with its technological fluency – which will enter the job market by 2020. Globalization foster open boundaries to work with anyone and mobility increase the flexibility to work anywhere and with any device. Nevertheless, the main driver of innovation for these changes is “software”.

The German term “Arbeit 4.0” (en. “Work 4.0”) addresses the direct and indirect interaction between digitization and all new processes of work design, organization of work, and working conditions [3] as well as training and qualification. Software solutions especially digital assistance systems are increasingly used within the professional work context [5], e.g. cooperative robots in industrial manufacturing or augmented reality glasses in commissioning. The possibilities are versatile, but they also encourage further discussions about the impacts on the working life of humans. The employees’ as well as organizational acceptance of these technologies is crucial and includes quality aspects of usability, user experience, and Work 4.0.

Despite all efforts, todays software products still lack of quality with regard to functionality and usability [6, 7]. Thus, current software engineering (SE) methods as well as software quality models seem to be insufficient. Methods don’t focus on end users – the employees working with the software – in terms of active involvement during the development process. Furthermore, quality models lack focussing on work and user experience. The topic of usability is already included within newer quality models, e.g. ISO/IEC 25010 [8], but its concepts are communicated in a way that foster imprecise interpretations. A more human-centered quality model will help to integrate the human focus in quality discussions about systems to be developed, to select adequate methods for user participation, and to shape a common understanding about user goals and needs.

In this paper we present an approach that defines a concept for a human-centered acceptance quality model. Furthermore, we conducted contextual inquiries within three projects with industrial partners to elicit acceptance criteria for digital assistance systems within industrial manufacturing. Based on these criteria implications for a review of the software quality model of ISO/IEC 25010 are formulated.

2 Background

In order to create a quality model that focusses on acceptance and that is based on an individual human-centered and organizational business-centered perspective it is important to create a common understanding. This paper focusses on digital assistances systems within industrial manufacturing. Therefore, digital assistance systems are defined in the following subsection followed by a brief introduction to technology acceptance of these systems. Afterwards, existing quality models are discussed as a base for an acceptance quality model.

2.1 Digital Assistance Systems

The aim of digital assistance systems is to provide employees with the information they need just as quickly and easily as possible at any time and from anywhere. Assistance systems include all technologies that assist the employees in carrying out their work and enable them to concentrate on their core competencies. These are, in particular, technologies for providing information, such as visualization systems, mobile devices, tablets and smart glasses or tools which perform calculations. This ranges from the simple display of work instructions via visual or multi-medial support (e.g. picking systems) to context-sensitive augmented reality for the employees [10].

Fig. 1.
figure 1

Degrees of digital assistance (example of industrial picking)

Full size image

In comparison to the actual analog situation, digital assistance can be distinguished into three levels of digitization: The digital copy, the digital innovation and the digital revolution (see Fig. 1).

The “digital copy” represents an accurate reflection of a none-digital reality about something already existing. In industrial picking, the digital copy could be an online checklist on a tablet that looks exactly the same as the checklist on a paper-based clipboard. The next level of “digital innovation” adds something new to the already existing digital copy. This could be a dynamically ordered online checklist based on a big data analysis about the dependencies of individual check items. The highest level of a digital assistance systems is the “digital revolution”, something completely new in the digital world that still does not exist in the real world and implies a disruption of structures, workflows or tasks (cf. [11, 12]). This could be a pick-by vision system with a head-mounted optical see-through display, also known as augmented reality (AR), where digital information are presented directly within the real work environment. Out of scope are “human substitutes”, complete autonomous systems without any human intervention.

The higher the digital disruption (e.g. compared to the previous value system, complexity), the more critical is the acceptance of the software solution on the part of the organization and especially on the part of the employees [13].

2.2 Technology Acceptance

The acceptance of digital assistance systems or technology in general is crucial for the successful introduction and use within an organization. Venkatesh and Bala [14] proposed the concept of the “Technology Acceptance Model (TAM) v3” that is mainly described by the two aspects perceived ease of use & perceived usefulness, which affect the aspects of the behavioral intention & use behavior.

  • Perceived ease of use is defined as “the extent to which a person believes that using an IT will enhance his or her job performance”.

  • Perceived usefulness is defined as “the degree to which a person believes that using an IT will be free of effort”.

  • Behavioral intention is defined as the prediction of how the users will behave with a system based on the perceived ease of use and perceived usefulness.

  • Use behavior describes the actual behavior during the usage of a system.

These aspects are influenced by multiple variables (e.g. computer self-efficacy, computer anxiety, computer playfulness).

Regarding the development of digital assistance systems these variables are either shaped by the system itself or by the organizational culture. Thus, there are three interconnected factors that assure the acceptance: Process, people and product. Involving all necessary stakeholders right from the start during the conception and development of a system influences the acceptance through out the process. Creating a social and cultural environment where employees like to work and to talk about their problems and their ideas influences the people in their openness. Creating an assistance system (product) with an appropriate usability and user experience (UX) influences the employees through the system itself.

Comparing the definitions of usability, its three attributes, and user experience

  • Effectiveness: “Accuracy and completeness with which users achieve specified goals.” [15]

  • Efficiency: “Resources expended in relation to the accuracy and completeness with which users achieve goals.” [15]

  • Satisfaction: “Freedom from discomfort, and positive attitudes towards the use of the product.” [15]

  • User Experience: “Person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service” [16]

with TAMs definitions of perceived ease of use and perceived usefulness we can see the close relation between technology acceptance, usability and user experience. Therefore, usability and user experience represent fundamental quality influences on technology acceptance. Other influences on technology acceptance are technical product qualities like reliability, security or performance efficiency, etc.

2.3 Quality Models

As software quality has an influence on the acceptance a further look has to be taken on system and software quality models.

Software in its nature is abstract and intangible. Shewhart [17] early divides the quality of manufactured products into an objective as well as subjective side:

“There are two common aspects of quality: One of them has to do with the consideration of the quality of a thing as an objective reality independent of the existence of man. The other has to do with what we think, feel or sense as a result of the objective reality. In other word, there is a subjective side of quality.”

Based on this, researchers and practitioners have tried to make the benefits and costs more visible for measurement proposing various characterizations for the objective side of software quality [18,19,20].

McCall et al. [18] presented one of the eldest known quality models developed at General Electrics. They identified three major perspectives: Product revision (ability to carry out changes), product transition (adaptability towards new environments) and product operations (its operation characteristics) including 11 quality factors (e.g. maintainability, correctness, portability) and 23 quality criteria (e.g. traceability, storage efficiency).

Boehm et al. [19] described three primary uses in their quality model: As-is utility (extent to which the software can be used), maintainability (ease of modification and retesting) and portability (adaptability towards new environments). Further on they presented seven quality factors (e.g. portability, reliability, efficiency) that are classified into these primary uses and that are defined by 15 criteria (e.g. accuracy, completeness, consistency).

Grady [20] defined five characteristics known by the abbreviation FURPS, which stands for functionality, usability, reliability, performance, and supportability. These characteristics are of two different types: Functional (F) and non-functional (URPS).

These quality model have lots of similarities and mainly differ in some of their factors. A common pattern in all of these quality models is their hierarchy:

Characteristics \(\rightarrow \) factors (ext. view) \(\rightarrow \) criteria (int. view) \(\rightarrow \) metrics

Nevertheless, the value of these models is a pragmatic one and doesn’t exist in the presence of more or less factors. An accurate measurement of a software will determine its value. One major challenge from a human-centered perspective remains in the separation of functionality and usability. This results in a technology-centered development of software, where usability is just the “user interface manicure” instead of user needs defining the necessity of functionality and visualization of information [21].

A newer standardized quality model exist in ISO/IEC 25010 [8]. Two types of quality models are defined in the standard (see Fig. 2):

  • The quality in use model consists of “five characteristics related to outcomes of interaction with a system”.

  • The product quality model includes eight characteristics describing the fixed quality of a product.

Fig. 2.
figure 2

ISO/IEC 25010 quality factors

Full size image

The difficulties of this model from a human-centered perspective is the misleading, imprecise use of the usability concept. In this technical view, usability is focused again on user interface aesthetics, but also extended with some sub factors that have some similarities with usability dialogue principles [22], e.g. learnability or user error protection. Regarding ISO 9241-11 [15] usability is defined as the “extend to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. Following this definition the factor usability in the product quality model is misleading, because usability is also included in the factors effectiveness, efficiency and satisfaction that are used within the quality of use model.

Furthermore, effectiveness from a usability point of view specifies the functional appropriateness of a software along the users’ tasks. Functional appropriateness is a sub factor of functional suitability in the product quality model. Thus, usability is not just a quality requirement, but also closely related to quality in use and functionality.

Apart from that, the terms effectiveness, efficiency, satisfaction, freedom from risks, and context coverage are too abstract as they vary in the interpretation of the respective perspectives. For example, using the term satisfaction the human-centered perspective would think of “hedonic and pragmatic satisfaction of the users” while the business-centered perspective would think of the “management satisfaction” and the technology-centered perspective of the “support and maintenance satisfaction”.

In addition, ISO/IEC 25010 is missing the dependencies that exists between various factors. For example, satisfaction in terms of the user’s trust of a system has a causality with the technical factors confidentiality, integrity, and reliability.

One of the reasons for often unsuccessful software products or software products that only met parts of users’ needs may be the limited focus on technical quality criteria (e.g. functionality, reliability, maintainability, etc.). From an organization’s point of view, the success of a software is determined by the quality of the interplay between the different success criteria from the three points of view: human-centered quality (individual acceptance), business-centered quality (organizational acceptance), and technological-centered quality (technical acceptance).

3 Human-Centered Acceptance Quality Model

Creating a human-centered acceptance quality model for digital assistance systems existing system and software quality models have been analyzed in a first step. The aim was to identify if human-centered quality factors are already existent and how current models are structured. Results were discussed within an expert-based German UPA working group on quality standards as well as within a practitioner workshop at the German HCI conference “Mensch & Computer”.

Further on, a meta model to specify a quality model for acceptance criteria was built. Different approaches on classifications for usability, user experience and Work 4.0 (e.g. ISO/DIS 9241-220 [2, 9, 23]) had been analyzed to identify an initial set of acceptance criteria. These criteria had been formulated according to the defined meta model and had been used as a foundation for the individual as well as the organizational perspective to add on to the technological one. To consolidate this initial set of acceptance criteria, a focus group had been established that consists of twenty participants (executives as well as work council members) of various industrial manufacturing companies. The focus group got different questions to set up own acceptance criteria from their perspective. Afterwards, this set of criteria was compared to the initial set.

Using three current projects with industrial partners the consolidated set of acceptance criteria had been used in the requirements elicitation stages of assistance systems to be developed. Two projects used the acceptance quality model to discuss their vision and different partial conflicting requirements. Contextual Inquiry has been used within these projects to elicit problem scenarios and user needs, and to specify user requirements. These requirements were mapped to the acceptance quality model to argue about the priority of individual requirements and to decide about conflicting requirements. The third project was used to validate the model within a user study. Therefore, a paper-based survey was developed and distributed among 180 assembly and quality assurance employees to evaluate the importance of the acceptance criteria. In addition, 24 employees participated within semi-structured interviews.

Based on this knowledge, also ideas for adapting ISO/IEC 25010 [8] with a more human-centered perspective had been formulated.

3.1 Meta Model for Acceptance Criteria

As mentioned previously, a meta model for an acceptance quality model had been built to describe the elicited acceptance criteria and their dependencies (see Fig. 3). The model adapts the hierarchy described in Sect. 2.3 (characteristics \(\rightarrow \) factors \(\rightarrow \) criteria \(\rightarrow \) metrics) to

acceptanceCategory \(\rightarrow \) qualityDimension \(\rightarrow \) acceptanceCriterion \(\rightarrow \) metric

Fig. 3.
figure 3

Formalized model to describe acceptance criteria

Full size image

The focus of this meta model is on the acceptance criterion itself. Every criterion can have further sub criteria or can likewise be a sub criterion of another one. As there are lots of dependencies between different criteria and perspectives a criterion can also be influenced by another criterion, can influence another criterion itself, or can have a conflict with another criterion.

In addition, each criterion belongs to one acceptance category. This can either be the individual acceptance from the human-centered perspective, the organizational acceptance from the business-centered perspective, or the technological acceptance from the technology-centered perspective.

Next, each criterion belongs to one or more quality dimensions. The quality dimensions describes the nature of a criterion as it can have multiple topics. Initially, a set of five dimensions has been set up including usability, user experience, Work 4.0, business, and technology.

In addition, each criterion has at least one metric to measure the degree of its fulfillment.

As there are also criteria which cannot be influenced by the digital assistance system itself, but by a change in the organizational culture, a further attribute had been added that describes the achievement influences of a criterion.

Finally, a visual representation for the meta model had been developed to use acceptance criteria within focus groups, workshops, or discussions (see Fig. 4). A criterion is represented as a rectangle. Its acceptance category (I, O, T) and achievement influencer (S, C) are on the left side of the rectangle whereas the quality dimensions (U, UX, W, B, T) are on the right side. Dependencies are represented with either a directed line with an arrow (influencedBy) or undirected line with a lightning (conflictWith). Sub criteria are connected with an arrow whereby the open arrowhead points to the parent.

Fig. 4.
figure 4

Visual representation of the acceptance quality model

Full size image

3.2 Acceptance Criteria for Digital Assistance Systems in Industrial Manufacturing

The objectives for an acceptance quality model for digital assistance systems are manifold:

  • Common language and understanding in a project team: As project teams are often interdisciplinary it is necessary to provide a common big picture of the different terms and goals (e.g. the different interpretations of satisfaction as discussed in Sect. 2.3).

  • Common prioritization of quality goals: A common quality model based on acceptance criteria shall enable the project team to agree on a common prioritization in order to define the scope of the assistance systems.

  • Elicitation of requirements: Acceptance criteria can either lead to further requirements that are implicitly assumed or existing requirements can be structured along their perspective (human, business, technology) and related acceptance criteria.

  • Predictable effects: Effects on the acceptance of a system caused by decisions for example made about workflows or the user interface can be predicted along the acceptance criteria or their dependencies.

  • Support method selection: Having a common and prioritized set of acceptance criteria shall support the selection of appropriate methods used within a project, e.g. usability and UX methods.

  • Metrics for evaluation: Related to the previous objective metrics can be derived from the acceptance criteria and appropriate evaluation methods can be selected.

TAM [14] as well as ISO/IEC 25010 [8] had been used as a base for the identification of initial acceptance criteria for digital assistance systems in industrial manufacturing. Further acceptance criteria had been elicited within workshops with practitioners from the management and work council of industrial manufacturing companies. Due to the limited space only an excerpt of the current generalized and expandable set of acceptance criteria is shown in Table 1.

Table 1. Acceptance criteria for digital assistance systems (excerpt)
Full size table

3.3 Evaluation

As mentioned previously, workshops with practitioners had taken place in order to expand as well as to consolidate the specified acceptance criteria. Three current projects with industrial manufacturing had been used to evaluate the feasibility of the acceptance quality model.

  • Case 1: Energy-efficiency installation: The organization of this case advises, installs and maintains combined heat and power (CHP) units of all sizes for various properties. Therefore, they are kind of responsible for the energy-saving result. However, the decision for an optimal CHP is very complex and depends on the interaction of all the energy consumers of a property as well as the people working there. In order to demonstrate the actual energy savings as well as to interpret deviations in the behavior and to plan maintenance work ahead, it is necessary to analyze and interpret a large number of energy data. The aim of a project was to introduce an assistance system to support all actors from the craftsman through the energy engineer to the owner of the property.

  • Case 2: Facade engineering: The organization of this case is active within the context of building construction and supports their customers (e.g. architects, engineers, constructors) with free of charge services in order to convince them to buy their building elements later on. Some stand-alone solutions exist that supports the involved people with some assistance. With the aim to improve the overall construction planning and coordination process and to connect existing solutions, a project has been set up to analyze the context of use within the company (sales, engineering, consulting) as well as with the users of their services.

  • Case 3: ATM assembly: The organization of this case produce all different kinds of self-service systems, e.g. automated teller machines (ATM). The automated production of the parts needed is followed by a manual assembly line. Multiple quality gates within this assembly line ensure the quality of the products. As the number of produced machines is low within each order due to individualization by the customer, quality problems are reported at a time when they cannot be fixed anymore. A project has been set up to identify possibilities for digitalization of paper-based documentation and quality assistance for the worker.

In the first two cases the model had been used to discuss about consequences on changing the workflows and to agree upon the priorities of requirements. The organizations decided to involve their customers as well as their own employees and users. Thus, the integration of usability research methods in the existing software engineering processes were comprehensible for all project team members.

Additionally, 180 employees (assembly workers, tester, and supervisors) in case three participated in a paper-based survey. First, they were ask about their current tasks and which are criteria for their acceptance of a digital assistance system. Then, they were ask to have a look on a list of presented criteria. They had to mark criteria that are of importance for their individual acceptance as well as for the organizational acceptance in case of the supervisors. Based on this results, the list of acceptance criteria had been extended with 13 new criteria.

3.4 Implications for ISO/IEC 25010

Taking a look back on ISO/IEC 25010 [8] software quality seems to be more complex. Currently, the quality in use and product model have a technology-centered perspective. Both quality models will need to take two further perspectives into account adding also the human-centered and business-centered perspective.

Therefore, in the quality in use model the characteristics “effectiveness”, “efficiency”, “satisfaction”, and “freedom from risk” should have all three perspectives, because the general terms are interpreted in different ways and hinder communication. For example, efficiency in a human-centered perspective means “having short navigation ways”, “getting responses from the system in appropriate times”, “having a fixed status of entered data while going back and forth in the process”, etc. From a business-centered perspective efficiency means “cost efficiency in terms of money” or “restructuring working time due to decreased capacity utilization of employees”. A technology-centered perspective might think of “support efficiency” or the “processing time on a CPU”. The other way round, this means that from an overall Work 4.0 perspective the saved time due to better efficiency shouldn’t have an impact on work intensification for the employees. Thus, the positive quality characteristic of business-centered efficiency might lead to a negative quality characteristic of human-centered satisfaction. Hence, the employee might reject an inadequate assistance system even if the quality characteristic of efficiency seems to be good from the business perspective. Dependencies between quality characteristic should be added.

Furthermore, usability in the product quality model only refers to factors of the user interface and may be renamed to it. As already noted, usability is even bigger than just the visible outcome known as “user interface”. Usability also includes the human-centered view of the quality in use model or the quality factor “functional appropriateness” in terms of suitability for the task. Thus, usability should not just be a quality factor, but kind of a layer that includes quality factors from both models. The same argument exists for the term of user experience as an even more comprehensive term than usability.

4 Conclusion and Outlook

Summing up, the aim of this paper is to enrich the communication and the integration of human-centered activities in software engineering methods. Therefore, a human-centered acceptance quality model for digital assistance systems in industrial manufacturing has been presented that includes individual, organizational as well as technological acceptance criteria. This model should support a common language and understanding within a project as well as common prioritization of quality goals, and common metrics for the evaluation of digital assistance systems.

Current software quality models were analyzed regarding their focus on human-centered quality factors. Multiple workshops and focus groups were run to extend and continuously consolidate an initial set of acceptance criteria. Projects with industrial partners has been used to evaluate the feasibility of the model.

Current insights confirm the involvement of all stakeholders (project team, employees, work council, researcher) during context of use analyses and requirements discussion. As the projects are ongoing, no statement can be made about the final acceptance of the assistance systems. But the acceptance within the process has been increased and the consideration of all needs shall influence the later acceptance of the ready-to-run systems.

Future work will contain further evaluations of the model and the impacts on the development processes likewise.

In addition, managers are eager to measure the outcome of change. The current model already contains the concept of metrics, but not every acceptance criteria is documented with an appropriate metric yet. This will be the next step of research.

Finally, the model has to be integrated into requirements management concepts to foster the prediction of positive and even more important negative impacts of digital assistance systems on workflows and work organization and, therefore, on Work 4.0.