2D and 3D Iconography on Augmented Reality Interfaces

Abstract

Given the increasing amount of studies about augmented reality environments and interface construction for this type of applications, the adequate use of icons in this context is gaining more and more relevance. The new forms of interaction used in augmented reality originate changes on the maneuver of digital elements, among them the iconic ones. This paper aims studying the use of icons in digital environments and discussing their correct use in augmented reality environments.

1 Introduction

This work aims to discuss the relation between the use of two and three dimensional (2D and 3D) icons in highly realistic environment interfaces, and the consequent factors of the choice for any of these alternatives. Beyond, the possibility of hybrid 2D/3D interfaces construction is analyzed.

The use of icons in digital interfaces is certainly a vastly studied research field. There are methods of construction and analysis directly developed to test interface efficiency and functional appropriateness [1, 2]. However, in augmented reality (AR) environments, the real world image is overlapped with digital information, which causes a change on the perception and manipulation of icons [3, 4]. This becomes even more evident at the moment that non-traditional types of visualization and interaction interfaces are used, like HMDs (Head-Mounted Displays), data gloves and others found in mobile platforms [5, 6]. To comprehend the impact of iconography in this mixed interfaces is necessary to analyze the functions and relations that these icons will be representing.

Nowadays, studies about interaction in AR environments have shown that those ambient present some differences when compared to the methods used in virtual reality (VR) applications. These differences cause some variations on the criteria used to construct an adequate iconography for these interfaces and on how they are noticed by the user [7]. To better comprehend these differences and how they affect the icon definition process and its use in augmented reality is the main objective of this paper. Based on the research performed by the authors, this work outstands due to its focus on iconography in AR, which has not been largely explored until now.

2 Background

2.1 Theory of Icons

Icons are widely interpreted as images that have a relationship of similarity (an abstraction) to something known, and possessing at least one detail in common with the represented object. This concept makes icons the key to transmit information successfully [1].

Pierce [8] defines icons in his second tricotomy, relative to representamem, as the most basic way of representing metaphorically the reality, being directly related to what is defined by him as the firstness of the information transmission process. As stated by Pierce, icons bring with themselves characteristics similar to the object or information they symbolize.

The term icon, however, is generally used to represent any reality or information metaphorically. This term’s generalization is even more used when the main focus is the use of icons on digital interfaces [9].

2.2 Icons on Digital Interfaces

The ISO/IEC standard, specifically the ISO/IEC 11581-1:1995, specifies a definition of icons and their use on digital interfaces. Based on this definition, icons are elements of a distinct language supported by a metaphorical representation through pictograms. The icons differ from other screen elements because of the fact they can represent objects, markers, tools and controls in order to allow their manipulation by users. Icons also have the communicative function of giving the status or performance of activities, transmitting the information necessary to the ideal fulfillment of the task [2].

The most common ways of utilizing icons on digital interfaces is their use with buttons, alert signals and object representation (e.g., icons representing files and folders) [9], as shown in Fig. 1.

Fig. 1.

Examples of icon representation: (a) action; (b) object; (c) status

Icon conception is based on heuristic parameters related to legibility, clarity and level of understanding. Since it is a kind of illustration, its construction is oriented by some graphic elements such as style, color, contrast, texture, luminosity and volume, taking into consideration the type of function it is wished to represent and the interface’s project context: targeted audience, functionality and historical precedents [1].

3 Icons on AR and VR Environments

In AR the interfaces construction process becomes more complex, because of the possibility of physical and virtual manipulation, icon reaction and multiple user interaction with the virtual environment. All of these elements must be considered in this type of interface [10, 11].

3.1 Adequacy

Skogen [12], in her pilot studies, performed an experiment in order to define the ideal level of complexity to be used in the creation of a digital interface icon, maximizing its comprehension. As a result, it became clear that the use of aesthetically more elaborated icons that have a higher number of familiarities to the desired objects or actions presented better performance during task’s realization. This proved that the higher the abstraction, the less efficient will be the communication provided by icons.

As shown in Fig. 2, Skogen positioned a set of icons in four different quadrants over a board, according to their level of complexity and familiarity. On this board, miniatures of the icons evaluated by Skogen were also placed in one of the four quadrants in order to verify which of them obtained the best results on a user based research realized earlier.

Fig. 2.

Example of icon qualification by Skogen [4]

The methodology used in this study allows evaluating the ideal level of abstraction that should be used in the construction of more adequate iconographic projects. However, the work developed by Skogen deals only with the association between icons and corresponding represented objects, not taking into consideration the complete interface and aspects such as circumstances and user motivations for icon manipulation. From this point of view, Skogen’s evaluation is essential for icon construction, but must be extended with further actions in order to succeed when using these techniques on complex interface systems like the AR ones. Concepts related to user needs and motivations must be demonstrated and will be discussed later.

3.2 Activity

Ark [13] shows in his experiments the relationship established between the use of 3D and 2D icons in digital interfaces, and also in his “ecological” interfaces. He has defined this kind of interface as being interfaces prepared to simulate environments or specific situations. In his conclusions, the choice of the icon type affected directly user’s search and reaction time, and the 3D icons were perceived by the system’s operator with more difficulty. However, the 2D icons’ level of understanding and comprehension were revealed smaller when compared to 3D ones, as illustrated in Fig. 3. In this figure can be observed that even when the results on the second day are smaller than the ones on the first day (thanks to the familiarization obtained with the interface), the relation among the types of icons stays the same.

Fig. 3.

Performance results obtained by Ark during his experiments [6]

Based on Ark’s results, it can be deduced that the choice of icons must be done according to the type of functionality that will be represented by them. Static icons that have operational function, such as the ones which represent buttons and objects, can be used in their 3D form, since they do not require a significant small response time. However, icons indicating alerts or some information must be perceived as fast as possible by the user, and for this reason the 2D representation is more adequate.

3.3 Manipulation

Smallman [14] also presents in his studies results similar to the ones found by Ark. However, he presents some new arguments in favor of the capabilities inherent to 3D icons when it comes to represent a greater amount of information, since they can assume a higher number of functions or object characteristics. Besides the fact that 2D icons’ search time is smaller than 3D ones, the communication level of a 3D representation works better when the goal is to transmit a higher amount of information, like the example illustrated in Fig. 4 where the icon itself has information related to direction, orientation, behavior, and airplane model.

Fig. 4.

Icon model presented by Smallman

In the same way that concepts like user needs and motivations were highlighted in Skogen’s work, it is fundamental to point out in Ark’s and Smallman’s work the importance they gave to iconographic system suitability to the task of the application interface at hand. In the case of AR interfaces, to understand this suitability requires comprehending the environment’s behavior as well as the data that will be digitally inserted on it.

4 AR Interaction and the Use of Icons

Different from what occurs in VR, AR has not yet its interaction techniques completely explained and classified. One of the reasons that justify this lack of classification could be the fact that the majority of VR interaction techniques can be used in AR applications without problems.

Broll and his colleagues [8] recently proposed a new classification of interaction techniques specifically for AR environments. They have stated that 3D or VR interaction techniques cannot be directly applied to AR. Therefore, they have subdivided different interaction techniques into spatial interaction, command-based interaction, virtual control interaction and physical control interaction.

Despite this classification includes a great part of interaction techniques used in AR, it shows itself insufficient and is not widely accepted by scientific community.

The construction of AR interfaces presents some important questions related to the use of an adequate iconography. Many authors have studied and debated these questions in wider contexts. In this section, it will be discussed questions targeting perception in a 3D environment, suitability in ecological environments, like the variations presented in Ark’s ecological interfaces and the possibility of using real icons.

At first, interaction in 3D immersive environments already adds to icon construction a variety of possibilities not applicable to 2D environments. For example, volume and perspective are two characteristics that considerably alter the perception of the constructed icon. This type of ambient allows direct interaction to the iconic objects, since the user does not need to click on the icon anymore; now, he/she can press, activate by direct looking or simply walking near the icon.

This variation in icon manipulation enables the icon to have not just one function, but a series of them, since the function can vary according to the way the manipulation occurs [15]. The next point that deserves attention in both VR and AR is the way icons can adapt themselves to the augmented environment and be perceived by the user. In his study about icons in virtual environment interfaces, Ark [13] presents that, beside the fact that 3D icons are more suitable when applied to an ecological interface ambient having a 3D basis, they suffer from a low search response time, if compared to 2D ones.

By bringing this problem to the context of AR environments, a question emerges about these icons visualization and adaptation in the world. It is clear the need of highlighting icons presence in the virtual environment through the use of differential rendering, positioning or shape and volume variations. This could decrease user’s search time and enhance the application’s functionality.

According to the interaction technique used, there is a variation in icon manipulation and its functionality, as shown in that exemplifies the mivaDesk application (see Sect. 4). The interaction techniques followed the classification suggested by Bowman. Each one of the interfaces adopts different types of iconography depending on the interaction technique used (Fig. 5).

Fig. 5.

Examples of AR interaction techniques and the presence of icons on each interface: (a) gesture manipulation and 3D widgets (mivaDesk menu); (b) walking navigation (mivaDesk idle interface); (c) system control (mivaDesk file transfer interface); (d) symbolic input (mivaDesk soft keyboard)

At last, the use of an AR ambient focuses the attention to a brand new possibility related to information representation: the use of real icons. As explained in Sect. 2, basically, icons are representations of real objects or information familiar to users. Because of that, it is hard to a user that is already accustomed to some specific system or culture to comprehend some new interface constructed. In AR, however, it is possible to link information or even activities to an object or real phenomena: a real object from an AR interface can become the iconic reference to some system’s action (e.g., a product can be turned into a shortcut to an ingredient sheet and its nutrients table or stock management). These possibilities are based on recent researches and studies, like 3D reconstruction from 2D images and markerless AR. The last one is a relative new concept. It utilizes many computer vision algorithms to extract image characteristics, which could be transformed into 3D reference points. A set of these points gives origin to a mapping of the real scene, and can be used to position the desired virtual objects through a previously chosen viewpoint.

5 Discussions

Based on iconography construction and functionality principles defined in previous sections, related to generic interfaces and specific AR ones, some considerations can be made. Based on the studies shown, it is possible to enumerate three important points about the use of 2D and 3D icons in AR environments: manipulation, activity and adequacy.

The first of them, manipulation, helps to evaluate the ways of interaction that each interface must support, how the user can interact with the environment and its elements, how visualization occurs, and what are the available possibilities of element control by the user. Section 3.3 highlighted the importance of user necessities and task objectives in an iconography project. This stage provides essential information to be used during icon creation in order to easily control the interface. The creation of application scenarios through the use of history boards and watched simulations allows highlighting specific interface manipulation cases and some iconography needs. Through these procedures, it is possible to list all the necessary iconography related to the type of interaction available to system users.

In sequence, activity is related to the function icons will perform. It is necessary to define adequately which functions and under which circumstances each one of them will be used (equipment used and interaction techniques adopted), and what is needed for their ideal performance. Response time, visibility and importance factors will be the start for iconographic planning of each interface used. Through them it will be possible to make decisions and to define methods for evaluation and later icon performance check. In order to optimize this process, it is recommended the construction of a table containing each icon function and technical characteristics needed, based on interface objectives and flowcharts.

At last, the icons adequacy to the targeted applicable environment and AR ambient must be verified, in order to better adapt them to their functions and usage methods. After this verification for a specific interface, it is possible to elaborate graphical aspects for icon construction: if the icon must have 2 or 3 dimensions, which shape is indicated, color, style, contrast, volume etc. Also referred to adequacy, it is necessary to observe the set of experiences of potential system users to select the most efficient graphical iconic language that communicates to these individuals.

By using these highlighted points on icon utilization in AR interfaces, a solid base of procedures for obtaining data related to their construction can be established. These procedures are summarized in Fig. 6, which shows an example of methods table for evaluation of distinct interface aspects.

Fig. 6.

Example of a group of procedures to be used for icons development, based on manipulation, adequacy and activity aspects.

This procedure of the methodological definition of data acquisition can be applied to any type of digital interface. However, in case of AR interfaces, in order to minimize the number of problems related to user interaction, this action becomes vital for the project success. In Fig. 7, it is possible to observe the general scheme of procedures that recover data supporting interface iconographic projects.

Fig. 7.

General diagram showing relations between manipulation, activity and adequacy

The selection of which aspects must be evaluated and the choice of methods and techniques which will be used in data acquisition should vary from case to case, hence providing the project team with a major amount of relevant information. The methods and techniques listed in Fig. 6 can be found in Baxter [16], as well as in other works related to this research area [6, 17, 18].

Through the use of this methodology, an optimization on the construction of iconographic systems for AR interfaces is expected. The retained methodology considers project orientations based on a more complete comprehension about the usage circumstances of these interfaces: “Who will use it?”, “Where are they used?”, “What are they used for?”, “In which way are they used?”.

6 Conclusions

The use of the methodology and concepts detailed in this paper proved to be relevant during all design processes and lead to important decisions about the application interface regarding the 2D and 3D icons used. The understanding of the impacts of wrong iconography in VR and AR enviroments can bring to the community more control over the experience provided and allow better results in cognitive and functional perception from the users.

As future work, a case Study based on VR interfaces will implemented. Some usability and performance tests should validate the final product an check the benefits of methodology.