Abstract
With the advent of technology, new interaction modalities became available which augmented the system interaction. Even though there are vast amount of applications for the ubiquitous devices like mobile agents, smart glasses and wearable technologies, many of them are hardly preferred by users. The success of those systems is highly dependent on the quality of the interaction design. Moreover, domain specific applications developed for these ubiquitous devices involve detailed domain knowledge which normally IT professionals do not have, which may involve a substantial lack of quality in the services provided. Hence, effective and high quality domain specific applications developed for these ubiquitous devices require significant collaboration of domain experts and IT professionals during the development process. Accordingly, tools to provide common communication medium between domain experts and IT professionals would provide necessary medium for communication. In this study, a new modelling tool for interaction design of ubiquitous devices like mobile agents, wearable devices is proposed which includes different interaction modalities. In order to better understand the effectiveness of this newly proposed design tool, an experimental study is conducted with 11 undergraduate students (novices) and 15 graduate students (experienced) of Computer Engineering Department for evaluating defect detection performance for the defects seeded into the interface design of a neuronavigation device. Results show that the defects were realized as more difficult for the novices and their performance was lower compared to experienced ones. Considering the defect types, wrong information and wrong button type of defects were recognized as more difficult. The results of this study aimed to provide insights for the system designers to better represent the interaction design details and to improve the communication level of IT professionals and the domain experts.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
As diagrams transfer, and leverage knowledge that is essential for solving problems, they can be more powerful than sentential representations depending on the usage [1]. Diagrams provide compressed information; hence, they are very effective in information systems for transferring information between stakeholders of the system.
The ubiquity and real time access feature of mobile agents, smart glasses and wearable technologies provide diverse interaction alternatives to people in different domains. The success of those systems is highly dependent on the quality of the interaction design. Moreover, domain specific applications developed for these ubiquitous devices involves detailed domain knowledge which normally IT professionals do not have, which may involve a substantial lack of quality in the services provided. Hence, effective and high quality domain specific applications developed for these ubiquitous devices require significant collaboration of domain experts and IT professionals during the development process. However, people from different domains have different mind sets and perceptions about the world which could create significant communication problems between them during the system development life cycle.
Hence, tools to provide common communication medium between domain experts and IT professionals would provide necessary medium for communication. There are such tools used for this purpose largely by IT professionals like UML representations which provide several graphical tools like activity, class, and sequence diagrams. They are vastly used by IT professionals for system design in which the main focus if action or the process. Therefore, a design represented by UML for a mobile application or a desktop application may result in the same design which would lack the different interaction modalities like gesture-based controls. To our knowledge, none of the available design diagrams host specific interaction characteristics of the ubiquitous devices.
Hence in this study, first a new modelling tool for interaction design of ubiquitous devices like mobile agents, wearable devices is proposed which includes different interaction modalities which is used to model interface of neuronavigation device interface that is quite critical since is used during the surgery, in operation rooms which imposes several constraints. In order to better understand the effectiveness of this newly proposed design tool, an experimental study is conducted with graduate and undergraduate students of Computer Engineering Department of Cankaya University to better understand the comprehension of the proposed tool. Background section below contains related studies found in the literature, Research Procedure section explains the experiment, Result section analyzes the experiment results and Discussion and Conclusion section talks about the insights gained through this study. The results of this study are expected to provide insights to the researchers, IT professionals and domain experts to improve interface design process of ubiquitous devices.
2 Background
There are several studies aim to improve representation languages which are commonly used in software engineering. Formal languages are a model for software requirements and its specifications but it is not adequate for particular design. On the basis of this idea, several studies have been performed on how representation languages can be used for this purpose. TERMOS [2] is a UML-based formal language that uses graphical scenarios and specifies these scenarios in test and verification activities. This is not a new approach, but the originality of this study is application of scenario-based verification in mobile computing system. TERMOS incorporates three elements that are spatial view, event view and communication events. It specifies properties that are subset of the spatial configuration nodes to taken into account spatial configuration and changes events. For this reason, several scenarios are considered that are: positive requirement, negative requirement and test purpose [2].
Vegard and Aagedal [3] define entity’s change of location or possibility of its movement in a framework by extending UML class and activity diagrams. This approach is very close to Grassi et al. [4] modelling, but the big difference from their studies is to include vertical mobility. Also they performed a case study application based on the framework to validate their profile.
On the other hand, many researchers investigate their own representation methods instead of extending or improving existing languages. They revised commonly used GUI objects and presented new GUI notation due to limitation of existing UML and the other languages. These objects allow the designer to design interfaces for software development processes and interact with each other in a composite pattern. Jose and Paternò [6] develop a mobile framework based on jigsaw metaphor for the people who have not any knowledge of development process, to help implementing and executing their own application. Puzzle allows end users to easily explore the framework and create or change their own application according to their requirements. This study purposed that include end user in development process to increase possibility of combination to add innovative view of the applications. However, the aim of the study does intent to overcome existing languages’ problems.
Concerning to healthcare software, usability problems have direct effect on patient safety [5] which is even critical for the software employed in operation room embodying several constraints. Therefore, it is important to include healthcare experts in the software development process to ensure usability, quality, effectiveness and efficiency. Effective systems can be established through collaboration of IT and healthcare professionals in development process to ensure the usability goals are met. But this reveals a new problem which is communication gap between expert judgment and IT professionals’ knowledge. To overcome this problem there are various studies in IT field such as different modelling and representation guidelines. These methods do not adequately address specific domain and still need improvement. Erturan [7] presented new representation which is called MAFR. MAFR notation consists of composite representation elements. MAFR tool is used in a case study to investigate if the new approach is more preferable than UML representation for both IT and healthcare professionals [7].
Hence, in this study, MIF tool is proposed for modelling ubiquitous environment interaction, based on MAFR. Defects were seeded into the MIF diagram of neuronavigation device interface design. Then, the defect detection process was analyzed to obtain insights about the cognitive processes of the novice and experienced participants. Mainly, three different types of defects, namely, wrong information (WI), wrong flow (WF) and wrong button (WB) have been seeded into the MIF representations. The following research question is aimed to be answered is ‘Which types of defects (WI, WF, or WB) are easy to detect in MIF representations? Is there a difference in success of defect detection process for novice and experienced participants?
3 Research Procedure
The MAFR tool [7] was proposed for mobile agent interface design to incorporate touch gesture interaction. This study extends MAFR by adding speech, eye and gesture modalities into the interface design.
We have interviewed 2 surgeons to gather their preferences of alternative interaction styles for neuronavigation device. They have stated they cannot use the device affectively during the operation as they couldn’t use their hands and have to get the help of nurse to use the touch interface of the system. This causes misunderstandings and waste of time which may create critical situations during the surgery. The MIF representation elements were used to design interface of neuronavigation device interface by the surgeons’ input through MIF elements. We have performed an experiment to observe and collect data for defect detection process of novice and experienced participants during MIF diagram review.
The experimental study is conducted with 11 undergraduate students (3rd and 4th year students) and 10 graduate students of Computer Engineering Department of Cankaya University. The MIF element explanations, neuronavigation system interface requirements and experiment design were provided to the participants prior to the experiment.
The interface was designed using MIF and 9 defects of wrong information (WI), wrong button (WB) and wrong flow (WF) were seeded to the diagram (Table 1). During the experiment, the participants were asked to find the defects and report them using the defect detection collection tool as given in Fig. 1.
This tool is used to record the defect explanations, the duration and order of the defects found by participants.
Table 2 depicts the defects seeded into the MIF with their defect types. The defect locations seeded in the MIF diagram can be reached from http://www.cankaya.edu.tr/~gtokdemir/defectlocations.pdf.
Based on the defect detection average duration and order, we have calculated defect difficulty of each defect using formula of Cagiltay et al. [8].

Formula 1. Defect detection difficulty level formula [8].
After calculation of the defect detection difficulty, each participant’s performance was calculated through Formula 2.

Formula 2. Participant’s defect detection performance [8].
Additionally, questionnaire was applied to get perceptions of the participantsFootnote 1.
4 Results
As seen in Table 3, two defects (D06 and D08) were not detected by any participant in the low-experienced group which were both WI type defects. Accordingly, as these defects were never detected, they are considered as hard to detect ones and the sum of Dfi values calculated for the other defects for this group (13075) is assigned for those. All defects were detected by the experienced group.
As it is seen from Table 3, average of DFi values for the experience group (2314) is lower than that of novices (4358) which indicate that, the defects were realized as more difficult for the novices.
There were 11 novice and 10 experienced participants involved in this study. When defect detection performance of the participants are analysed it is clearly seen that, mean value of the novices’ performance (24) is lower than that of experienced ones (30) (Table 4).
When defect difficulty levels (DFi) are analysed according to the defect types, as seen from Table 5, wrong information and wrong button type of defects are recognized as more difficult to detect compared to the wrong flow type of defects.
In the questionnaire, most of the participants from both groups reported that detecting wrong button type of defect was the hardest one which is in parallel with the experimental results. Participants from both groups mostly declared that, using the symbols and descriptions about the experimental study helped them to detect the defects easily. They reported that, since the user interface documentation used in the experiment was complex one, recognizing the symbols and buttons was a hard process for them.
5 Discussions and Conclusion
This study proposes an interface design representation for ubiquitous computation including multimodal interactions. In this respect we have extended the MAFR representation [7] which was developed for mobile applications to include mobile agent interactions. The new representation is called “Multimodal Interaction Flow Representation for Ubiquitous Environments- MIF” which includes interactions through mobile agents, smart glasses and wearable technologies in ubiquitous environments.
A case study was performed to assess the comprehension of these diagrams by designing an interface of a neronavigation device through MIF elements used in operation room by collecting surgeons’ requirements. This design is used in experimental study to assess defect detection process of participants in reviewing these diagrams. For this purpose interface description of the neuronavigation device, MIF element explanations, experiment setting description were prepared and sent to the participants prior to the experiment to let them get familiar with the context. 9 defects of type wrong information (WI), wrong flow (WF) and wrong button (WB) were seeded to MIF diagram and during the experiment, 21 students of Computer Engineering Department of Cankaya University were asked to review the given defected diagram and find the defects seeded by comparing it with the interface description. During the review process, participants were asked to report the defects they found to record their explanations, order and defect detection duration through a web based tool. Defect detection difficulty levels and participant performance values were calculated. At the end of the experiment, a questionnaire was applied to gather perceptions of the participants.
Results revealed that novice participants have never found 2 of WI type defects whereas experienced participants have found all of the defects. Overall, the defects were realized as more difficult for the novices. Additionally, novice participants’ performance was lower compared to experienced ones. Considering the defect types, wrong information and wrong button type of defects were recognized as more difficult to detect compared to the wrong flow type of defects by both groups which was supported by the questionnaire answers. Moreover participants stated that using the symbols and descriptions about the experimental study helped them to detect the defects easily. The results of this study are expected to give insights to future research in ubiquitous computing interactions. As a future study we will compare the effectiveness and efficiency of the proposed MIF representation with the available languages like UML and repeat the experiment with surgeons to get their perceptions about the MIF elements.
Notes
- 1.
Available at http://www.cankaya.edu.tr/~gtokdemir/questionnaire.pdf.
References
Larkin, J.H., Simon, H.A.: Why a diagram is (sometimes) worth ten thousand words. Cognit. Sci. 11(1), 65–99 (1987)
Waeselynck, H., Micskei, Z., Rivière, N., Hamvas, Á., Nitu, I.: TERMOS: a formal language for scenarios in mobile computing systems. In: Sénac, P., Ott, M., Seneviratne, A. (eds.) MobiQuitous 2010. LNICST, vol. 73, pp. 285–296. Springer, Heidelberg (2012)
Dehlen, V., Aagedal, J.Ø.: A UML profile for modeling mobile information systems. In: Indulska, J., Raymond, K. (eds.) DAIS 2007. LNCS, vol. 4531, pp. 296–308. Springer, Heidelberg (2007)
Grassi, V., Mirandola, R., Sabetta, A.: A UML profile to model mobile systems. In: Baar, T., Strohmeier, A., Moreira, A., Mellor, S.J. (eds.) UML 2004. LNCS, vol. 3273, pp. 128–142. Springer, Heidelberg (2004)
Johnson, C.W.: Why did that happen? Exploring the proliferation of barely usable software in healthcare systems. Qual. Saf. Health Care 15, 76–81 (2006)
Danado, J., Paternò, F.: Puzzle: a mobile application development environment using a jigsaw metaphor. J. Vis. Lang. Comput. 25(4), 297–315 (2014)
Erturan, Y.N.: A method to improve the communication between information technology and healthcare professionals during mobile healthcare application development process. M.Sc thesis (2013)
Cagiltay, N.E., Tokdemir, G., Kilic, O., Topalli, D.: Performing and analyzing non-formal inspections of entity relationship diagram (ERD). J. Syst. Softw. 86, 2184–2195 (2013)
Acknowledgements
This research is conducted for the Surgical Navigation Project (CAN) which is supported by TUBITAK (113S094). The research team would like to thank TUBITAK support for realizing this research study.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Tokdemir, G., Altun, G., Cagiltay, N.E., Hakan Maras, H., Borcek, A.O. (2015). Multimodal Interaction Flow Representation for Ubiquitous Environments - MIF: A Case Study in Surgical Navigation Interface Design. In: Kurosu, M. (eds) Human-Computer Interaction: Interaction Technologies. HCI 2015. Lecture Notes in Computer Science(), vol 9170. Springer, Cham. https://doi.org/10.1007/978-3-319-20916-6_73
Download citation
DOI: https://doi.org/10.1007/978-3-319-20916-6_73
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20915-9
Online ISBN: 978-3-319-20916-6
eBook Packages: Computer ScienceComputer Science (R0)