Keywords

1 Introduction

The use of mobile devices such as smart phones improved the way people interact and use information and services previously available just for traditional desktop computers. New tools and services are being developed every day in order to support users performing several daily tasks [6]. Thus, a great amount of users that could be considered traditional (people who does not know advance science computer) and/or specialists, use mobile applications for many reasons that previously could just be achieved using a desktop computer.

Actually it is not wrong to say that companies that desire to provide information systems platforms for their customers or any kind of initiative that intends to use software as tool for supporting, must consider having mobile application as a product [3, 6].

However, mobile application development process is not a simple and/or trivial activity. New challenges for developers have arisen such as: (1) adapt the content to screen size; (2) power of memory and process of mobile devices is lower than desktop computers; (3) considerations about where the devices are, as, usually, mobile users are in external area; (4) form of operation; and (5) interaction features [3, 6, 14].

Some other features that explain why mobile user interface and interaction are different from desktops and require care in several aspects include: content; consistence; screen orientation; minimum work load; and customization [6].

To verify whether the mobile interfaces are well developed, and so users can achieve their goals, some usability tests can be conducted.

Performing usability tests in mobile application is a complex task due to the reason that traditional methods and techniques could not be enough to support them in this kind of device. Some questions about usability testing in mobile device are around the ability of traditional techniques in supporting the context of use since the users usually perform their tasks in open environment and they may not pay attention just in specific application, but can be accessing several activities at the same time [6, 8].

An example of usability testing that should be adapted is the Observation Technique, that consists of a strategy to gather data from user accesses while interacting with the system. In desktop and mobile applications, the use of webcams, microphones, external cameras, and verbalization can be common, but how these techniques can be used to collect and generate data about testing in mobile device is a challenge that this research aims to solve [1, 6, 8].

So, to support this issue we present a tool that helps conducting Usability Testing in mobile application using an Observation Technique. This tool was designed, developed and installed in mobile devices and works on background collecting image and sound data through the webcam, or only sound data using microphone besides registering snapshots that can be used in the data analysis.

This environment was developed for Android devices since this platform is extremely robust and because Android is widely used in several smart phones models and due to this reason it could be applied and validate in many different devices models. Also, Android is a free platform and provides resources for software development for free.

This paper presents the first release of this tool which aims to verify two main questions in mobile usability testing: (1) how much data can be registered in testing considered that all data should be stored in the device during the test; and (2) verify if it is a good technique to support testing trying to verify whether it can be really used in daily activities.

Next section presents the background used for developing this tool design, implementation features and validation activities

2 Background

This section presents the background about the main subjects that support this paper: Mobile Devices, Usability Testing in Mobile Devices and Techniques to support Usability Testing.

2.1 Mobile Devices

Nowadays, mobile device is used as a hybrid concept of tablets and phone cells. Usually named as smart phone, a mobile device is portable and provide a wide group of resources that allows people to perform several tasks that varies from a simple phone call to the use of internet services, such as internet banking, purchasing products and others [16].

These devices combine functions of a traditional phone with camera, web browser, music player, compass, global position systems and other resources according to the model and/or manufacturer. The combination of all these resources allowed the existence of a hardware that provides functions that were previously accessible only in desktop computers [3, 16]. Together, with the increasing of Human-Computer Interfaces quality and needs for user interaction, the smart phones gathered a great amount of users that perform most of their tasks, personal and professionally using smart phones [1, 9].

According to Cybis [6] it is really import to understand the context of use of a mobile device and how it interfere in user lifestyle and work. Some issues are highlighted below:

  1. (1)

    smart phones and mobile devices are usually used for fast tasks, in short periods of time and in specific context where the user pay real attention to these tasks such as send a message or search an information in a social network;

  2. (2)

    environment device is usually dynamic due to the reason that users can perform their tasks anywhere and can try to accomplish more than just one task at the same moment. Thus, the interactions must be easy, fast and right not to delay the user;

  3. (3)

    mobile devices´ hardware are smaller, keyboards are digitals, interface format is different, the amount of information available must be careful analyzed because the interface area could not be enough for all the components and de designers should prioritize some contents.

Also, the HCI development process from mobile applications needs to be adapted or amended in order to accomplish mobile devices. The size of the screen, the components applied for interaction, the content distribution in the interface, the cross platform and it particularities leads the developers to face new challenges in development process [3, 6].

In order to improve and facilitate the mobile software development, some guidelines were defined and provided elements to make the interfaces of this kind of devices more usable. Benyon [3] presents some elements proposed by Microsoft:

  • Using shorts menu texts;

  • Using “&” character instead of “and”;

  • Using layers to group and separate components in menu;

  • Position button “Delete” close to menu.

Cybis [6] also presents some other guidelines proposed by Nielsen, Cuello, Himann and Wroblewski. In general, these guidelines focus on the facility of use, quality of interaction considering many ways to do the same task and content for mobile device.

Beside HCI features, mobile applications present some limits. Actually the memory capacity is a concern for developers due to the reason that mobile devices have less memory than a traditional computer. For example, a desktop can present 1 Terabyte of memory and a mobile presents at about 40 Gigabytes. This issue is discussed here because memory and battery were two concerns studied as can represent special problems when using these devices.

Thus, considering the features presented, we assumed mobile application development process as a group of tasks that require new concepts when applied to the usability evaluation techniques used for mobile software.

Next section presents information about usability testing on mobile applications.

2.2 Usability Testing in Mobile Devices Using Observation Techniques

Mobile devices present new features related to hardware, software, interaction, user profile and context of use. Due to these reasons, Johnson [8] and Cybis [6] explain that traditional techniques and resources used to support usability tests in desktop and web application may not be appropriated or sufficient for conducting usability tests in mobile application and so, should be reviewed and/or improved.

In this context, there are some challenges involved in the usability tests of mobile systems, such as: device size, virtual hardware such as keyboards and buttons and the way how users interact. Concerns related to performing the test in lab or real environment; using emulator, prototypes (papers/drawings) or real software and how to register data about the usability testing, are leading HCI researches to search for new approaches or to adapt old techniques in order to attend mobile devices [3, 6].

Consulting the actual literature, we had not found a perfect approach to support usability testing for mobile devices. In several cases, the evaluators may combine more than one usability evaluation technique in order to conduct an effective testing, but this approach is not perfect as issues related to the memory capabilities and devices battery usage rates were not present.

So, researches for providing techniques that enable the user to use a mobile device in various environments can be considered as important themes for study and is the main concern of this work. Specific particularities are always present: the evaluator/user can be exposed to interruptions, there should occur interferences during the evaluation process and the need of recording the interaction is one of them [6].

The observation of the interaction between user and the system is a technique that can be used to conduct an usability testing. Recording images, voice and snapshots are examples of data that should be gathered to support the usability measures analysis. For desktop applications the use of a software for monitoring and collecting data as voice, images, logs and snapshots is already available using tools as Morae [11] and ErgoSV [5]. However, for mobile applications, these resources are not widely used.

For conducting usability evaluation experiments for mobile applications, the recording of images or data related to external environment such as sounds or participant voice can be done using secondary devices such as video cameras and/or microphones positioned on a table and directed to mobile to register the screen images.

But the use of secondary devices can be a problem since the evaluators need to collect more data from different perspective. The increasing of the devices amount can create a complex environment to collect and analysis data. Also, the data analysis will require a full review of the data individually by device and forward to synchronize data in order to have a nice data understanding.

A device presented in Fig. 1 was proposed by Betiol [4] and has a camera, microphone and resource to transfer data by radio frequency. Independent structures allow positioning the camera in order to collect images and audio with quality. The device works with battery not requiring power via cables which helps the transportation and the realization of remote usability tests.

Fig. 1.
figure 1

(Source: Betiol, p. 120)

Device to support usability testing in mobile devices

Full size image

Although this technique works, the use of a secondary resource ends up being more costly and not leaving the user so comfortable to perform an interaction easily and efficiently.

Currently there is a lack of performing usability testing on the mobile device, making use of resources such as camera, microphone and internet a need. Researches related to usability tests in mobile application such as [2, 4, 9] used secondary devices to register data as images and audio; or used specific software installed directly on the device to register log files and/or use code injections to monitor user interaction, but not to register images and audio. Considering this lack and the current resources available in mobile devices, propose a tool that could be installed in the device and work in background to collect image, sound and snapshots and so, trying to avoid the use of secondary elements.

Next section we present related works to this research.

2.3 Related Work

The development of a software to be used in mobile device is a reality and researches are being conducted in order to provide, improve or analyze methods and techniques to support all development stages among it the usability evaluation activities.

Studies focused on usability evaluation in mobile applications are not too recently, i.e., did not arise to support the smart phones, but when the mobile devices were simpler and provided less functions and resources than the current phone, but the utilization by people was already increasing.

The research conducted by Betiol [4] was performed in older phones and aimed to analyze how de context of use could interfere in the evaluation results. Tests were conducting through emulators and real phones. The users performed their tasks in lab and fields. Betiol also proposed a tool to support the usability testing based on observation. With this resources data as images and sound could be registered during a mobile interaction. A resource image is presented at Fig. 2.

Fig. 2.
figure 2

ErgoMobile work approach

Full size image

We can cite the work performed by Kaikonnen et al. [9] that aimed to evaluate the usability of a mobile application called Wire. It presents two questions: (1) whether the tests are sufficient to carry out this evaluation and; (2) whether there are differences at results of tests conducted in laboratories to tests performed in field.

A comparison was made between performing tests in the laboratory and in the field to find out what the effect of the environment would influence. When recording mobile videos, there is a lack of use of the features of the smartphone itself. So, small video cameras were hung to record the screen of the device and collect the data, and then perform an analysis.

As a result, problems were encountered in the application’s navigation, as well as phenomena and problems both in the field environment and in the laboratory. Therefore this research presented that the hypothesis that one environment is worse than the other cannot be generalized. Also, the obtained data showed that no matter what is the environment, users and the type of the application can also influence the final results of the usability environments tests.

Another work we can mention is [15] that aims to ascertain the influence of the mobile context of the application, as well as lighting the movement and location where the usability tests are performed for mobile devices. For this, the ContextCollector application was developed, which captures the user interaction through the sensors of the mobile device that is being used. This application can be used to obtain data by both developers and testers, without having to be in the presence of the evaluator, and, so, reducing the costs to perform usability tests, as it does not need to ask the user to be in a specific test environment in laboratory. As a result obtained, it is generally perceived that the influence of the poorly lit environment affected the performance of the users. Thus, a usability assessment done with the user standing still in a well-lit environment would present better results.

Next section presents the ErgoMobile environment, its features and applicability.

3 ErgoMobile

This section presents the ErgoMobile, a tool to support usability evaluation based on user observation. ErgoMobile was proposed after the study of some gaps that happen in usability testing in mobile devices due to the particular features of this kind of device and it software, such as: the difficult to collect data about user interaction [13]; the use of secondary devices that ends up to extra workload and equipment to be manipulated by participant [4]; and the battery and memory of mobile devices that have limited capacity [13].

  • Collect a great amount of data and monitor the test. This is one of the main interest in this environment, since the mobile device should register data in the local memory and it can be a problem due to the lack of storage capability;

  • Do not compromise the battery usage. Mobile devices have batteries with different capacity and durability and if the tool requires a large amount of use, there should be a problem since the usability test could be stopped as the device could be turned off;

  • Allow the evaluators and users to conduct usability testing without the need of a secondary device linked to the mobile. A software could be installed in the mobile device and be accessed in background monitoring the user tasks; and

  • Provide a tool to support the collection of data such as: images; words pronounced and snapshots.

Aiming to analyze the effectiveness and efficiency of the ErgoMobile, we chose a mobile platform to develop the software since it must be considered the great amount of devices that use AndroidFootnote 1 and the facility to obtain and use development tools and frameworks.

Next subsection presents ErgoMobile Approach.

3.1 ErgoMobile Tool

ErgoMobile approach was created in order to allow the evaluators and participants to conduct usability testing in mobile devices using Observation Techniques such as Filming and Verbalization. The idea is to have an environment that supports the evaluation activity without the need of use of secondary devices such as additional cameras, microphones and others.

To accomplish this goal we proposed a tool that can be installed in a mobile device and works in background, registering data about the usability. ErgoMobile works creating a video with images from the frontal camera and audio from microphone and registering snapshots of the interface that can be used in the analysis stage to allow the evaluator visualizing the used. If the device have frontal camera, a file containing a video (image and audio) is created, but if this is not the case, the ErgoMobile only activates the microphone and the data collected will be stored in an audio file and another one will store the interfaces snapshots.

Specifically, the data storage is gathered in specific folders at the device´s operational system. So, the ErgoMobile creates a video/audio file that is saved in a folder named “ErgoMobile Videos” and the snapshots are saved in a folder named “ErgoMobile-images_dateOfTest” where “date of test” refers to the data when the usability test was done, for example, “ErgoMobile-images_20170223”. Figure 2 presents the ErgoMobile work approach.

Next subsection presents the tests performed in order to check both memory and battery capacity/durability.

3.2 Memory and Battery Power

A mobile software development process provides some concerns that in desktop application may not be the main focus, such as battery power and memory capability. Modern mobile devices have several functions, such as colored screens and other components that can consume battery or use several portions of memory [13].

For the ErgoMobile project, this was not different since it proposes an approach with a software working on background to collect data about the test. Due to this reason, this environment should compete for resources as memory and also increase the battery consumption because ErgoMobile uses the camera(s), microphone and other resources beyond the resources/environments under evaluation.

Specifically, the memory power had a particular concern since the mobile devices provide less memory than traditional desktop computers. The process of collecting images from camera sounds and snapshot and save them in local memory can create a great amount of data.

Due to this reason, before developing the tool, we created a prototype in order to analyze the memory and battery issues. A particular mobile software was created and used just to collect data and simulate tests collecting data and consuming memory.

Three tests were performed using different devices that presented the features:

  • Device 01: 8 Gigabytes of memory. Thirty seconds of tests were performed in low and high quality of images;

  • Device 02: 8 Gigabytes of memory. Three minutes of tests were performed in both quality rates; and

  • Device 03: 8 Gigabytes of memory. Eighteen minutes of tests were performed in both quality rates.

Table 1 presents the results of the tests with devices’ battery and memory.

Table 1. Battery and memory tests results
Full size table

Battery usage data is not presented in the Table 1 since the results related to battery consumption had few variations when the ErgoMobile was being used or not. There were not significant interferences in the usage and the smart phones batteries supported the processes of usability testing and data collecting working in background without compromising the power of battery and due to this reason no action was no interferences on use were perceived.

About the tests with image, although cameras quality rates had some different results, the images were appropriated even in low quality camera and so, we chose the low quality since it allowed to store a greater amount of data.

After performing the tests in battery and memory and after analyzing the results, we assume that it is possible to develop the application since it can work in mobile phones as the typical devices provide sufficient battery and memory.

Next section is presented the ErgoMobile process to conduct a test.

3.3 Usability Testing Process Using the ErgoMobile

ErgoMobile was developed in order to be easy to configure and use and so support an usability test. Although the user must install the application in the mobile device using any technique (this is not discussed in this paper) we concerned about developing a tool simple that did not required too much workload. The process to conduct an usability testing using ErgoMobile is based on the stages:

  • Configuration, Devices Tests (Optional) and Start the Evaluation;

  • Performing Usability Testing; and

  • Data Visualization (Beta release).

Configuration, Devices Tests (Optional) and Starting the Evaluation

This stage aims to:

  1. (1)

    Indentify the participant who will use the evaluated software and collect specific data such as name, the evaluated software and the function that should be performed. This interface has just three fields and is easy to interact. This is a required task and the test could not be performed without these information;

  2. (2)

    After identification, the user is headed to a second interface that provides three resources: two buttons that allow checking the resource used for a testing such as device camera and/microphone; and another to start the usability testing. The resources checking are not required tasks, but is strongly recommended. By achieving this option, the participant and evaluator can perform the checking of camera and microphone to ensure that both components are properly working. As some devices do not have camera, our tool presents two ways to make the verification: (1) (with) Camera: Allows the user to check the camera and automatically the microphone; and (2) Audio: Allows the user to check just the microphone resource. By selecting one of these functions the user is redirected to an interface that contains components to support the checking. The user can return to the interface that provides the buttons for checking and for usability testing and by accessing from this interface, start the “Usability Test”;

  3. (3)

    The “Usability Test” button heads the user to an interface that will start the usability testing; in this interface two other buttons are available: “New Test” for beginning a new usability testing; and “Close Application” to finish ErgoMobile.

Figure 3 presents the mobile’s screen for the process of Configuration, Devices Testing and Starting the Evaluation.

Fig. 3.
figure 3

Process of configuration, devices tests and starting the evaluation

Full size image

Performing Usability Testing

After starting the test using the Button “New Test”, the ErgoMobile will work in background registratiering images from the camera, recording sounds and storing snapshots. By working in background, the ErgoMobile does not interfere in any action performed by the participant. The signal that ErgoMobile is working is an icon presented at the top of interface as presented in Fig. 4.

Fig. 4.
figure 4

ErgoMobile working in background

Full size image

When the participants finish the usability testing, they can close the ErgoMobile accessing the menu at the top of mobile interface. Some functions that are being executed by the mobile device can be presented. One of them is a button with the label “ErgoMobile - To finish the test click here”. Clicking in this button, the monitoring activity performed by the ErgoMobile will be finished and the data collected will be saved. This process can be visualized in Fig. 5.

Fig. 5.
figure 5

Resource to finish an usability testing in ErgoMobile

Full size image

Data Visualization

The data visualization process is a resource that is not the main focus of the project and, due to this reason, in this paper we are just shortly explaining it. The first release of ErgoMobile aimed to provide an approach and a tool to collect data and observe the user while working in background in a mobile usability testing.

Thus, the evaluator can access data using the mobile´s folders manager to localize the folders that were stored when using the ErgoMobile and then download these files to a computer.

Aiming to validate the data obtained, we developed a really simple tool that allows the visualization and synchronization of the video and snapshots data and so, the evaluator can visualize what happened in the evaluation. This tool is presented in Fig. 7 and we intend to improve it in future works presented at Fig. 6.

Fig. 6.
figure 6

Data visualization software used to validate the ErgoMobile

Full size image

The data visualization tool is a software developed using C# programming language [7] that loads the video/audio and snapshots and play both data synchronously in order to provide a kind of images combination that presents what happened during the usability test.

As previously mentioned, this data visualization software release is easy to be used and was developed in order to validate the ErgoMobile data collecting process. So, some activities are manual such as the data acquisition for analysis. The evaluator/participant should transfer the data from smart phone using an email or other resource for copy the data from the mobile to the computer to be analyzed.

Although this was a no unsophisticated tool, it allowed us to visualize the video and snapshots registered by ErgoMobile and with this data we concluded that the quality of images was satisfactory and there will not have problem to use it in a future and more modern data visualization tool.

This section presented the ErgoMobile process to conduct and support usability testing of mobile devices. Next section presents the process we had conducted to validate this environment and also relate future works that are expected to be accomplished.

4 Validation

To validate ErgoMobile, we performed two experiments: (1) to validate the capacity to storage data and manage device memory; (2) to validate the capacity of ErgoMobile to support data collect and observation in a usability testing.

Although the verification of memory was similar to the test performed with the prototype, we decided to perform it again we were worried whether the use of the ErgoMobile could interfere on memory management.

To conduct the usability tests, two participants with knowledge in usability testing were selected and three mobile devices were used. The devices were the same used to support the testing conducted to check battery and memory capacity using a prototyping and explained in Sect. 3.3.

The participants performed usability tests in native applications such as home banking and entertainment applications that were downloaded and installed in the devices. Considering that, the usage of the ErgoMobile aimed to analyze the process of data colleting and user observation. We hadn’t created a specific testing protocol, but just encouraged the participants to use the mobile applications and verbalize their opinions about interactions and interfaces.

Each participant performed about 30 min of tests in the devices. Issues related to the battery and memory capacity had not presented variation in relation to tests using the prototypes. Also, the ErgoMobile succeeded in storing files that maintained images, audio and snapshot related to usability tests. However, in certain moments of the tests, the ErgoMobile worked slowly and we figured out that occurred a heavy workload which caused concerns because when we conducted the battery and memory tests these problems were not present.

Thus, we conducted some searches in forums and websites about Android that presented an issue related to the Android Operational System. We discovered it that has a problem with its garbage collector and at certain time and with a considerable processing of two or more different applications simultaneously, the Random Access Memory (RAM) consumption forces the system stopping running some applications for a while. This was the ErgoMobile’s case. But this problem was not considered as a limit for using the ErgoMobile as the results were properly obtained.

Anyway, the presented problem is a concern for using the ErgoMobile and developing a new version can be a strategy for include routines that would improve the RAM management. This requirement was target as future works.

Regarding to the process of observing the usability testing, we asked to participants to inform their experience using an application being observed by ErgoMobile. The answers did not vary significantly, highlighting the issues:

  • The ErgoMobile did not interfere in the use of evaluated systems and the participants almost were not bothered by this system during the testing. In some sense, the participants said that if they had not been warned before, they would not notice the ErgoMobile working;

  • The process to configure the ErgoMobile was a little complex, but they were helped by the researches. If we aim to develop the ErgoMobile as a distributable software, a improvement in configuration process may be required; and

  • Sometimes the usability testing became slow. It happened mainly due to the problem with RAM management cited before. Future improvements are planned to solve this problem.

Regarding to the data registered by the ErgoMobile, the evaluation of the quality of audios, sounds and snapshots presented that all these data were good and allowed the analysis of the test and the identification of functions, words pronounced and the sequence of tasks conducted by the participants.

Thus, we concluded the ErgoMobile could be an appropriate approach and tool to support usability testing with observation in mobile devices applications. Considering that this was the first release and the initial efforts to develop this tool, the results of the first experiments were really exciting because the ErgoMobile works according to the planned and allowed the conduction of an observation in mobile usability testing without the need of secondary devices and with the minimum impact to the user tasks under evaluation.

Also, the data registered in the tests presented that it could be possible to perform usability tests in relevant times even registering a film/audio file and snapshots

We assume that this tool can be used for usability testing in laboratories and also in real environment because the software can be easily installed and configured by the participants. These actions avoid the use of secondary devices that ends to be a difficult task for participant and development teams manage.

Next section presents future works.

5 Future Work

This paper presented the ErgoMobile, an approach to support usability testing for mobile applications. The presented release tool was the first effort to provide a tool that can work in background and collect data about user and snapshots of interfaces used. About this project we intend to continue the research in the possible follow issues:

  • Improve the RAM memory technique;

  • Provide the tool in other platforms such as IPhone and Windows Phone;

  • Create a new tool to support the data analysis and create statistic data about the test;

  • Improve the tool to collect more data than image, sound and snapshot using interval mobile´s resources such as compass, accelerometer and GPS;

  • Create an approach to transfer data from mobile device to an integrated and online database; and

  • Provide efforts to create a usable tool that can be acquired via app stores.

6 Conclusion

Mobile devices and mobile applications brought new challenges for development teams. Device size, processing and memory power and variety of users and tasks that can be done changed the way as software for mobile should be developed and provide new concerns.

Usability testing is one of the new challenges for HCI developers since traditional methods and techniques could not be appropriated for mobile applications as well as they are for desktop computers. The observation technique, for example, usually requires the use of secondary devices for collecting data about user, software and interaction making the usability test more complex and uncomfortable.

This paper presented the ErgoMobile, an application that can be installed into the smart phone and work in background to collect data without interfere in interaction between user and software. This research aimed to analyze the capacity of data collection regarding to memory and battery power and verify whether the ErgoMobile could really support usability testing applying the User Observation technique.

The results were satisfactory due to the reason that the approach allowed the observation of users and interaction in an usability testing. The amount of snapshots and the size of video/audio file were big enough to be stored in the smart phone memory and register data for test analysis.

This tool should receive improvements as mentioned in Future Work, but the first results present that the ErgoMobile could be used as resource for usability testing avoiding the use of secondary devices and becoming an easy and practical testing process allowing the collection of significative amount of data from videos, audios and snapshots.