Design and Implementation of a Toolkit for Usability Testing of Mobile Apps

Abstract

The usability of mobile applications is critical for their adoption because of the relatively small screen and awkward (sometimes virtual) keyboard, despite the recent advances of smartphones. Traditional laboratory-based usability testing is often tedious, expensive, and does not reflect real use cases. In this paper, we propose a toolkit that embeds into mobile applications the ability to automatically collect user interface (UI) events as the user interacts with the applications. The events are fine-grained and useful for quantified usability analysis. We have implemented the toolkit on Android devices and we evaluated the toolkit with a real deployed Android application by comparing event analysis (state-machine based) with traditional laboratory testing (expert based). The results show that our toolkit is effective at capturing detailed UI events for accurate usability analysis.

Appendix

In above sections, we only dealt with the number of usability problems, without specifically pointing out what these usability problems are. Here we summarize the usability problems identified from the user study and categorize them by their severity levels.

• Cosmetic

  • 1. At the home activity of AppJoy, the users tried to trigger a menu by long-clicking an item, but the long-click listener was not implemented.

  • 2. At location-based search activity, if the search result is empty, the activity shows a blank screen, instead of cueing “no result found.”

  • 3. At recommended application activity, there is an “install” button which misled the users to think that by clicking this button, the installation process will be finished (actually there are several steps).

  • 4. The position of menu items are not consistent.

  • 5. At Help web page, some links do not function.

• Minor

  • 1. At location-based search activity, users clicked on an non-editable box to alter location.

  • 2. At location-based search activity, recommended free applications did not provide information about whether they have been installed.

  • 3. At my downloads activity, several users complained the lack of search functionality.

  • 4. Some recommended applications cannot be found in Android Market.

• Major

  • 1. After the users completed installing an application, the “install” button did not disappear so that they thought the installation was not successful.

  • 2. The Help web page was outdated.

  • 3. The most recent activity was supposed to contain only applications that were not downloaded (according to the developer’s design), but that was not the fact.

  • 4. At the detailed information activity, the text font was too small so that some important information was easily overlooked.

• Catastrophe

  • 1. The meaning of recommendation options at the home activity is not clear, and the users cannot well understand it across all tasks.

The severity level of a usability problem are mostly related with the frequency of its occurrence and the impact if it happens. We gathered up the notes taken by the evaluators, and summed up in which task and through which participant did we notice each usability problem in Table 9.

Table 9 Occurrence of usability problems

One cosmetic usability problem was discovered by the evaluators while the users performing the tasks, none of the users actually triggered the usability problem, so we did not specify a particular participant number here. Some usability problems occurred in almost all tasks or across almost all users, so we note All here for it.