Keywords

1 Introduction

Usability has been receiving increased attention among both designers and researchers. Besides the criteria of effectiveness and efficiency, thanks to user’s satisfaction and pleasure -being one of the strongest determinants, recently- usability of web sites has become more important. Usability issues in web sites have been examined in academic literature. Previous studies have provided important insights into web site usability.

Systematic review is important to evaluate the work done in a specific area, compare the obtained results, find out the focused topics and the issues left missing, and explore further research topics that shall be done. Starting from this point, the present study takes a systematic review in order to examine the research trends in web site usability. The study aims to review systematically all the studies of web site usability conducted in the last decade, in order to present the research trends in usability issues of web sites. To summarize, this comprehensive research on the usability issues of web sites presents the research trends with various trending topics and thus, provide important implications for future research.

2 Theoretical Background

There are various definitions of usability. According to United States Department of Health and Human Services, “usability refers to the quality of a user’s experience when interacting with products or systems, including web sites, software, devices, or applications. Usability is about effectiveness, efficiency and the overall satisfaction of the user” [1]. Nielsen [2] explains usability with regards to five attributes: learnability, efficiency, memorability, errors and satisfaction. Usability is also defined as “the degree to which people (users) can perform a set of required tasks” [3]. Similar to this definition, Dumas and Redish [4] states “usability means that the people who use the product can do so quickly and easily to accomplish their own tasks”. Rubin and Chisnell [5] mention that a product or a service can be accepted as usable if the user can accomplish what s/he wants to do without having any problems or without asking for help. However, the most widely used definition of usability is the one indicated by ISO 9241. The ISO standard 9241-11, defines usability as “the extent to which a product can be used by specified users to achieve specific goals with effectiveness, efficiency and satisfaction in a specified context of use” [6].

When we considered the research on usability issues for web sites, we found out that the first studyFootnote 1 was published in 1995. The question of “What are the usability issues examined in the publications throughout the years?” comes into mind. It is possible to find some answers to this question by looking at the systematic review studies. There is not any systematic review study related to web site usability, published before 2008. It is considered that the field of usability has become more important in the literature and the increasing number of studies on web site usability also increased the number of systematic reviews since 2008. We could reach a limited number of systematic review studies on web site usability published after 2008. These studies focus especially on web site usability evaluation methods (UEMs).

Insfran and Fernandez [7] designed a systematic review study. They indicate that theirs is the first systematic review study related to usability evaluation. They searched for “usability AND web AND development AND (evaluation OR experiment OR study OR testing)” query in IEEExplore and ACM digital libraries and reached to 410 studies published between 1998 and 2008. According to inclusion criteria, an article should contain the website UEMs, be related to web applications, and be a full text. On the other hand, as the exclusion criteria, the articles those aim to review the literature on the principles or recommendations for web design, use a combination of methods for measuring usability, use usability metrics, published in a special issue, written in non-English, or published in conference proceedings are not included in the research. Finally, the number of articles reviewed as a part of the study declined to 51.

Fernandez et al. [8] designed a systematic mapping for the studies including web UEMs and published between 1996 and 2009. The studies were scanned by using IEEEXplore, ACM Digital Library, Springer Link and Science Direct databases. Additionally, conference papers and journals were scanned manually. “Web (web OR website OR internet OR www) AND Usability (usability OR usable) AND Evaluation (evalu*⁄OR assess*⁄OR measur*⁄OR experiment⁄OR stud*⁄OR test*⁄OR method*⁄OR techni*⁄OR approach*)” query was used for the search. As the inclusion criteria, the studies should related to web applications and report the employment of the current methods on the web. On the other hand, the studies that were not related to web, suggestions for web design, evaluated the usability features and their measurement, emphasized only accessibility, mentioned how to integrate UEMs, or reported on the functionality were not included in the research. Additionally, introductory studies, iterative, or non-English articles were removed from the list. Finally, 206 of 2703 studies were included in the research.

Nawaz and Clemmensen [9] prepared a general evaluation for the studies on website usability published between 2001 and 2011. They searched for “website” and “usability” words and selected the studies for only Asia, which is the first author’s country. They eliminated the iterative studies and included studies in English. They expected the studies would be related to cultural differences, UEMs, website design methods, religious and public websites, and would include quantitative research and rural users. They used ACM Digital Library, Scopus, Web of Science (SSCI), and Science Direct for the search. They included 60 studies according to inclusion criteria.

Table 1 presents the highlighted results from these three systematic reviews.

Table 1. Highlighted results from systematic reviews on website usability
Full size table

It’s possible in this paper to cite many other studies related to website usability. The fact is that there are some common research questions in the usability review studies which are mostly related to UEMs used and usability issues focused. However, some other questions are still missing in most of the review studies and needs to be answered. These questions are related to web design life cycle stages, suggestions offered, number of publications per year, countries of origin, website contents, user profiles, and numbers of users. On the other hand, there is always a need for systematic review studies because there are always new publications in a specific theme every year. Therefore, the present review study aims to be comprehensive and to present up-to-date information by examining the last ten years’ publications.

3 Methodology

A systematic review is “a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review” [10]. As Torgerson [11] states, “A systematic review differs from a traditional narrative review in that its methods are explicit and open to scrutiny. It seeks to identify all the available evidence with respect to a given theme.” In other words, systematic reviews allow researchers to compare the results of all reports in a specific research theme. Through systematic review, it is possible to see what the trend is, what the studies are generally focused on, and what they are missing.

The present study takes a systematic review approach based on the theoretical framework by Kitchenham [12]. This framework performs a systematic review identified in three main phases. In the first phase, which is called Planning the Review, we first identified the need for a review through the following research questions:

  1. 1.

    What is the distribution of the number of publications per year?

  2. 2.

    What is the distribution of web site contents in usability studies?

  3. 3.

    What is the user profile distribution in usability studies?

  4. 4.

    What are the most frequently used methods in usability studies?

  5. 5.

    What are the most frequently addressed usability issues in usability studies?

  6. 6.

    Which are the most cited studies?

Then, we developed a review protocol as follows. The search was undertaken through the online library collection of Istanbul University in 2015. Searches were restricted to peer-reviewed articles, written in English, and published between 2005 and 2014. The search string used was: ((usability OR hci OR “human computer interaction” OR ux OR “user experience” OR “user test” OR “user tests”) AND (“web site” OR “web page” OR “web sites” OR “web pages”) AND (user OR users)). The only contextual criterion for a publication to be included in the systematic review was that the publication should aim to examine the usability of a web site(s).

In the second phase which is Conducting the Review, the search process was completed through the following steps:

  • Publications that include the keyword set within their abstract and published in the years between 2005 and 2014 were listed. This search yielded to 1047 results in total.

  • Publications listed in the above search results were checked, if their full text were downloadable through the online library of Istanbul University. Then, 450 of them were downloaded as full text articles.

  • These full texts were checked if they were (i) published as an article in a peer-reviewed journal or as a conference paper published in the proceedings book and (ii) whether they were written in English. Multiple copies were also excluded. Then, 233 full texts were left to be checked whether their context match the aim of our study.

  • Finally, we checked these full texts to see if they were suitable for our systematic review study in terms of their research objectives. Publications (i) which aim to examine the usability issues but not the usability of a web site, or (ii) which examine a web site but not in terms of the usability issues were excluded. Therefore, at the end of the systematic search process, 199 full texts were included into this study for reviewing.

Then, the selected publications were analyzed and synthesized according to above research questions. Finally, the research report was written as stated by Reporting the Review phase which is the third phase.

4 Results

We detailed the results of the systematic review analysis based on our six research questions, as follows.

4.1 What is the Distribution of the Numbers of Publications Per Year?

It can be seen through Fig. 1 that 2005 (28 studies) and 2006 (26 studies) are the years with the highest publication number and 2013 (13 studies) with the lowest.

Fig. 1.
figure 1

Distribution of the publication years of the studies

Full size image

However, the number of studies issued in the other years are pretty close to each other and within the average range (Avg=20; SD=5).

4.2 What is the Distribution of Web Site Contents in Usability Studies?

Regarding the context of web sites which were tested in the studies, five main categories were formed: Education, Business, Health, Socio-cultural, and Civil services. The distribution of classification of web sites tested in the studies is presented in Fig. 2, except the studies whose contexts were not specified (24.1 %). There is also a category of Multi-context, which indicates the studies with more than one context. Among the studies under the education category (71 studies; 35.7 %), library web sites draw the attention alone with the highest ratio (47 studies; 23.6 %), which is higher than all others.

Fig. 2.
figure 2

Distribution of web site contents (Color figure online)

Full size image

4.3 What is the User Profile Distribution in Usability Studies?

It is possible to classify the special user groups in usability studies as listed by Rızvanoğlu [13]. We modified that list with the addition of the special user group of individual differences, such as spatial visualization ability. Excluding studies in which the web sites were tested without any user participation (29.1 %), the rest of the participant profile distribution in the studies is as below Fig. 3. Also, the web sites were tested with more than one special user group in six of all the studies. Although the studies with non-special user groups (43.7 %) consist the majority, there are some studies with various special user groups. Among those, the highest rate belongs to the studies with disabled users (7.5 %).

Fig. 3.
figure 3

User profile distribution (Color figure online)

Full size image

On the other hand, it is remarkable that the term “accessibility” was mentioned at least once in 14 (out of 15) studies with disabled users whereas it was mentioned only in 10 (out of 30) studies with the other kind of special user groups. In spite of the fact that accessibility is a concept that is related to universal design [13], i.e. any kind of user groups, that term was associated mostly with the disabled user group regarding the studies reviewed in the present study.

4.4 What are the Most Frequently Used Methods in Usability Studies?

We propose to classify UEMs under five main categories: User-based, Expert-based, Data-driven, Design process, Literature review. Users directly participate to the usability evaluation in user-based methods whereas in expert-based methods, usability evaluation is carried out only by usability experts. User-based methods include usability testing, questionnaire, interview, prototyping (if evaluated by users), focus group, eye tracking, card sorting, remote usability testing, etc. Expert-based methods consist of expert evaluation, Heuristic evaluation, cognitive/barrier walkthrough. Data-driven methods are based on stored data and its analysis by various methods, such as Web analytics, log analysis, data mining, etc. The methods related to design process are the ones which are applied mostly during the design phase, without any participation of users directly, such as user profiling, storyboarding, prototyping, etc. Literature review is the method of only reviewing the literature, without any usability test. Figure 4 shows the distribution of UEMs according to these five categories.

Fig. 4.
figure 4

Methods used in usability studies

Full size image

Most of the studies (138 out of 199) used user-based UEMs so that we detailed this category as in Fig. 5. “Others” category consisted of diary, participatory design, guerilla testing, 5-Second test, and EMG. Also, in some of the studies, remote usability testing (1), card sorting (1), and usability testing (19) were accompanied by “think aloud” protocol. The most frequently used methods are questionnaire (98) and usability testing (81). On the other hand, it is remarkable that eye tracking was used only in seven studies, in spite of its attractiveness thanks to its technology.

Fig. 5.
figure 5

Methods used in usability studies

Full size image

4.5 What are the Most Frequently Addressed Usability Issues in Usability Studies?

Regarding the usability issues, studies were reviewed by paying attention to each study’s own terminology. Figure 6 shows the distribution of usability issues. If no specific usability issue or only a common usability factor -such as effectiveness- was mentioned in a study, then we classified it under general usability issues category. Methodological issues category consists of the studies on which either how to evaluate, research, redesign, test the tool, apply the method (29) or how to design (12). “Others” category indicates the issues (icon design, color, personalization, etc.) which were addressed in less than 10 studies, in total.

Fig. 6.
figure 6

Usability issues addressed in usability studies

Full size image

Excluding general usability issues, the most frequently addressed usability issue is navigation (56) and it is followed by methodological issues (41), UI design (38), and accessibility (36).

4.6 Which are the Most Cited Studies?

It would be useful to catch the key points in studies by assessing them according to how many times they were cited. Therefore, citation numbers listed in Web of Science were reported for each study by March, 2016. Three of the studies have been cited more than 50 times, 13 studies 20–49 times, 26 studies 10–19 times, and 70 studies 1–9 times whereas 18 studies haven’t been cited yet and the other 69 studies have no citation record in Web of Science. The top 3 studies are summarized in Table 2.

Table 2. Summary of the most cited studies
Full size table

The most cited study (101 citations) is a literature review on general usability issues and proposes a new model of usability measurement called Quality in Use Integrated Measurement (QUIM). Thus, it is possible to see the importance of theoretical studies.

5 Conclusion

Usability evaluation is widely considered as necessary for designing web sites. Although previous systematic review studies have provided important insights into web site usability studies, there is still need for comprehensive and up-to-date research in the field. The present study analyzes and reports web site usability studies published in the years from 2005 to 2014, which are 199 studies in total.

The major findings are as follows.

  • 2005 (28 studies) and 2006 (26 studies) are the years with the highest publication number and 2013 (13 studies) with the lowest. However, the number of studies issued in the other years are within the average range (Avg = 20; SD = 5).

  • The top 3 content of web sites were education (71 studies, including 47 library web sites), business (28 studies), and lastly, health (21 studies).

  • User experience of special user groups was not taken into consideration much (disabled users in 15 studies, expertise differences in 10, cultural differences in 7, individual differences in 5, gender differences in 3, child users in 3, and elderly users in 2).

  • Most of the studies (138 out of 199) used user-based UEMs. The most frequently used user-based methods are questionnaire (98) and usability testing (81).

  • Excluding general usability issues, the most frequently addressed usability issue is navigation (56) and it is followed by methodological issues (41), UI design (38), and accessibility (36).

  • The most cited study (101 citations) is a literature review on general usability issues and proposes a new evaluation model.

This systematic review provides useful insights for researchers and practitioners based on a systematic investigation of web site usability studies. First of all, the study is important that it represents a collection to see the overall picture as well as the present research trends and missing points in the usability studies. By doing so, this study also helps enrich researchers’ understanding of usability research design perceptions as it reviews both the methodology and the target user profiles in the studies. In addition, the study expands the usability literature by summarizing the related studies. Also, it has the importance of being a reference for the reviewing process with the analysis structure of research questions, especially with the proposed classification for UEMs.

Future studies can be conducted by extending the research questions of the present study to discover the trends by other elements, such as various other characteristics of the target user group, or various environments on which the web sites are being published. In order to utilize from the results of this systematic review, a relational analysis on the data set, comparisons across the research questions are also required for future work. Also generalizing the results to all web site usability studies may require some confirmatory studies with the use of additional keywords (i.e. “redesign” or “website”) while searching with an extended publication date range. Moreover, there is also need for systematic review studies on usability issues for other online products, such as mobile applications.

In conclusion, this comprehensive systematic review study presents the research trends between 2005 and 2014 on the usability issues of web sites. The inclination of research design in web site usability in the last ten years also provides important implications for future research.