Abstract
Different approaches used to define how a website shows information have an impact on how users evaluate its usability. As shown in the present study, how people accomplish a search of visual content in a newspaper website is an important factor to review while designing it. In this study, 47 participants were randomly assigned to evaluate one of two different newspaper websites and asked to do visual and written searches. The evaluation metrics were: task success and task time. Also the participants made an overall evaluation of the site, answering two Likert questions and an open-ended question to measure qualitative aspects. Finally, we measured the overall satisfaction with a SUS questionnaire. The results show that a poor performance in the search of visual content lead to lower usability perception, this might be a main aspect to improve when defining priorities to enhance overall usability.
Keywords
1 Introduction
In the past few years people have changed the way they read news, as they are moving from the traditional paper to online version of newspapers such as website or mobile apps; In 2011, surveys conducted by the Pew Project for Excellence in Journalism, showed that 34 % of respondents said they read news online within the past 24 h (as opposed to 31 % who favored physical newspapers); and a full 41 % said they get most of their news online, 10 % more than those who said they got most of their news from a physical newspaper [1, 2]. Therefore, the search of information in these sites becomes very important due to the amount of news that is posted everyday.
Good usability of the websites plays an important role because it can define if the user is going to develop fidelity to the site and use it often or, with new users, it determines whether they will ever use it again after a first try [3]. As with many other interactive products, with newspaper websites the evaluation of the user experience becomes necessary to find the characteristics that users seek.
Previous usability studies have been conducted to evaluate newspapers’ sites from around the world. Among them, in the UK [4], “The Guardian” was evaluated as the best among other major newspapers, and the same was for “La Crónica de Hoy” in a Mexican study conducted by a local usability consultancy [5].
To get in context, in Fig. 1 screenshots were taken from both websites. As seen, the general layouts are very similar and have similar information and options.
In this work, first, a general evaluation comparing these both sites (“The Guardian” and “La Crónica de Hoy”) is reported to know and detect the usability differences and similarities between them. Then, an analysis was conducted to contrast the results of performing text vs. visual searches.
2 Method
This study follows a typical usability test approach where users were asked to do tasks for which standard metrics such as task success and duration were collected; after completing all of the tasks, users were asked some questions to evaluate qualitative aspects of the user experience. For the present study, the tasks were the following:
-
Task 1 - Search for an image of the tennis player Novak Djokovic, winner of the 2010 Wimbledon tennis tournament.
-
Task 2 - Search for information about the Strauss-Kahn’s case.
-
In this task, the participant was asked to write the date of the article.
-
-
Task 3 - Search for information in the “Business” section.
-
In this task, the participant was asked to write the title of the main article in the section.
-
In this study, 47 participants were randomly assigned to evaluate one of the previously mentioned websites. All the participants were bilingual with great experience reading and writing in both, English and Spanish. We did not detect any challenge related to interacting with an English-based web site despite the fact that all participants’ mother tongue was Spanish.
Participants were asked to do two different types of search tasks, which are usually done while visiting this kind of sites. First, to perform a search of textual information and, second, a search of visual content (a photograph). The metrics of the evaluation were: task success and task time. The participants also made an overall evaluation of the site, answering two questions (“Ease of finding information” and “Visual appealing”), with a 7-point Likert scale [6], and an open-ended question to measure qualitative aspects. Finally, the overall satisfaction was measured with a SUS questionnaire.
The time elapsed from the presentation of the task to the click of the “Next” button that showed the selection of the answer was recorded automatically.
The metrics captured for each task were: on one hand, self-reported task success that is whether the participant reported if they had definitely found the answer, or not, and, on the other hand, task time.
In the post-study self-reported data, the captured metrics were: ease of finding information and visual appeal (with the 7-point scale), challenging or frustrating aspects (open-ended questions), effective or intuitive aspects (open-ended questions) and SUS scores.
3 Results
The results shown in this section are derived from using the geometrical means, and tests such as Chi Square test, t-test and the Mann-Whitney U test, which were used to calculate the statistical significance depending on the type of data, with an alpha of 0.05. In general, the differences between both sites were not significant; both sites were rated with similar scores. However, we found that, although both of them are well designed and have an overall good user experience, there are some differences that make one of the sites score better than the other. The rest of this section will describe some of these differences.
3.1 Comparison Between the Websites
Task Success. After completing every task, the participants were asked whether or not they believed they were able to finish the task successfully. This was a self-reported assessment. If the participant was sure that the task was done it was marked as a success. If the user was not sure the task did not count, and it was marked unsuccessful. “The Guardian” users reported 76.12 % of success vs. an overall 81.69 % of success reported by “La Crónica” users. These results are shown in Fig. 2.
As seen in Fig. 2, the difference between both results is minimal. With the purpose of evaluating if the difference was statistically significant, the Chi-squared test was performed, as the data was dichotomous, i.e., accomplish or not the task. The result was not statistically significant between both sites, c2(1, N = 140) = 0.83, p > 0.05.
Task Time. The second aspect evaluated was the time the participants had to take to finish a task. The geometrical mean of the “The Guardian” users was 55.7 s, while for “La Crónica” was 64.75 s. Despite that in Fig. 3 an apparent difference is noticeable it was not statistically significant, t(136) = -1.35, p > 0.05.
Satisfaction Assessment. Following the execution of all tasks, the participants were asked to express comments regarding any challenge or frustrating experience they had while using the site.
On one hand, for the site of “The Guardian” (GU), 13 out of 23 participants (56.5 %) commented that they had negative experiences; for the site of “La Crónica” (CR), 11 out of 24 participants (45.8 %) indicated finding this kind of aspects.
The negative comments about the GU site given by at least two participants were:
-
Not being able to search for images (4)
-
Too much information, bad distribution of the information (9)
The negative comments about the CR site given by at least two participants were:
-
Color and content of the menu (2)
-
Problems with the search of images (3)
-
Presentation of the information, too much content (3)
On the other hand, for “The Guardian” site, 47.8 % of the participants (11 out of 23) indicated easy or intuitive aspects characterizing a good user experience; for “La Crónica” site, 62.5 % (15/24) indicated effective or intuitive aspects.
The effective or intuitive aspects about the GU site given by at least two participants were:
-
Good, intuitive and helpful menu (4)
-
Good headings that helps finding information (3)
-
The search tool is easy to use (2)
-
Overall good design (2)
The effective or intuitive aspects about the CR site given by at least two participants were:
-
Helpful and efficient search tool with good presentation of the results (9)
-
Overall good design (6)
Overall Assessment. Among the collected comments concerning negative comments, on one hand, “The Guardian” had 13 negative ones, which are all related to the overload and bad distribution of the information, and the difficulty of finding an image. On the other hand, “La Crónica” had only six comments related to these aspects and two more negative ones about the color and content of the menus. As to positive comments, the same amount of users (nine) said that both sites have good and easy to use search tools. Nevertheless, the main difference in the comments is present in the ones related to the overall design, in which, nine came from users of “La Crónica” vs. two from “The Guardian” users. In this work, it is argued that the main reason of this difference may lie in the amount of negative comments that the “The Guardian” received.
Given that the qualitative assessment favors “La Crónica”, it is worth to contrast this with the results of applying the SUS test. Following the open questions on the application, a SUS was asked to be completed and, as can be seen in Fig. 4, ten users rated with more than 60 points to “The Guardian” vs. 15 users to “La Crónica”, making the last one 31.9 % easier to use as assessed by the participants. However, a significant difference was not found, t(45) = -1.14, p > 0.05.
Despite of this lack of significant difference, this leaded to the question of why there is a trend towards lower scores that the users give to “The Guardian” even when both are good sites, and from our results it was perceived that it seems that the main difference is between the visual and written search of content.
Consequently, a specific analysis to explore the differences between visual vs. written search was conducted. The aim was to find if the differences mentioned in the comments were visible in other tests and how they affected the general scores.
3.2 Comparison Between Visual and Written Search
Task Success. Figure 5 shows the task success by type of task. With a geometrical mean of 65.72 % for the visual task and 84.8 % for the written tasks, a statistical difference was found between both type of searches, c2(1, N = 140) = 4.62, p < 0.05. This difference is mainly due to the lower rate of success of “The Guardian” which was significant too, c2(1, N = 69) = 6.59, p < 0.01. These results show the first revealing finding, which leads to identify that the design of “The Guardian” makes it more difficult for the user to do the visual task.
Task Time. The same pattern is observed in the task time results as shown in Fig. 6. There was a significant difference between the two types of search, U = 1,595, p < 0.05. On one hand, the times from “La Crónica” users does not vary, but “The Guardian” users needed more time to complete the visual search than the written ones, which is a significant difference, t(66) = 3.28, p < 0.05. On the other hand, the difference between newspapers in the written tasks was significant, U = 720.5, p < 0.01.
As can be seen in the results of task success and task time, “The Guardian” shows results with significant differences between both types of tasks. This leads to the conclusion that this newspapers’ website is good for written searches, but it is not as good in the visual content searches. This is observed in the lower rate of success in these tasks and the longer time required to complete the tasks compared to “La Crónica” users.
3.3 Post-Study Self-Reported Data
After finishing their three tasks, the participants were asked to rate the site they used on two 7-point scales (higher numbers better): Ease of finding information and Visual appeal.
Ease of Finding Information. The difference between both websites was not significant, t(44) = -0.69, p > 0.05. The geometric mean was 5.07 for the CR website and 4.58 for the GU site, as shown in Fig. 7.
Visual Appealing. As with the previous result, the difference between both websites was not significant, t(44) = -0.66, p > 0.05. The geometric mean was 4.55 for the CR website and 4.22 for the GU site, which can be seen in Fig. 8.
The results shown in this section, although they are not significant, showed that in general, “La Crónica” was rated with better scores than “The Guardian”. These differences are important to understand the reason of the different results found in the SUS. This also shows the importance of how the information is presented to the users, in the main pages as well as the search tool results. A significant impact is seen in the general usability scores mainly because of a poor performance of this particular part of the interface.
4 Conclusions
The differences between the two search functions shows that, despite the fact that both sites were rated with good usability scores, slightly favoring “La Cronica”, the design of the interface and the search tools were a significant factor to evaluate the usability of a newspapers’ website. The finding is supported by the users’ comments, which shows that the overload of information, the ease of use of the menus and the search tool were important aspects to define a website with better usability. As seen in the metrics, the difficulty to do the visual searches can lower the overall usability perception of the user, as seen in the SUS score, and in the perception of how easy is to find information and the general visual appealing.
This generates important information for future development of this kind of websites, finding new ways to make different kind of searches more intuitively and to pay attention in how they present visual content. This is because the users tend to evaluate the visual content first, and they will find difficult to understand the content or to find the needed information if they do not understand it. The information found in these results can lead to new studies to derive a deeper understanding of how to improve the usability of the search tools in the newspapers’ websites.
References
O’Dell, J.: For the First Time, More People Get News Online Than From Newspapers. Mashable. http://mashable.com/2011/03/15/online-versus-newspaper-news (Accessed on 20 September 2014)
Livemint: Internet is growing, but offline users are falling behind: McKinsey. http://www.livemint.com/Industry/XUNAjQbC0m8LbefgJMBKKL/Internet-is-growing-but-offline-users-are-falling-behind-M.html (Accessed on 20 October 2014)
Garrett, J.J.: The Elements of User Experience: User Centered Design of the Web. AIGA New Readers, New York (2003)
Webcredible, 360º news providers overview: An independent evaluation of the user experience across channels, London (2011)
UDEM, Study’s result, Primer Estudio sobre la Experiencia del Usuario en Portales de Periódicos en Línea Mexicanos. http://www.webusability.com.mx/news/resultados.html (Accessed on 3 October 2014)
Tullis, T., Albert, B.: Measuring the User Experience Collecting, Analyzing, and Presenting Usability Metrics. Elsevier/Morgan Kaufmann, Amsterdam (2008)
Acknowledgements
This work has been supported by Asociación Mexicana de Cultura A.C. and the Consejo Nacional of Ciencia and Tecnología of México (CONACyT). We thank Frida Rojas for her help in the improvement of the quality of the text.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
González, V.M., García, J., Muro, B. (2015). Searching for Information: Comparing Text vs. Visual Search with Newspapers Websites. In: Yamamoto, S. (eds) Human Interface and the Management of Information. Information and Knowledge Design. HIMI 2015. Lecture Notes in Computer Science(), vol 9172. Springer, Cham. https://doi.org/10.1007/978-3-319-20612-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-20612-7_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20611-0
Online ISBN: 978-3-319-20612-7
eBook Packages: Computer ScienceComputer Science (R0)