Evaluating books finding tools on social media: A case study of aNobii

https://doi.org/10.1016/j.ipm.2013.07.005Get rights and content

Highlights

  • A user study of the book finding tools on aNobii was conducted.

  • A set of performance measures were tested on these exploratory tools.

  • Browsing friends’ bookshelves was more conducive to novelty and serendipity.

  • Users’ preference structure shown to impact on the performance of the tools.

Abstract

A user study of aNobii was conducted with an aim to exploring possible criteria for evaluating social navigational tools. A set of measures designed to capture various aspects of the benefits provided by the tools was proposed. To test the applicability of these measures, a within-subject experimental design was adopted where fifty regular aNobii users searched alternately with three book-finding tools: browsing “friends’ bookshelves”, “similar bookshelves”, and “books by known authors”. Other than the self-report user experience and search result measures, the “choice set” model was used as a novel framework for navigational effectiveness. Further analyses were conducted to explore whether three aspects of reader preference, “preference insight”, “preference diversity”, and “reading involvement” might influence the performance of the tools.

Some major findings are as follows. While the author browsing function was shown to be most efficient, browsing friends’ bookshelves was shown to generate more interesting and informative browsing experiences. Three evaluative dimensions were derived from our study: search experience, search efficiency, and result quality. The disagreement of these measures shows a need for a multi-faceted evaluative framework for these exploration-based navigational tools. Furthermore, interaction effects on performance were found between users’ preference characteristics and tools. While users with high preference insight relied more heavily on author browsing to obtain more accurate results, highly involved readers tended percentage wise to examine and select more titles when browsing friends’ bookshelves.

Introduction

The growing popularity of social media has captured the public imagination and attracted attention from academic and industry researchers. For information scientists, the study of social media carries a special sense of urgency as they have rapidly become the main venues where information is consumed and exchanged. Socially-enabled navigational tools embedded in social media afford novel modes of access beyond those of traditional bibliographic means. For example, on aNobii, a social networking site for avid readers, users can now explore previously unknown items through socially-enabled navigational tools such as browsing friends’ and system-generated similar bookshelves. These book-finding tools can greatly complement traditional subject access devices in two ways. Firstly, they are particularly effective when users’ information needs are ill-defined and inexpressible, such as is often the case in searching for leisure reading. Secondly, they greatly increase opportunities for serendipitous finds through associative browsing. Fellow readers’ comments and ratings can also provide important judgment cues for book quality.

The newly emerging information environment on social media presents a fertile ground for information behavior research as new tools often engender new user information behaviors. Users now can rely on socially-enabled cues such as recommendations and reviews by others to come to know items of interest. Yet so far little research has been done to study users’ information behaviors in such a cue-rich environment, nor has there been a consensus on how to evaluate the tools embedded in these sites. The exploratory, as opposed to keyword-matching, mode of access on social media also presents a great challenge to evaluation methodology. It remains unclear what evaluative criteria, other than relevance-based precision and recall measures, can be used to validate the benefits provided by these exploration based tools. Other than search results, the user experience might be an important aspect of value provided by these tools. For example, researchers have pointed out exploratory systems can prove beneficial to support an interactive learning process beyond search results (Marchionini, 2006, White et al., 2006, White et al., 2008).

Furthermore, what really matters in users’ searching of information items for leisure on these sites is users’ preference rather than topic relevance (Koolen, Kamps, & Kazai, 2012). While in marketing research users’ responses to customized offers have been shown to be influenced by their knowledge and involvement with the products (Franke et al., 2009, Kwon et al., 2009, Shen and Ball, 2011, Simonson, 2005). Yet little has been investigated, from a system evaluation perspective, on how different user traits might influence their seeking behaviors and system performance with these socially-based recommendation tools. In the book finding environment on social media, users are constantly exposed to novel and sometimes serendipitous items. Thus, their willingness and ability to explore items might turn out to be an important factor on the effectiveness of these tools. One can imagine that readers with a more refined preference mightreact differently than more casual readers. It might also be relatively more difficult to recommend novel and potentially interesting items for well-read readers and more resourceful readers who rely on their own information resources for book finding. Among cultural product consumers, a distinction is often made between sophisticated and casual consumers (Caves, 2002). When examining online DVD rental data, Elberse (2008) found that consumers who have ever chosen obscure products tended to consume a much wider range of alternatives, including popular items, while those who knew of few alternatives tended to stick with more popular offerings. It is reasonable to hypothesize that users’ knowledge and involvement with cultural products influence their response to customized recommendations. Therefore besides testing various evaluation criteria in this study, we also wish to explore how different aspects of users’ preference might influence user behaviors and tool effectiveness.

In the following sections, we will begin with a discussion of the impact of socially-based book-finding tools on users’ access to readings for leisure, followed by elucidating the procedures and results of the experiment.

Section snippets

Access to readings for pleasure

Community-based navigational tools can be especially effective where keyword-based subject access is less successful (Koolen et al., 2012). An area where subject access has been found to be inadequate is access to imaginary work. With imaginary works, what is sought is the emotive reading experience that the work evokes rather than the subject it addresses. The difficulty of subject access to imaginary works lies in the fact that the relationship between the features in the work and the types

Research questions

The aims of the study are twofold: Firstly, to explore possible criteria for evaluating the social navigational tools. A set of measures designed to capture various aspects of the benefits provided by the tools was proposed. An experiment was then conducted using a modified interface of aNobii so that we could compare performance of these tools against the traditional author browsing function on these measures. Secondly, the study examines how three aspects of preference, “preference insight”,

The testing site

The study set out to explore potential criteria by which the social navigational tools on social media can be evaluated. The online book sharing website aNobii was chosen as the test site for our study, for a couple of reasons. Firstly, it has attracted a relatively large user community in Mandarin in Taiwan (where the study was conducted) among all the book-sharing platforms. Secondly, it also provides a wide range of social navigational tools that enable us to draw rich user-interaction data.

Descriptive statistics

Table 3 provides the descriptive statistics of the participants’ gender, time since joining aNobii, and frequency of use. Over 82 percent of the participants have joined for more than one year. The numbers of books owned in the “bookshelf” ranged greatly from 81 to 2528, with a mean of 431.65 and SD of 495.30; the numbers of friends ranged from 5 to 52, with a mean of 15.6 and SD of 11.53. A significant correlation was found between the number of books owned and total friends, r = .31, p < .05 (see

Discussion

Our results showed that while the similar bookshelf did consistently the poorest, the other two book-finding tools, author search and friends’ bookshelves exchanged leads in different measures, depending on the underlying criteria these measures represented. In the search experience category, “friends’ bookshelves” did better in all but one measure, “preference matching”. The advantage of the browsing known- author approach in preference matching was also reflected in users’ visiting and saving

Conclusion

The growing availability of social navigational tools on social media presents a challenge to evaluative methodology. The first main contribution of our research is to propose and test a set of subjective and objective evaluative measures by which the benefits of these tools can be validated and compared. Other than the usual self-report user experience and result based measures, the “choice set” model from retailing was used to provide a framework by which navigational effectiveness can be

References (40)

  • Aiello, L. M., Barrat, A., Cattuto, C., Ruffo, G., & Schifanella, R. (2010). Link creation and profile alignment in the...
  • R.E. Caves

    Creative industries: Contracts between art and commerce

    (2002)
  • R.L. Celsi et al.

    The role of involvement in attention and comprehension processes

    Journal of Consumer Research

    (1988)
  • Dourish, P., & Chalmers, M. (1994). Running out of space: Models of information navigation. Presented at the HCI,...
  • Elberse, A. (2008). Should you invest in the long tail? Harvard Business Review....
  • N. Franke et al.

    Testing the value of customization: When do customers really prefer products tailored to their preferences?

    Journal of Marketing

    (2009)
  • A.G. Greenwald et al.

    Audience involvement in advertising: Four levels

    Journal of Consumer Research

    (1984)
  • J. Herlocker et al.

    Evaluating collaborative filtering recommender systems

    ACM Transactions on Information Systems

    (2004)
  • Koolen, M., Kamps, J., & Kazai, G. (2012). Social book search: Comparing topical relevance judgements and book...
  • T. Kramer et al.

    The effect of cultural orientation on consumer responses to personalization

    Marketing Science

    (2007)
  • Cited by (14)

    • Finding fiction: Search moves and success in two online catalogs

      2016, Library and Information Science Research
      Citation Excerpt :

      The strongest predictor of successful book selection was the duration of visits to the author/title information, suggesting that the content description did not play a crucial role in fiction selection. In fiction searching, the concept of social browsing (Peters, 2011) refers to the ability to re-orientate browsing by social navigational tools (Tang, Sie, & Ting, 2014). Tang et al. conducted a user study of aNobii, an online book sharing website.

    • Digital humanities and digital social reading

      2021, Digital Scholarship in the Humanities
    View all citing articles on Scopus
    View full text