A study of awareness in multimedia search

Abstract

Awareness of another’s activity is an important aspect of facilitating collaboration between users, enabling an “understanding of the activities of others” (Dourish & Bellotti, 1992). In this paper we investigate the role of awareness and its effect on search performance and behaviour in collaborative multimedia retrieval. We focus on the scenario where two users are searching at the same time on the same task, and via an interface, can see the activity of the other user. The main research question asks: does awareness of another searcher aid a user when carrying out a multimedia search session?

To encourage awareness, an experimental study was designed where two users were asked to compete to find as many relevant video shots as possible under different awareness conditions. These were individual search (no awareness), Mutual awareness (where both users could see the other’s search screen), and unbalanced awareness (where one user is able to see the other’s screen, but not vice-versa). Twelve pairs of users were recruited, and the four worst performing TRECVID 2006 search topics were used as search tasks, under four different awareness conditions. We present the results of this study, followed by a discussion of the implications for multimedia information retrieval systems.

Introduction

The growth of online multimedia resources is continuing to rise, with web sites such as YouTube¹ and Flickr² allowing users access to increasing quantities of multimedia data. The problem of accurately and quickly searching multimedia documents in such systems has accordingly grown in significance: users now have an increased ability to obtain and store multimedia files, but often have no effective facilities available to search the data. One possible approach to the searching of such multimedia collections is the use of collaboration search, where two or more individuals search together to find new material (Adcock et al., 2007, Adcock and Pickens, 2008, Adcock et al., 2008). While search is often considered an activity carried out alone, recent research has suggested that collaborating together on search tasks is not unusual. The survey carried out by Morris (2008) found that 53.4% of the respondents answered “yes” to the question “Have you ever cooperated with other people to search the Web?”.

One important aspect of collaboration is awareness, which enables “an understanding of the activities of others, providing a context for your own activity” (Dourish & Bellotti, 1992). When engaged on a collaborative task, awareness of others allows a user to know who is doing what, enabling the coordination and sharing of information. In this paper, we are interested in how awareness of another user’s searching may or may not aid a different user searching at the same time, but in a different place. Specifically, we are interested in investigating three main questions: (1) does having awareness of another user alter a user’s search performance? (2) does awareness alter a user’s searching behaviour? and (3) what parts of a user’s searching process is most useful to another user engaged on the same task?

In order to investigate these questions, an experiment was carried out which used a game scenario, where pairs of users competed against each other to find as many relevant video shots as possible, using a video retrieval system.³ With this set up, the aim is to provide an incentive for users to be aware of the other user, to investigate how awareness may or may not aid searching in an environment which encourages its use.

Motivating this research is the continuing need for methods of improving retrieval within the context of a community of users. We theorize that creating community based systems where users can be automatically connected together depending on their immediate information needs may provide opportunities for richer search collaborations between users. Such collaborations are of especial interest in multimedia searching, where the performance of video and image search engines are currently very low, and hampered by the problems inherent in the semantic gap (Smeulders, Worring, Santini, Gupta, & Jain, 2000). The semantic gap is the difference between the low-level visual features which can be extracted and used by content-based search systems (such as color and texture), and the high-level concepts in which users work and search. Given this, encouraging and facilitating collaboration between users is one way of attempting to mitigate the current problems in multimedia retrieval.

The information retrieval domain which is considered here is that of multimedia video retrieval, as exemplified by the TRECVID effort (Smeaton, Over, & Kraaij, 2006c). The problems inherent in video retrieval, and in particular the problem of the semantic gap, means that this is a challenging research area. Current state of the art systems, such as those taking part in TRECVID-2006 (Smeaton & Over, 2006), still show a performance considerably below that of their text-based cousins.

We are principally concerned with two types of awareness: Mutual awareness, as considered by Schmidt, 1998, Simone and Bandini, 2002, where two (or more) users are both aware of each other in a joint collaboration; and Watching awareness, where user A is able to observe the searching of user B, while user B cannot observe A. This latter type of awareness leads to a further type, which we call Watched awareness, where user A knows user B is able to watch their searching, but cannot watch user B. The latter two types of awareness are, of course, two different sides of an unbalanced awareness between two users. The reason for being interested in such cases is due to its potential occurrence in a large application: at any point in time, there may be many potential watchers, and many others being watched, but neither of these situations may be balanced–unbalanced awareness situations may well take place. The potential effect of such unbalanced awareness on search is an open question.

One aspect which will not be dealt with in this paper is that of privacy, an issue which will undoubtedly be a problem for real-world systems. Privacy, however, does not preclude studies of the potential advantages, if any, of synchronous awareness in multimedia information retrieval. In the study reported here, we are interested in investigating the potential benefits of awareness, within a scenario which is conductive to awareness itself.

The remainder of this paper is structured as follows. Section 1.1 outlines the research questions considered in this paper, followed by Section 2 which describes related work. Section 3 describes the experiment, outlining the experimental design, the data collection used (Section 4), the experimental procedure (Section 5), and the interfaces (Section 6). Section 7 presents the results, which is followed by a discussion in Section 8. The paper ends with conclusions and future work.

Four main research questions are considered in this paper:

• RQ1: Does awareness alter a user’s search performance?

• RQ2: Does awareness alter a user’s searching behaviour?

• RQ3: Does having awareness of another alter the effort a user must put in to searching?

• RQ4: What parts of a user’s searching process is most useful to another user engaged on the same task?

As shorthand, in this paper we will use the term “awareness” when meaning “awareness of another user engaged on the same task”. For RQ1, we are most interested in a user’s ability to find more relevant material, while for RQ2 the emphasis is on how user searching behaviour changes, e.g. does the number of queries executed change? RQ3 is concerned with quantifying the effort users put into searching, and in particular, for the video retrieval interfaces considered in this paper, the effort put into browsing videos. The final research question is broader than the other three, being a more opened ended investigation into what parts of the remote user’s search screen is of most use to another user, e.g. are search terms, search results, etc. copied and used by a local user.