skip to main content
10.1145/1631272.1631402acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
short-paper

Comparing fact finding tasks and user survey for evaluating a video browsing tool

Published: 19 October 2009 Publication History

Abstract

There are still no established methods for the evaluation of browsing and exploratory search tools. In the (multimedia) information retrieval community evaluations following the Cranfield paradigm (as e.g. used in TRECVID) have been widely adopted. We have applied two TRECVID style fact finding approaches (retrieval and question answering tasks) and a user survey to the evaluation of a video browsing tool. We analyze the correlation between the results of the different methods, whether different aspects can be evaluated independently with the survey, and if a learning effect can be measured with the different methods. The results show that the retrieval task correlates better with the user experience according to the survey than the question answering tasks. It turns out that the survey rather measures the general user experience while different aspects of the usability cannot be analyzed independently.

References

[1]
S. F. Adafre and M. de Rijke. Exploratory search in wikipedia. In SIGIR Workshop on Evaluating Exploratory Search Systems, 2006.
[2]
W. Bailer, C. Schober, and G. Thallinger. Video content browsing based on iterative feature clustering for rushes exploitation. In Proc. TRECVID Workshop, pages 230--239, Gaithersburg, MD, USA, Nov. 2006.
[3]
W. Bailer and G. Thallinger. A framework for multimedia content abstraction and its application to rushes exploration. In Proc. ACM CIVR, Amsterdam, NL, Jul. 2007.
[4]
B. J. Jansen, R. Ramadoss, M. Zhang, and N. Zang. Wrapper: An application for evaluating exploratory searching outside of the lab. In SIGIR Workshop on Evaluating Exploratory Search Systems, 2006.
[5]
V. Jijkoun and M. de Rijke. A pilot for evaluating exploratory question answering. In SIGIR Workshop on Evaluating Exploratory Search Systems, 2006.
[6]
W. Kraaij and W. Post. Task based evaluation of exploratory search systems. In SIGIR Workshop on Evaluating Exploratory Search Systems, 2006.
[7]
Y. Qu and G. W. Furnas. Model-driven formative evaluation of exploratory search: A study under a sensemaking framework. Inf. Process. Manage., 44(2):534--555, 2008.
[8]
A. Smeaton and P. Wilkins. TRECVID 2004: Interactive search questionnaires. http://www-nlpir.nist.gov/projects/tv2004/questionnaires.html.
[9]
A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and TRECVid. In Proc. ACM MIR, pages 321--330, Santa Barbara, California, USA, 2006.
[10]
A. H. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. In Proc. ACM SIGIR, 2001.
[11]
R. W. White, B. Kules, S. M. Drucker, and m. schraefel. Supporting exploratory search. Commun. ACM, 49(4):36--39, 2006.
[12]
R. W. White, G. Marchionini, and G. Muresan. Editorial: Evaluating exploratory search systems. Inf. Process. Manage., 44(2):433--436, 2008.
[13]
R. W. White, G. Muresan, and G. Marchionini. Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems. SIGIR Forum, 40(2):52--60, 2006.

Cited By

View all
  • (2018)Interactive Video SearchProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3241473(2101-2103)Online publication date: 15-Oct-2018
  • (2018)On Influential Trends in Interactive Video Retrieval: Video Browser Showdown 2015–2017IEEE Transactions on Multimedia10.1109/TMM.2018.283011020:12(3361-3376)Online publication date: 14-Nov-2018
  • (2013)A Psychophysiological Approach to the Usability Evaluation of a Multi-view Video Browsing ToolAdvances in Multimedia Modeling10.1007/978-3-642-35725-1_42(456-466)Online publication date: 2013

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '09: Proceedings of the 17th ACM international conference on Multimedia
October 2009
1202 pages
ISBN:9781605586083
DOI:10.1145/1631272
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. evaluation
  2. user study
  3. video browsing

Qualifiers

  • Short-paper

Conference

MM09
Sponsor:
MM09: ACM Multimedia Conference
October 19 - 24, 2009
Beijing, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2018)Interactive Video SearchProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3241473(2101-2103)Online publication date: 15-Oct-2018
  • (2018)On Influential Trends in Interactive Video Retrieval: Video Browser Showdown 2015–2017IEEE Transactions on Multimedia10.1109/TMM.2018.283011020:12(3361-3376)Online publication date: 14-Nov-2018
  • (2013)A Psychophysiological Approach to the Usability Evaluation of a Multi-view Video Browsing ToolAdvances in Multimedia Modeling10.1007/978-3-642-35725-1_42(456-466)Online publication date: 2013

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media