skip to main content
10.1145/1180639.1180721acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
Article

Extreme video retrieval: joint maximization of human and computer performance

Published: 23 October 2006 Publication History

Abstract

We present an efficient system for video search that maximizes the use of human bandwidth, while at the same time exploiting the machine's ability to learn in real-time from user selected relevant video clips. The system exploits the human capability for rapidly scanning imagery augmenting it with an active learning loop, which attempts to always present the most relevant material based on the current information. Two versions of the human interface were evaluated, one with variable page sizes and manual paging, the other with a fixed page size and automatic paging. Both require absolute attention and focus of the user for optimal performance. In either case, as users search and find relevant results, the system can invisibly re-rank its previous best guesses using a number of knowledge sources, such as image similarity, text similarity, and temporal proximity. Experimental evidence shows a significant improvement using the combined extremes of human and machine power over either approach alone.

References

[1]
Chang, E.Y., Tong, S., and Goh, K.-S. Support Vector Machine Concept-Dependent Active Learning for Image Retrieval. IEEE Transactions on Multimedia (anticipated 2005), http://mmdb2.ece.ucsb.edu/~echang/mm000540.pdf.
[2]
Chang, S.-F., (moderator), Multimedia Access and Retrieval: The State of the Art and Future Directions. In Proc. ACM Multimedia '99 (Orlando FL, Nov. 1999), ACM Press, 443--445.
[3]
Chen, M-Y., and Hauptmann, A., Searching for a Specific Person in Broadcast News Video, International Conference on Acoustics, Speech, and Signal Processing (ICASSP'04), Montreal, Canada, May 17-21, 2004
[4]
Derthick, M., Interfaces for Palmtop Image Search. Proc. JCDL (Portland, OR, July 2002), 340--341.
[5]
Forsyth, D., and Ponce, J. Computer Vision: A Modern Approach. Prentice Hall, Englewood Cliffs, NJ, 2002.
[6]
Freund, Y., and Schapire, R.E. A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting. Journal of Computer and System Sciences, 55, 1, 1997, 119--139.
[7]
Gosselin, P.H., and Cord, M. RETIN AL: An active learning strategy for image category retrieval. In Proc. IEEE Conf. Image Processing (Singapore, October 2004), 2219--2222.
[8]
Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
[9]
Hauptmann, A.G., and Christel, M.G. Successful Approaches in the TREC Video Retrieval Evaluations. Proc. ACM Multimedia '04, ACM Press (2004), 668-675.
[10]
Hauptmann, A. G., Christel, M., Conescu, R., Gao, J., Jin Q., Lin, W.-H., Pan, J.-Y., Stevens, S. M., Yan, R., Yang, J., and Zhang, Y. CMU Informedia's TRECVID 2005 Skirmishes, in TRECVid 2005 - Text REtrieval Conference TRECVID Workshop, Gaithersburg, MD, 14-15 Nov. 2005.
[11]
Lee, H. and Smeaton, A.F. Designing the User Interface for the Fischlar Digital Video Library, J. Digital Info. 2(4), http://jodi.ecs.soton.ac.uk/Articles/v02/i04/Lee/, May 2002.
[12]
McCallum, A., and Nigam, K. Employing EM in pool-based active learning for text classification. In Proc. Int'l Conf. on Machine Learning. Morgan Kaufmann, 1998, 350--358.
[13]
Naphade, M., and Smith, J.R. Active Learning for Simultaneous Annotation of Multiple Binary Concepts. In Proc. IEEE Intl. Conf. on Multimedia and Expo (ICME) (Taipei, Taiwan, June, 2004), 77-80.
[14]
Naphade, M.R., and Smith, J.R. On the Detection of Semantic Concepts at TRECVID. Proc. ACM Multimedia '04, ACM Press (2004), 660--667.
[15]
Nguyen, H.T., and Smeulders, A. Active Learning Using Pre-clustering. In Proc. Int'l Conf. on Machine Learning (Banff, Canada, July 2004). ACM Press, 2004.
[16]
Over P, Kraaij W and Smeaton A.F. TRECVID 2005 - An Introduction. TRECVid 2005 - Text REtrieval Conference TRECVID Workshop, Gaithersburg, MD, 14-15 Nov. 2005.
[17]
Schneiderman, H., and Kanade, T. Probabilistic Modeling of Local Appearance and Spatial Relationships of Object Recognition. In Conf. Computer Vision and Pattern Recognition (CVPR '98) (Santa Barbara, CA, June, 1998). IEEE Computer Society, 1998, 45--51.
[18]
Spence, R., Rapid, Serial and Visual: A presentation technique with potential. Information Visualization, 1(1):13-19, 2002.
[19]
Tong, S., and Chang, E. Support Vector Machine Active Learning for Image Retrieval. In Proc. ACM Multimedia 2001 (Ottawa, Canada, October, 2001). ACM Press, 2001, 107--118.
[20]
Wang, L., Chan, K.L., and Zhang, Z. Bootstrapping SVM Active Learning by Incorporating Unlabelled Images for Image Retrieval. In Conf. Computer Vision and Pattern Recognition (CVPR '03) (Madison, WI, June, 1998). IEEE Computer Society, 2003, 629--634.
[21]
Yan, R., Yang, J., and Hauptmann, A., Learning Query-Class Dependent Weights in Automatic Video Retrieval, Proceedings of ACM Multimedia 2004, New York, NY, pp. 548--555, October 10-16, 2004
[22]
Snoek, C. G. M., van Gemert, J. C.,Geusebroek, J. M., Huurnink, B., Koelma, D. C., Nguyen, G. P., De Rooij, O., Seinstra F. J., Smeulders, A. W. M., Veenman, C. J., Worring, M., The MediaMill TRECVID 2005 Semantic Video Search Engine. In Proceedings of the 3rd TRECVID Workshop, Gaithersburg, USA, November 2005
[23]
Yan, R., and Hauptmann, A.G., Efficient Margin-Based Rank Learning Algorithms for Information Retrieval. Conference on Image and Video Retrieval (CIVR 2006), pp. 113--122 Tempe, AZ, July 2006.
[24]
Chen, M-Y., Wactlar, H., Hauptmann, A., and Christel, M., Putting Active Learning into Multimedia Applications: Dynamic Definition and Refinement of Concept Classifiers ACM Multimedia 2005, National University of Singapore, Singapore, November 6-11, 2005

Cited By

View all
  • (2024)Interpreting pretext tasks for active learning: a reinforcement learning approachScientific Reports10.1038/s41598-024-76864-214:1Online publication date: 28-Oct-2024
  • (2024)An Adaptive Query Approach for Extracting Medical Images for Disease Detection ApplicationsArabian Journal for Science and Engineering10.1007/s13369-024-09152-wOnline publication date: 24-May-2024
  • (2023)User Preference and Performance using Tagging and Browsing for Image LabelingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580926(1-13)Online publication date: 19-Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '06: Proceedings of the 14th ACM international conference on Multimedia
October 2006
1072 pages
ISBN:1595934472
DOI:10.1145/1180639
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. active learning
  2. human performance optimization
  3. relevance feedback
  4. video retrieval

Qualifiers

  • Article

Conference

MM06
MM06: The 14th ACM International Conference on Multimedia 2006
October 23 - 27, 2006
CA, Santa Barbara, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)1
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Interpreting pretext tasks for active learning: a reinforcement learning approachScientific Reports10.1038/s41598-024-76864-214:1Online publication date: 28-Oct-2024
  • (2024)An Adaptive Query Approach for Extracting Medical Images for Disease Detection ApplicationsArabian Journal for Science and Engineering10.1007/s13369-024-09152-wOnline publication date: 24-May-2024
  • (2023)User Preference and Performance using Tagging and Browsing for Image LabelingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580926(1-13)Online publication date: 19-Apr-2023
  • (2022)Abbreviating Labelling Cost for Sentinel-2 Image Scene Classification Through Active LearningPattern Recognition and Image Analysis10.1007/978-3-031-04881-4_24(295-308)Online publication date: 26-Apr-2022
  • (2021)II-20: Intelligent and pragmatic analytic categorization of image collectionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.303038327:2(422-431)Online publication date: Mar-2021
  • (2021)Location Selection for Air Quality Monitoring with Consideration of Limited Budget and Estimation ErrorIEEE Transactions on Mobile Computing10.1109/TMC.2021.3065656(1-1)Online publication date: 2021
  • (2021)Efficient sampling for decision making in materials discovery*Chinese Physics B10.1088/1674-1056/abf12d30:5(050705)Online publication date: 1-May-2021
  • (2021)A survey on active learning and human-in-the-loop deep learning for medical image analysisMedical Image Analysis10.1016/j.media.2021.10206271(102062)Online publication date: Jul-2021
  • (2021)Multi-label active learning from crowds for secure IIoTAd Hoc Networks10.1016/j.adhoc.2021.102594121:COnline publication date: 23-Aug-2021
  • (2020)Batch Mode Active Learning on the Riemannian Manifold for Automated Scoring of Nuclear Pleomorphism in Breast CancerArtificial Intelligence in Medicine10.1016/j.artmed.2020.101805103:COnline publication date: 1-Mar-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media