skip to main content
research-article

Memory recall based video search: Finding videos you have seen before based on your memory

Published: 14 February 2014 Publication History

Abstract

We often remember images and videos that we have seen or recorded before but cannot quite recall the exact venues or details of the contents. We typically have vague memories of the contents, which can often be expressed as a textual description and/or rough visual descriptions of the scenes. Using these vague memories, we then want to search for the corresponding videos of interest. We call this “Memory Recall based Video Search” (MRVS). To tackle this problem, we propose a video search system that permits a user to input his/her vague and incomplete query as a combination of text query, a sequence of visual queries, and/or concept queries. Here, a visual query is often in the form of a visual sketch depicting the outline of scenes within the desired video, while each corresponding concept query depicts a list of visual concepts that appears in that scene. As the query specified by users is generally approximate or incomplete, we need to develop techniques to handle this inexact and incomplete specification by also leveraging on user feedback to refine the specification. We utilize several innovative approaches to enhance the automatic search. First, we employ a visual query suggestion model to automatically suggest potential visual features to users as better queries. Second, we utilize a color similarity matrix to help compensate for inexact color specification in visual queries. Third, we leverage on the ordering of visual queries and/or concept queries to rerank the results by using a greedy algorithm. Moreover, as the query is inexact and there is likely to be only one or few possible answers, we incorporate an interactive feedback loop to permit the users to label related samples which are visually similar or semantically close to the relevant sample. Based on the labeled samples, we then propose optimization algorithms to update visual queries and concept weights to refine the search results. We conduct experiments on two large-scale video datasets: TRECVID 2010 and YouTube. The experimental results demonstrate that our proposed system is effective for MRVS tasks.

References

[1]
A. Amir, J. Argillandery, et al. 2005. IBM Research TRECVID-2005 video retrieval system. In Proceedings of the TRECVID Workshop.
[2]
F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. 2004. Multiple kernel learning, conic duality, and the SMO algorithm. In Proceedings of the International Conference on Machine Learning.
[3]
P. Browne and A. F. Smeaton. 2005. Video retrieval using dialogue, keyframe similarity and video objects. In Proceedings of the International Conference on Image Process 3, 1208--1211.
[4]
L. Chaisorn, K. W. Wan, et al. 2010. TRECVID 2010 Known-item Search (KIS) task by I2R. In Proceedings of the TRECVID Workshop.
[5]
S.-F. Chang, W. H. Hsu, W. Jiang, L. S. Kennedy, D. Xu, A. Yanagawa, and E. Zavesky. 2006. Columbia University Trecvid-2006 video search and high-level feature extraction. In Proceedings of the TRECVID Workshop.
[6]
X. Y. Chen, J. Yuan, et al. 2010. TRECVID 2010 known-item search by NUS. In Proceedings of the TRECVID Workshop.
[7]
M. D. Fairchild. 2005. Color Appearance Models 2nd Ed. Addison-Wesley.
[8]
M. R. Hestenes. 1969. Multiplier and gradient methods. J. Optimization Theory Appl. 303--320.
[9]
W. H. Hsu, L. S. Kennedy, and S. F. Chang. 2007. Reranking methods for visual search. IEEE Trans. Multimedia 14, 14--22.
[10]
W. M. Hu, D. Xie, Z. Y. Fu, W. R. Zeng, and S. Maybank. 2007. Semantic based surveillance video retrieval. IEEE Trans. Image Process 16, 1168--1181.
[11]
W. M. Hu, N. H. Xie, L. Li, and X. L. Zeng. 2011. A survey on visual content-based video indexing and retrieval. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 41, 797--819.
[12]
L. Kennedy and S.-F. Chang. 2010. Visual ontology construction and concept detection for multimedia indexing and retrieval. In Semantic Computing, 155.
[13]
L. Kennedy, A. P. Natsev, and S.-F. Chang. 2005. Automatic discovery of query-class-dependent models for multimodal search. In Proceedings of the ACM International Conference on Multimedia.
[14]
Y. Liu, T. Mei, and X. S. Hua. 2009a. CrowdReranking: Exploring multiple search engines for visual search reranking. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 500--507.
[15]
Y. Liu, Tao Mei, X. Q. Wu, and X.-S. Hua. 2009b. Multigraph-based query-independent learning for video search. IEEE Trans. Circuits Syst. Video Technol. 19, 12, 1841--1850.
[16]
Y. F. Ma and H. J. Zhang. 2002. Motion texture: A new motion based video representation. In Proceedings of the International Conference on Pattern Recognition. 548--551.
[17]
C. D. Manning, P. Raghavan, and H. Schtze. 2009. An Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK.
[18]
S.-Y. Neo, J. Zhao, M.-Y. Kan, and T.-S. Chua. 2006. Video retrieval using high level features: Exploiting query matching and confidence-based weighting. In Proceedings of the ACM International Conference on Image and Video Retrieval.
[19]
L. Q. Nie, M. Wang, Z.-J. Zha, G. D. Li, and T.-S. Chua. 2011. Multimedia answering: Enriching Text QA with media information. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 695--704.
[20]
J. Sivic, M. Everingham, and A. Zisserman. 2005. Person spotting: Video shot retrieval for Dace sets. In Proceedings of the International Conference on Image Video Retrieval.
[21]
C. G. M. Snoek, B. Huurnink, et al. 2007. Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9, 975--986.
[22]
C. G. M. Snoek, K. E. A. VandeSande, et al. 2008. The MediaMill TRECVID 2008 semantic video search engine. In Proceedings of the TRECVID Workshop.
[23]
C. G. M. Snoek and M. Worring. 2009. Concept-based video retrieval. In Foundations and Trends in Information Retrieval 2, 215--322.
[24]
TRECVID2010. 2010. TRECVID2010. http://www-nlpir.nist.gov/projects/tv2010/tv2010.html (2010).
[25]
D. Wang, X. Li, J. Li, and B. Zhang. 2007. The importance of query concept-mapping for automatic video retrieval. In Proceedings of the International Conference on Multimedia. 285--288.
[26]
R. Yan and A. G. Hauptmann. 2007. A review of text and image retrieval approaches for broadcast news video. Inf. Retrieval 10, 445--484.
[27]
A. Yanagawa, S.-F. Chang, L. Kennedy, and W. H. Hsu. 2007. Columbia University's baseline detectors for 374 LSCOM semantic visual concepts. ADVENT Tech. rep. 222-2006-8.
[28]
J. Yang and A. G. Hauptmann. 2006. Exploring temporal consistency for video analysis and retrieval. In Proceedings of the International Conference on Multimedia Information Retrieval.
[29]
T. Yao, C.-W. Ngo, and T. Mei. 2013. Circular reranking for visual search. IEEE Trans. Image Process. 22, 4, 1644--1655.
[30]
J. Yuan, Z.-J. Zha, Y.-T. Zheng, M. Wang, X. D. Zhou, and T.-S. Chua. 2011a. Learning concept bundles for video search with complex queries. In Proceedings of the ACM International Conference on Multimedia.
[31]
J. Yuan, Z.-J. Zha, Y.-T. Zheng, M. Wang, X. D. Zhou, and T.-S. Chua. 2011b. Utilizing related samples to enhance interactive concept-based video search. IEEE Trans. Multimedia 13, 1343--1355.
[32]
E. Zavesky and S.-F. Chang. 2008. CuZero: Embracing the Frontier of interactive visual search for informed users. In Proceedings of the International Conference on Multimedia Information Retrieval.
[33]
Z. J. Zha, L. J. Yang, T. Mei, M. Wang, and Z. F. Wang. 2009. Visual query suggestion. In Proceedings of the International Conference on Multimedia.

Cited By

View all
  • (2023)Transform-Equivariant Consistency Learning for Temporal Sentence GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363474920:4(1-19)Online publication date: 27-Nov-2023
  • (2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
  • (2018)The social mediated age of information: Twitter and Instagram as tools for information dissemination in higher educationNew Media & Society10.1177/146144481876825920:11(4155-4176)Online publication date: 7-Apr-2018
  • Show More Cited By

Index Terms

  1. Memory recall based video search: Finding videos you have seen before based on your memory

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 10, Issue 2
    February 2014
    142 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/2579228
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 14 February 2014
    Accepted: 01 September 2013
    Revised: 01 April 2013
    Received: 01 October 2012
    Published in TOMM Volume 10, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Memory recall based video search
    2. color similarity matrix
    3. interactive video search
    4. related sample
    5. visual query suggestion

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)17
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Transform-Equivariant Consistency Learning for Temporal Sentence GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363474920:4(1-19)Online publication date: 27-Nov-2023
    • (2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
    • (2018)The social mediated age of information: Twitter and Instagram as tools for information dissemination in higher educationNew Media & Society10.1177/146144481876825920:11(4155-4176)Online publication date: 7-Apr-2018
    • (2018)Multimedia Retrieval that Works2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR.2018.00019(63-68)Online publication date: Apr-2018
    • (2018)VideoHandlesComputers and Graphics10.1016/j.cag.2015.01.00448:C(99-106)Online publication date: 23-Dec-2018
    • (2016)Semantic Photo Retargeting Under Noisy Image LabelsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/288677512:3(1-22)Online publication date: 20-May-2016
    • (2016)Integration of Visual Temporal Information and Textual Distribution Information for News Web Video Event MiningIEEE Transactions on Human-Machine Systems10.1109/THMS.2015.248968146:1(124-135)Online publication date: Feb-2016
    • (2016)Metric Feature Indexing for Interactive Multimedia Search2016 13th Conference on Computer and Robot Vision (CRV)10.1109/CRV.2016.22(109-115)Online publication date: Jun-2016
    • (2015)Efficient Activity Retrieval through Semantic Graph QueriesProceedings of the 23rd ACM international conference on Multimedia10.1145/2733373.2806229(391-400)Online publication date: 13-Oct-2015
    • (2015)aMM: Towards adaptive ranking of multi-modal documentsInternational Journal of Multimedia Information Retrieval10.1007/s13735-015-0088-x4:4(233-245)Online publication date: 28-Sep-2015

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media