Wei-Hong Ang
ABSTRACT
The major challenge in visual lifelog retrieval is the semantic gap between textual queries and visual concepts. This paper presents our work on the Lifelog Search Challenge 2021 (LSC'21), an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multimodal lifelogs. We propose LifeConcept, an interactive lifelog search system that is aimed at accelerating the retrieval process and retrieving more precise results. In this work, we introduce several new features such as the number of people, location cluster, and object with color. Moreover, we obtain visual concepts from the images with computer vision models and propose a concept recommendation method to reduce the semantic gap. In this way, users can efficiently set up the related conditions for their requirements and search the desired images with appropriate query terms based on the suggestion.
- Marc Bolanos, Ricard Mestre, Estefan'ia Talavera, Xavier Giró-i Nieto, and Petia Radeva. 2015. Visual summary of egocentric photostreams by representative keyframes. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 1--6.Google Scholar
Cross Ref
- M. Bolaños, M. Dimiccoli, and P. Radeva. 2017. Toward Storytelling From Visual Lifelogging: An Overview. IEEE Transactions on Human-Machine Systems, Vol. 47, 1 (2017), 77--90. https://doi.org/10.1109/THMS.2016.2616296Google Scholar
- G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).Google Scholar
- Tai-Te Chu, Chia-Chun Chang, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 51--55.Google ScholarDigital Library
- Cathal Gurrin, Bjorn Þór Jonsson, Klaus Schoffmann, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy. 2021. Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21. In Proc. International Conference on Multimedia Retrieval (ICMR'21). ACM, Taipei, Taiwan.Google ScholarDigital Library
- T Karthikeyan, P Manikandaprabhu, and S Nithya. 2014. A survey on text and content based image retrieval system for image mining. International Journal of Engineering, Vol. 3 (2014).Google Scholar
- Afshan Latif, Aqsa Rasheed, Umer Sajid, Jameel Ahmed, Nouman Ali, Naeem Iqbal Ratyal, Bushra Zafar, Saadat Hanif Dar, Muhammad Sajid, and Tehmina Khalil. 2019. Content-based image retrieval and feature extraction: a comprehensive review. Mathematical Problems in Engineering, Vol. 2019 (2019).Google Scholar
- Jiayu Li, Min Zhang, Weizhi Ma, Yiqun Liu, and Shaoping Ma. 2020. A Multi-level Interactive Lifelog Search Engine with User Feedback. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 29--35.Google ScholarDigital Library
- Yair Poleg, Chetan Arora, and Shmuel Peleg. 2014. Temporal segmentation of egocentric videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2537--2544.Google ScholarDigital Library
- Peng Wang and Alan F Smeaton. 2012. Semantics-based selection of everyday concepts in visual lifelogging. International Journal of Multimedia Information Retrieval, Vol. 1, 2 (2012), 87--101.Google ScholarCross Ref
- Bo Xiong and Kristen Grauman. 2014. Detecting snap points in egocentric video with a web photo prior. In European conference on computer vision. Springer, 282--298.Google ScholarCross Ref
- An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021. Ten Questions in Lifelog Mining and Information Recall. In Proceedings of ACM International Conference on Multimedia Retrieval 2021 (ICMR 2021) (2021).Google ScholarDigital Library
Index Terms
- LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation
Recommendations
Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021
LSC '21: Proceedings of the 4th Annual on Lifelog Search ChallengeThe Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured,...
Multimodal retrieval with relevance feedback based on genetic programming
This paper presents a framework for multimodal retrieval with relevance feedback based on genetic programming. In this supervised learning-to-rank framework, genetic programming is used for the discovery of effective combination functions of (multimodal)...
Visual lifelog retrieval: humans and machines interpretation on first-person images
AbstractPeople usually forget the details of life experiences and encounter situations where they require to recall their past experiences. Therefore, lifelog retrieval turns out to be an emerging task in the AI community. Nowadays, people can record ...
Comments