Abstract
Organising and preprocessing are crucial steps in order to perform analysis on lifelogs. This paper presents a method for preprocessing, enriching, and segmenting lifelogs based on GPS trajectories and images captured from wearable cameras. The proposed method consists of four components: data cleaning, stop/trip point classification, post-processing, and event characterisation. The novelty of this paper lies in the incorporation of a visual module (using a pretrained CLIP model) to improve outlier detection, correct classification errors, and identify each event’s movement mode or location name. This visual component is capable of addressing imprecise boundaries in GPS trajectories and the partition of clusters due to data drift. The results are encouraging, which further emphasises the importance of visual analytics for organising lifelog data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alam, N., Graham, Y., Gurrin, C.: Memento: a prototype lifelog search engine for LSC’21. In: Proceedings of the 4th Annual on Lifelog Search Challenge, pp. 53–58. Association for Computing Machinery (ACM) (2021)
Andrew, A.H., Eustice, K., Hickl, A.: Using location lifelogs to make meaning of food and physical activity behaviors. In: 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops, pp. 408–411. IEEE (2013)
Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Fu, Z., Tian, Z., Xu, Y., Qiao, C.: A two-step clustering approach to extract locations from individual GPS trajectory data. ISPRS Int. J. Geo-Inf. 5(10), 166 (2016)
Gomi, A., Itoh, T.: A personal photograph browser for life log analysis based on location, time, and person. In: Proceedings of the 2011 ACM Symposium on Applied Computing, pp. 1245–1251 (2011)
Gong, L., Sato, H., Yamamoto, T., Miwa, T., Morikawa, T.: Identification of activity stop locations in GPS trajectories by density-based clustering method combined with support vector machines. J. Mod. Transp. 23(3), 202–213 (2015). https://doi.org/10.1007/s40534-015-0079-x
Gouveia, R., Karapanos, E.: Footprint tracker: supporting diary studies with lifelogging. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2921–2930 (2013)
Gurrin, C., Smeaton, A.F., Doherty, A.R., et al.: LifeLogging: personal big data. Found. Trends® Inf. Retrieval 8(1), 1–125 (2014)
Gurrin, C., et al.: Introduction to the fifth annual lifelog search challenge, LSC’22. In: Proceedings of the 2022 International Conference on Multimedia Retrieval, pp. 685–687 (2022)
Heller, S., Rossetto, L., Sauter, L., Schuldt, H.: Vitrivr at the lifelog search challenge 2022. In: Proceedings of the 5th Annual on Lifelog Search Challenge, LSC 2022, pp. 27–31. Association for Computing Machinery, New York (2022)
Hwang, S., Evans, C., Hanke, T.: Detecting stop episodes from GPS trajectories with gaps. In: Thakuriah, P.V., Tilahun, N., Zellner, M. (eds.) Seeing Cities Through Big Data. SG, pp. 427–439. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-40902-3_23
Jia, C., et al.: Scaling up visual and vision-language representation learning with noisy text supervision. arXiv:2102.05918 [cs], June 2021
Kikhia, B., Boytsov, A., Hallberg, J., ul Hussain Sani, Z., Jonsson, H., Synnes, K.: Structuring and presenting lifelogs based on location data. In: Cipresso, P., Matic, A., Lopez, G. (eds.) MindCare 2014. LNICST, vol. 100, pp. 133–144. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11564-1_14
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692 [cs], July 2019
McKenzie, G., Janowicz, K.: Where is also about time: a location-distortion model to improve reverse geocoding using behavior-driven temporal semantic signatures. Comput. Environ. Urban Syst. 54, 1–13 (2015)
Qiu, Z., Gurrin, C., Smeaton, A.F.: Evaluating access mechanisms for multimodal representations of lifelogs. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9516, pp. 574–585. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27671-7_48
Radford, A., et al.: Learning transferable visual models from natural language supervision. arXiv:2103.00020 [cs], February 2021
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs], February 2020
Schoier, G., Borruso, G.: Individual movements and geographical data mining. clustering algorithms for highlighting hotspots in personal navigation routes. In: Murgante, B., Gervasi, O., Iglesias, A., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2011. LNCS, vol. 6782, pp. 454–465. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21928-3_32
Tran, L.D., Nguyen, M.D., Nguyen, B., Lee, H., Zhou, L., Gurrin, C.: E-Myscéal: embedding-based interactive lifelog retrieval system for LSC’22. In: Proceedings of the 5th Annual on Lifelog Search Challenge, LSC 2022, pp. 32–37. Association for Computing Machinery, New York (2022)
Tulving, E.: Precis of elements of episodic memory. Behav. Brain Sci. 7(2), 223–238 (1984)
Zheng, V.W., Zheng, Y., Xie, X., Yang, Q.: Collaborative location and activity recommendations with GPS history data. In: Proceedings of the 19th International Conference on World Wide Web, pp. 1029–1038 (2010)
Zhou, C., Frankowski, D., Ludford, P., Shekhar, S., Terveen, L.: Discovering personally meaningful places: an interactive clustering approach. ACM Trans. Inf. Syst. (TOIS) 25(3), 12-es (2007)
Zimmermann, M., Kirste, T., Spiliopoulou, M.: Finding stops in error-prone trajectories of moving objects with time-based clustering. In: Tavangarian, D., Kirste, T., Timmermann, D., Lucke, U., Versick, D. (eds.) IMC 2009. CCIS, vol. 53, pp. 275–286. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10263-9_24
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tran, LD., Nie, D., Zhou, L., Nguyen, B., Gurrin, C. (2023). VAISL: Visual-Aware Identification of Semantic Locations in Lifelog. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13834. Springer, Cham. https://doi.org/10.1007/978-3-031-27818-1_54
Download citation
DOI: https://doi.org/10.1007/978-3-031-27818-1_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27817-4
Online ISBN: 978-3-031-27818-1
eBook Packages: Computer ScienceComputer Science (R0)