Tsuyoshi Okita
ABSTRACT
LLMs for vision and NLP domain has been popular by the widespread use of ChatGPT and GPT-4. This paper tackles to build LLMs for sensor domain of one-dimensional signals whose downstream task is activity recognition and emotion detection. We propose a new architecture of Transformer-based self-supervised learner which we name SENvT. This SENvT builds the LLMs for sensor data using 7 pretext objectives in multi-task learning together with contrastive learning. Experimental results show these three. First, we obtained better results for contrastive learning and the masked token task but not for other pretext tasks. Second, the masked token task was better in 60% rather than in 10%. Third, the RGW worked best in accuracy while the masked token task worked best in F1.
- [1] Philip Bachman, R. Devon Hjelm, and William Buchwalter, Learning representations by maximizing mutual information across views, NeurIPS, 2019.Google Scholar
- [2] Rishi Bommasani, et.al., On the Opportunities and Risks of Foundation Models arXiv, 2108.07258, 2022.Google Scholar
- [3] Barbara Bruno, Fulvio Mastrogiovanni, Antonio Sgorbissa, Tullio Vernazza, and Renato Zaccaria, Analysis of human behavior recognition algorithms based on acceleration data. In 2013 IEEE International Conference on Robotics and Automation, pages 1602–1607. IEEE, 2013Google Scholar
- [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, Emerging Properties in Self-Supervised Vision Transformers, arXiv, 2104.14294, 2021.Google Scholar
- [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ArXiv, 1810.04805, 2018Google Scholar
- [6] Vipula Dissanayake, Sachith Seneviratne, Rajib Rana, Elliott Wen, Tharindu Kaluarachchi, Suranga Nanayakkara, SigRep: Towards Robust Wearable Emotion Recognition with Contrastive Representation Learning, IEEE Access, Volume 10, 2023.Google Scholar
- [7] Carl Doersch, Abhinav Gupta, Alexei A. Efros, Unsupervised Visual Representation Learning by Context Prediction, In proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). IEEE: 1422–1430. 2015.Google Scholar
- [8] Carl Doersch, and Andrew Zisserman, Multi-task Self-Supervised Visual Learning, In Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). IEEE: 2070–2079. arXiv:1708.07860, 2017.Google Scholar
- [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv, 2010.11929, 2020.Google Scholar
- [10] Jonathan Gershuny, Teresa Harms, Aiden Doherty, Testing Self-Report Time-Use Diaries against Objective Instruments in Real Time, Sociological Methodology, Volume 50 Issue 1, pp.318-349, 2020.Google ScholarCross Ref
- [11] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko, Bootstrap your own latent: A new approach to self-supervised Learning, arXiv, 2006.07733, 2020.Google Scholar
- [12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, Masked Autoencoders Are Scalable Vision Learners, arXiv, 2111.06377, 2021.Google Scholar
- [13] Brian Kenji Iwana, Seiichi Uchida, Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher, ICPR 2020.Google Scholar
- [14] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han, On the Variance of the Adaptive Learning Rate and Beyond, arXiv, 1908.03265, 2019.Google Scholar
- [15] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, Hui Xue, Towards Robust Vision Transformer, arXiv, 2105.07926, 2021.Google Scholar
- [16] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng, A Holistic Approach to Undesired Content Detection in the Real World, arXiv, 2208.03274, 2023.Google Scholar
- [17] Mehdi Noroozi, Paolo Favaro, Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles arXiv, 1603.09246, 2016.Google Scholar
- [18] Duc Minh Dimitri Nguyen, Mehdi Miah, Guillaume-Alexandre Bilodeau Wassim Bouachir, Transformers for 1D signals in Parkinson’s disease detection from gait, arXiv, 2204.00423, 2022.Google Scholar
- [19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, arXiv 1807.03748, 2018.Google Scholar
- [20] OpenAI, GPT-4 Technical Report, arXiv, 2303.08774, 2023.Google Scholar
- [21] Attila Reiss and Didier Stricker, Introducing a new benchmarked dataset for activity monitoring. In Proceedings of 2012 16th International Symposium on Wearable Computers, pages 108–109. IEEE. 2012.Google Scholar
- [22] Daniel Roggen, Alberto Calatroni, Mirco Rossi Thomas Holleczek, Kilian Forster, Gerhard Troster Paul Lukowicz, David Bannach, Gerald Pirkl, Florian Wagner, Alois Ferscha, Jakob Doppler, Clemens Holzmann+, Marc Kurz+, Gerald Holl, Walk-through the OPPORTUNITY dataset for activity recognition in sensor rich environments, 2010.Google Scholar
- [23] Yash Sharma, Nick Coronato, Donald E. Brown Encoding Cardiopulmonary Exercise Testing Time Series as Images for Classification using Convolutional Neural Network, arXiv, 2204.12432, 2022.Google Scholar
- [24] Timo Sztyler, and Heiner Stuckenschmidt, On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1–9. IEEE. 2016.Google Scholar
- [25] Yonglong Tian, Dilip Krishnan, and Phillip Isola, Contrastive multiview coding, arXiv 1906.05849, 2019.Google Scholar
- [26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou, Training data-efficient image transformers and distillation through attention, arXiv, 2012.12877v2, 2021.Google Scholar
- [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need, arXiv, 1706.03762, 2017Google Scholar
- [28] Gary M. Weiss, Kenichi Yoneda, and Thaier Hayajneh, Smartphone and smartwatch-based biometrics using activities of daily living. IEEE Access, 7:133190–133202. 2019.Google ScholarCross Ref
- [29] Matthew Willetts, Sven Hollowell, Louis Aslett, Chris Holmes, and Aiden Doherty, Statistical machine learning of sleep & physical activity phenotypes from sensor data in 96,220 uk biobank participants, Scientific Reports, 8(1):1–10, 2018Google ScholarCross Ref
- [30] Hang Yuan, Shing Chan, Andrew P. Creagh, Catherine Tong, David A. Clifton, and Aiden Doherty, Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data, arXiv, 2206.02909, 2022.Google Scholar
- [31] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff, A Transformer-based Framework for Multivariate Time Series Representation Learning, in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), 2021.Google Scholar
- [32] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, Bingyi Kang, BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs, arXiv, 2307.08581, 2023.Google Scholar
Index Terms
- Towards LLMs for Sensor Data: Multi-Task Self-Supervised Learning
Recommendations
Supervised Shallow Multi-task Learning: Analysis of Methods
AbstractThe last decade has witnessed a continuous boom in the application of machine learning techniques in pattern recognition, with much more focus on single-task learning models. However, the increasing amount of multimedia data in the real world also ...
Learning compound tasks without task-specific knowledge via imitation and self-supervised learning
ICML'20: Proceedings of the 37th International Conference on Machine LearningMost real-world tasks are compound tasks that consist of multiple simpler sub-tasks. The main challenge of learning compound tasks is that we have no explicit supervision to learn the hierarchical structure of compound tasks. To address this challenge, ...
Multi-task Self-Supervised Learning for Human Activity Detection
Deep learning methods are successfully used in applications pertaining to ubiquitous computing, pervasive intelligence, health, and well-being. Specifically, the area of human activity recognition (HAR) is primarily transformed by the convolutional and ...
Comments