skip to main content
10.1145/3594739.3610745acmconferencesArticle/Chapter ViewAbstractPublication PagesubicompConference Proceedingsconference-collections
research-article

Towards LLMs for Sensor Data: Multi-Task Self-Supervised Learning

Published: 08 October 2023 Publication History

Abstract

LLMs for vision and NLP domain has been popular by the widespread use of ChatGPT and GPT-4. This paper tackles to build LLMs for sensor domain of one-dimensional signals whose downstream task is activity recognition and emotion detection. We propose a new architecture of Transformer-based self-supervised learner which we name SENvT. This SENvT builds the LLMs for sensor data using 7 pretext objectives in multi-task learning together with contrastive learning. Experimental results show these three. First, we obtained better results for contrastive learning and the masked token task but not for other pretext tasks. Second, the masked token task was better in 60% rather than in 10%. Third, the RGW worked best in accuracy while the masked token task worked best in F1.

References

[1]
[1] Philip Bachman, R. Devon Hjelm, and William Buchwalter, Learning representations by maximizing mutual information across views, NeurIPS, 2019.
[2]
[2] Rishi Bommasani, et.al., On the Opportunities and Risks of Foundation Models arXiv, 2108.07258, 2022.
[3]
[3] Barbara Bruno, Fulvio Mastrogiovanni, Antonio Sgorbissa, Tullio Vernazza, and Renato Zaccaria, Analysis of human behavior recognition algorithms based on acceleration data. In 2013 IEEE International Conference on Robotics and Automation, pages 1602–1607. IEEE, 2013
[4]
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, Emerging Properties in Self-Supervised Vision Transformers, arXiv, 2104.14294, 2021.
[5]
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ArXiv, 1810.04805, 2018
[6]
[6] Vipula Dissanayake, Sachith Seneviratne, Rajib Rana, Elliott Wen, Tharindu Kaluarachchi, Suranga Nanayakkara, SigRep: Towards Robust Wearable Emotion Recognition with Contrastive Representation Learning, IEEE Access, Volume 10, 2023.
[7]
[7] Carl Doersch, Abhinav Gupta, Alexei A. Efros, Unsupervised Visual Representation Learning by Context Prediction, In proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). IEEE: 1422–1430. 2015.
[8]
[8] Carl Doersch, and Andrew Zisserman, Multi-task Self-Supervised Visual Learning, In Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). IEEE: 2070–2079. arXiv:1708.07860, 2017.
[9]
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv, 2010.11929, 2020.
[10]
[10] Jonathan Gershuny, Teresa Harms, Aiden Doherty, Testing Self-Report Time-Use Diaries against Objective Instruments in Real Time, Sociological Methodology, Volume 50 Issue 1, pp.318-349, 2020.
[11]
[11] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko, Bootstrap your own latent: A new approach to self-supervised Learning, arXiv, 2006.07733, 2020.
[12]
[12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, Masked Autoencoders Are Scalable Vision Learners, arXiv, 2111.06377, 2021.
[13]
[13] Brian Kenji Iwana, Seiichi Uchida, Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher, ICPR 2020.
[14]
[14] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han, On the Variance of the Adaptive Learning Rate and Beyond, arXiv, 1908.03265, 2019.
[15]
[15] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, Hui Xue, Towards Robust Vision Transformer, arXiv, 2105.07926, 2021.
[16]
[16] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng, A Holistic Approach to Undesired Content Detection in the Real World, arXiv, 2208.03274, 2023.
[17]
[17] Mehdi Noroozi, Paolo Favaro, Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles arXiv, 1603.09246, 2016.
[18]
[18] Duc Minh Dimitri Nguyen, Mehdi Miah, Guillaume-Alexandre Bilodeau Wassim Bouachir, Transformers for 1D signals in Parkinson’s disease detection from gait, arXiv, 2204.00423, 2022.
[19]
[19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, arXiv 1807.03748, 2018.
[20]
[20] OpenAI, GPT-4 Technical Report, arXiv, 2303.08774, 2023.
[21]
[21] Attila Reiss and Didier Stricker, Introducing a new benchmarked dataset for activity monitoring. In Proceedings of 2012 16th International Symposium on Wearable Computers, pages 108–109. IEEE. 2012.
[22]
[22] Daniel Roggen, Alberto Calatroni, Mirco Rossi Thomas Holleczek, Kilian Forster, Gerhard Troster Paul Lukowicz, David Bannach, Gerald Pirkl, Florian Wagner, Alois Ferscha, Jakob Doppler, Clemens Holzmann+, Marc Kurz+, Gerald Holl, Walk-through the OPPORTUNITY dataset for activity recognition in sensor rich environments, 2010.
[23]
[23] Yash Sharma, Nick Coronato, Donald E. Brown Encoding Cardiopulmonary Exercise Testing Time Series as Images for Classification using Convolutional Neural Network, arXiv, 2204.12432, 2022.
[24]
[24] Timo Sztyler, and Heiner Stuckenschmidt, On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1–9. IEEE. 2016.
[25]
[25] Yonglong Tian, Dilip Krishnan, and Phillip Isola, Contrastive multiview coding, arXiv 1906.05849, 2019.
[26]
[26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou, Training data-efficient image transformers and distillation through attention, arXiv, 2012.12877v2, 2021.
[27]
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need, arXiv, 1706.03762, 2017
[28]
[28] Gary M. Weiss, Kenichi Yoneda, and Thaier Hayajneh, Smartphone and smartwatch-based biometrics using activities of daily living. IEEE Access, 7:133190–133202. 2019.
[29]
[29] Matthew Willetts, Sven Hollowell, Louis Aslett, Chris Holmes, and Aiden Doherty, Statistical machine learning of sleep & physical activity phenotypes from sensor data in 96,220 uk biobank participants, Scientific Reports, 8(1):1–10, 2018
[30]
[30] Hang Yuan, Shing Chan, Andrew P. Creagh, Catherine Tong, David A. Clifton, and Aiden Doherty, Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data, arXiv, 2206.02909, 2022.
[31]
[31] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff, A Transformer-based Framework for Multivariate Time Series Representation Learning, in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), 2021.
[32]
[32] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, Bingyi Kang, BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs, arXiv, 2307.08581, 2023.

Cited By

View all
  • (2024)Proposal of User Interface Based on Heavy User Usage Analysis in LLM ServiceArchives of Design Research10.15187/adr.2024.08.37.4.28737:4(287-313)Online publication date: 31-Aug-2024
  • (2024)Analysis of Human Activity Recognition by Diffusion ModelsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678439(458-463)Online publication date: 5-Oct-2024

Index Terms

  1. Towards LLMs for Sensor Data: Multi-Task Self-Supervised Learning

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UbiComp/ISWC '23 Adjunct: Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing
    October 2023
    822 pages
    ISBN:9798400702006
    DOI:10.1145/3594739
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. LLMs
    2. activity recognition
    3. multi-task self-supervised learning

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    UbiComp/ISWC '23

    Acceptance Rates

    Overall Acceptance Rate 764 of 2,912 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)225
    • Downloads (Last 6 weeks)10
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Proposal of User Interface Based on Heavy User Usage Analysis in LLM ServiceArchives of Design Research10.15187/adr.2024.08.37.4.28737:4(287-313)Online publication date: 31-Aug-2024
    • (2024)Analysis of Human Activity Recognition by Diffusion ModelsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678439(458-463)Online publication date: 5-Oct-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media