skip to main content
10.1145/1452392.1452407acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
poster

VoiceLabel: using speech to label mobile sensor data

Published: 20 October 2008 Publication History

Abstract

Many mobile machine learning applications require collecting and labeling data, and a traditional GUI on a mobile device may not be an appropriate or viable method for this task. This paper presents an alternative approach to mobile labeling of sensor data called VoiceLabel. VoiceLabel consists of two components: (1) a speech-based data collection tool for mobile devices, and (2) a desktop tool for offline segmentation of recorded data and recognition of spoken labels. The desktop tool automatically analyzes the audio stream to find and recognize spoken labels, and then presents a multimodal interface for reviewing and correcting data labels using a combination of the audio stream, the system's analysis of that audio, and the corresponding mobile sensor data. A study with ten participants showed that VoiceLabel is a viable method for labeling mobile sensor data. VoiceLabel also illustrates several key features that inform the design of other data labeling tools.

References

[1]
Bao, L. and Intille, S. S. 2004. Activity recognition from user-annotated acceleration data. In Proc. of Pervasive 2004.
[2]
Basu, S., Clarkson, B., and Pentland, A. 2001. Smart headphones: enhancing auditory awareness through robust speech detection and source localization. In Proc. of ICASSP 2001.
[3]
Bilmes, J., Li, X., Malkin, J., Kilanski, K., Wright, R., Kirchhoff, K., Subramanya, A., Harada, S., Landay, J., Dowden, P., Chizeck, H. 2005. The Vocal Joystick: A voice-based human-computer interface for individuals with motor impairments. In Proc. of HLT/EMNLP 2005.
[4]
Chen, M, Sohn, T., Chmelev, D., Haehnel, D., Hightower, J., Hughes, J., and LaMarca, A. 2006. Practical metropolitan-scale positioning for GSM phones. In Proc. of Ubicomp 2006.
[5]
Chen, S., Gaur, A., Muthukrishnan, S., and Rosenbluth, D. 2004. Wireless in loco sensor data collection and applications. In Proc. of WWW 2004 Workshop on Emerging Applications for Wireless and Mobile Access.
[6]
Consolvo, S., McDonald, D., Toscos, T., Chen, M., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand, L., Libby, R., Smith, I., and Landay, J. 2008. Activity sensing in the wild: a field trial of UbiFit garden. In Proc. of CHI 2008.
[7]
Fontaine, V., and Bourlard, H. 1997. Speaker-dependent speech recognition based on phone-like units models -- application to voice dialing. In Proc. of ICASSP 1997.
[8]
Goto, M., Itou, K., and Kobayashi, T. 2005. Speech interface exploiting intentionally--controlled nonverbal speech information. In Proc. of UIST 2005.
[9]
Hill, J., Szewczyk, R., Woo, A., Hollar, S., and Pister, D.C.K. 2000. System architecture directions for networked sensors. In Proc. of ASPLOS 2000.
[10]
Lamming, M., and Bohm, D. 2003. SPECs: Another approach to human context and activity sensing research, using tiny peer-to-peer wireless computers. In Proc. of Ubicomp 2003.
[11]
Lester, J., Choudhury, T., Kern, N., Borriello, G., and Hannaford, B. 2005. A hybrid discriminative-generative approach for modeling human activities. In Proc. of IJCAI 2005.
[12]
Lester, J., Choudhury, T., and Borriello, G. 2006. A practical approach to recognizing physical activities. In Proc. of Pervasive 2006.
[13]
Logan, B., Healey, J., Philipose, M., Tapia, E.M., and Intille, S. 2007. A long-term evaluation of sensing modalities for activity recognition. In Proc. of Ubicomp 2007.
[14]
Nissenbaum, H. 1998. Protecting privacy in an information age: The problem of privacy in public. In Law and Philosophy 17 (5-6).
[15]
Shilman, M., Tan, D., and Simard, P., 2006. CueTIP: A mixed-initiative interface for correcting handwriting errors. In Proc. of UIST 2006.
[16]
Saponas, T., Lester, J., Froehlich, J, Fogarty, J., Landay, J. 2008. iLearn on the iPhone: Real-Time Human Activity Classification on Commodity Mobile Phones. University of Washington CSE Tech Report UW-CSE-08-04-02.
[17]
Tapia, E.M., Intille, S.S., and Larson, K. 2004. Activity recognition in the home setting using simple and ubiquitous sensors. In Proc. of Pervasive 2004.

Cited By

View all
  • (2020)Combining Participatory and ESM: A Hybrid Approach to Collecting Annotated Mobility DataExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3383066(1-7)Online publication date: 25-Apr-2020
  • (2019)The State of Speech in HCI: Trends, Themes and ChallengesInteracting with Computers10.1093/iwc/iwz01631:4(349-371)Online publication date: 11-Sep-2019
  • (2019)Designing and evaluating mobile self-reporting techniquesPersonal and Ubiquitous Computing10.1007/s00779-019-01207-223:2(329-338)Online publication date: 1-Apr-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces
October 2008
322 pages
ISBN:9781605581989
DOI:10.1145/1452392
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. data collection
  2. machine learning
  3. mobile devices
  4. sensors
  5. speech recognition

Qualifiers

  • Poster

Conference

ICMI '08
Sponsor:
ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES
October 20 - 22, 2008
Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)Combining Participatory and ESM: A Hybrid Approach to Collecting Annotated Mobility DataExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3383066(1-7)Online publication date: 25-Apr-2020
  • (2019)The State of Speech in HCI: Trends, Themes and ChallengesInteracting with Computers10.1093/iwc/iwz01631:4(349-371)Online publication date: 11-Sep-2019
  • (2019)Designing and evaluating mobile self-reporting techniquesPersonal and Ubiquitous Computing10.1007/s00779-019-01207-223:2(329-338)Online publication date: 1-Apr-2019
  • (2017)An investigation of using mobile and situated crowdsourcing to collect annotated travel activity data in real-word settingsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2016.11.001102:C(81-102)Online publication date: 1-Jun-2017
  • (2016)ATLAYA : Relaxed Labeling and Flexible Analaysis Environment by Integration of Annotation and Analysis ToolsJournal of Japan Society for Fuzzy Theory and Intelligent Informatics10.3156/jsoft.28.89928:6(899-910)Online publication date: 2016
  • (2016)Exploring human activity annotation using a privacy preserving 3D modelProceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct10.1145/2968219.2968290(803-812)Online publication date: 12-Sep-2016
  • (2015)Leveraging Dual-Observable Input for Fine-Grained Thumb Interaction Using Forearm EMGProceedings of the 28th Annual ACM Symposium on User Interface Software & Technology10.1145/2807442.2807506(523-528)Online publication date: 5-Nov-2015
  • (2015)A field study comparing approaches to collecting annotated activity data in real-world settingsProceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/2750858.2807524(671-682)Online publication date: 7-Sep-2015
  • (2014)Improving Pointing in Graphical User Interfaces for People with Motor Impairments Through Ability-Based DesignAssistive Technologies and Computer Access for Motor Disabilities10.4018/978-1-4666-4438-0.ch008(206-253)Online publication date: 2014
  • (2014)Evaluation of Prompted Annotation of Activity Data Recorded from a Smart PhoneSensors10.3390/s14091586114:9(15861-15879)Online publication date: 27-Aug-2014
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media