research-article

Real-time crowd labeling for deployable activity recognition

Authors:

Walter S. Lasecki,

Young Chol Song,

Jeffrey P. BighamAuthors Info & Claims

CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work

Pages 1203 - 1212

https://doi.org/10.1145/2441776.2441912

Published: 23 February 2013 Publication History

Abstract

Systems that automatically recognize human activities offer the potential of timely, task-relevant information and support. For example, prompting systems can help keep people with cognitive disabilities on track and surveillance systems can warn of activities of concern. Current automatic systems are difficult to deploy because they cannot identify novel activities, and, instead, must be trained in advance to recognize important activities. Identifying and labeling these events is time consuming and thus not suitable for real-time support of already-deployed activity recognition systems. In this paper, we introduce Legion:AR, a system that provides robust, deployable activity recognition by supplementing existing recognition systems with on-demand, real-time activity identification using input from the crowd.

Legion:AR uses activity labels collected from crowd workers to train an automatic activity recognition system online to automatically recognize future occurrences. To enable the crowd to keep up with real-time activities, Legion:AR intelligently merges input from multiple workers into a single ordered label set. We validate Legion:AR across multiple domains and crowds and discuss features that allow appropriate privacy and accuracy tradeoffs.

References

[1]

L. Bao and S. S. Intille Activity Recognition from User-Annotated Acceleration Data. In Proc. of Pervasive 2004, v. 3001/2004, pp. 1--17, 2004.

[2]

J. P. Bigham, R. E. Ladner, and Y. Borodin. The Design of the Human-Backed Access Technology. In Proc. of the ACM Intl. Conf. on Computers and Accessibility (ASSETS 2011), pp. 3--10, 2011.

Digital Library

[3]

J. P Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatrowicz, B. White, S. White, and T. Yeh. VizWiz: Nearly Real-time Answers to Visual Questions. Proc. of the ACM Symp. on User Interface Software and Technology (UIST 2010), pp. 333--342, 2010.

Digital Library

[4]

Michael S. Bernstein, Joel R. Brandt, Robert C. Miller, and David R. Karger. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proc. of the 24th Symp. on User Interface Software and Technology (UIST 2011), 2011.

Digital Library

[5]

M. S. Bernstein, D. R. Karger, R. C. Miller, and J. Brandt. Analytic methods for optimizing realtime crowdsourcing. In Proc. of Collective Intelligence (CI 2012), To Appear, 2012.

[6]

M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. Proc. of the ACM Sypm. on User Interface Software and Tech. (UIST 2010), pp. 313--322, 2010.

Digital Library

[7]

M. Boyle, C. Edwards, and S. Greenberg. The effects of filtered video on awareness and privacy. Proceedings of the 2000 ACM conference on Computer supported cooperative work, pp. 1--10, 2000.

Digital Library

[8]

Y. Chu, Y. C. Song, H. Kautz, and R. Levinson. When Did You Start Doing That Thing That You Do? Interactive Activity Recognition and Prompting. AAAI 2011 Workshop on Artificial Intelligence and Smarter Living. 2011.

[9]

S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, Z. Popovic, and FoldIt Players. Predicting protein structures with a multiplayer online game. Nature, 466(7307):756--760, 2010.

[10]

T. Duong, H. Bui, D. Phung, and S. Venkatesh. Activity Recognition and Abnormality Detection with the Switching Hidden Semi-Markov Model. Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. I, pp. 838--845, 2005.

Digital Library

[11]

L. Fiore, D. Fehr, R. Bodor, A. Drenner, G. Somasundaram, and N. Papanikolopoulos. Multi-camera human activity monitoring. Journal of Intelligent and Robotic Systems, vol. 52, pp. 5--43, 2008.

Digital Library

[12]

S. Hudson, and I. Smith. Techniques for addressing fundamental privacy and disruption tradeoffs in awareness support systems. Proceedings of the 1996 ACM conference on Computer supported cooperative work, pp. 248--257, 1996.

Digital Library

[13]

N. Lane, Y. Xu, H. Lu, S. Hu, T. Choudhury, A. Campbell, and F. Zhao. Enabling Large-scale Human Activity Inference on Smartphones using Community Similarity Networks Proc. of Ubicomp 2011, 2011.

Digital Library

[14]

W. S. Lasecki, J. P. Bigham. Online Quality Control for Real-time Crowd Captioning. In Proc. of the ACM International Conference on Computers and Accessibility (ASSETS 2012), pp. 143--150, 2012.

Digital Library

[15]

W. S. Lasecki, K. I. Murray, S. White, R. C. Miller, J. P. Bigham. Real-Time Crowd Control of Existing Interfaces. In Proc. of the ACM Symp. on User Interface Software and Technology (UIST 2011), pp. 23--32, 2011.

Digital Library

[16]

W. S. Lasecki, C. D. Miller, A. Sadilek, A. Abumoussa, D. Borrello, R. Kushalnagar, J. P. Bigham. Real-Time Captioning by Groups of Non-Experts. In Proc. of the ACM Symp. on User Interface Software and Technology (UIST 2012), pp. 23--34, 2012.

Digital Library

[17]

B. Logan, J. Healey, M. Philipose, E. Tapia, and S. Intille. A Long-term Evaluation of Sensing Modalities for Activity Recognition. Proceedings of the 9th international conference on Ubiquitous computing, pp. 483--500, 2007.

Digital Library

[18]

G. Miller. Wordnet: A lexical database for English. Comm. ACM 38, 11. pp. 3941, 1995.

Digital Library

[19]

J. Modayil, R. Levinson, C. Harman, D. Halper, and H. Kautz. Integrating Sensing and Cueing for more effective Activity Reminders. Proc. of the AAAI Fall 2008 Symp. on AI in Eldercare: New Solutions to Old Problems. pp. 60--66, 2008.

[20]

E. D. Mynatt, A. S. Melenhorst, A. D. Fisk, W. A. Rogers. Aware technologies for aging in place: understanding user needs and attitudes. Pervasive Computing, IEEE, v. 3-2, pp. 36--41, 2004.

Digital Library

[21]

W. Niu, J. Long, D. Han, and Y.-F. Wang. Human activity detection and recognition for video surveillance. Proc. of ICME 2004, pp. 719--722, 2005.

[22]

S. Park and H. Kautz Privacy-Preserving Recognition of Activities in Daily Living From Multi-View Silhouettes and RFID-Based Training. AAAI Symp. on AI in Eldercare: New Solutions to Old Problems. 2008.

[23]

A. Shinsel, T. Kulesza, M. Burnett, W. Curran, A. Groce, S. Stumpf, and W.-K. Wong. Mini-Crowdsourcing End-User Assessment of Intelligent Assistants: A Cost-Benefit Study. Proc. of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC '11). pp. 47--54, 2011.

[24]

D. Patterson, D. Fox, H. Kautz, M. Philipose. Finegrained activity recognition by aggregating abstract object usage. In ISWC 05: Proceedings of the Ninth IEEE International Symposium on Wearable Computers. pp. 4451. 2005.

Digital Library

[25]

W. Pentney, et al. Sensor-based Understanding of Daily Life via Large-Scale Use of Common Sense. Proceedings of AAAI. 2006.

Digital Library

[26]

A. Sadilek, and H. Kautz. Recognizing Multi-Agent Activities from GPS Data. In 24th AAAI Conference on Artificial Intelligence (AAAI 2010). 2010.

[27]

A. Sadilek, and H. Kautz. Modeling and Reasoning about Success, Failure, and Intent of Multi-Agent Activities. In Proc. of UbiComp 2010.

[28]

Y. Sun, S. Roy, G. Little. Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks. AAAI Workshop on Human Computation. 2011.

[29]

E. Tapia, S. Intille, and K. Larson. Activity Recognition in the Home Using Simple and Ubiquitous Sensors. Pervasive Computing, pp. 158--175, 2004.

[30]

M. Toomim, T. Kriplean, C. Prtner, and J. A. Landay. Utility of human-computer interactions: Toward a science of preference measurement. In Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI 2011), pp. 2275--2284, 2011.

Digital Library

[31]

L. von Ahn, and L. Dabbish. Labeling images with a computer game. Proc. of the Conf. on Human Factors in Computing Systems (CHI 2004), pp. 319--326, 2004.

Digital Library

[32]

D. Wilson, and C. Atkeson. Simultaneous tracking and activity recognition (STAR) using many anonymous, binary sensors. Proc. of PERVASIVE 2005, pp. 62--79. 2005.

Digital Library

[33]

Y. Yan, R. Romer, F. Glenn, and D. Jennifer. Active Learning from Crowds. Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1161--1168, 2011.

Digital Library

[34]

L. Zhao, G. Sukthankar, and R. Sukthankar. Robust Active Learning Using Crowdsourced Annotations for Activity Recognition. AAAI 2011 Workshop on Human Computation. pp. 74--79, 2011.

[35]

L. Zhao, G. Sukthankar, and R. Sukthankar. Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations. IEEE Intl. Conf. on Social Computing. 2011.

Cited By

Zhang XCui SZhu TChen LZhou FNing H(2024)CASL: Capturing Activity Semantics Through Location Information for Enhanced Activity RecognitionIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2023.323806421:4(1051-1059)Online publication date: Jul-2024
https://doi.org/10.1109/TCBB.2023.3238064
Jones MByun CJohnson NSeppi K(2022)Understanding the Roles of Video and Sensor Data in the Annotation of Human ActivitiesInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210158939:18(3634-3648)Online publication date: Aug-2022
https://doi.org/10.1080/10447318.2022.2101589
Dudley JJacques JKristensson PKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Crowdsourcing Design Guidance for Contextual Adaptation of Text Content in Augmented RealityProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445493(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445493
Show More Cited By

Index Terms

Real-time crowd labeling for deployable activity recognition
1. Information systems
  1. Information systems applications

Recommendations

Time handling for real-time progressive activity recognition
SAGAware '11: Proceedings of the 2011 international workshop on Situation activity & goal awareness

In a dense sensor-based smart home (SH), a significant challenge is to segment the sensor data stream in real-time to continuously support progressive activity recognition (AR). In this paper, we evaluate an approach that supports the segmentation of ...
Real world activity recognition with multiple goals
UbiComp '08: Proceedings of the 10th international conference on Ubiquitous computing

Recognizing and understanding the activities of people from sensor readings is an important task in ubiquitous computing. Activity recognition is also a particularly difficult task because of the inherent uncertainty and complexity of the data collected ...
Activity recognition with hand-worn magnetic sensors

Activity recognition is a key technology for realizing ambient assisted living applications such as care of the elderly and home automation. This paper proposes a new activity recognition method that employs hand-worn magnetic sensors to recognize a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work

February 2013

1594 pages

ISBN:9781450313315

DOI:10.1145/2441776

General Chairs:
Amy Bruckman
Georgia Institute of Technology, USA
,
Scott Counts
Microsoft Research, USA
,
Program Chairs:
Cliff Lampe
University of Michigan, USA
,
Loren Terveen
University of Minnesota, USA

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 February 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CSCW '13

Sponsor:

SIGCHI

CSCW '13: Computer Supported Cooperative Work

February 23 - 27, 2013

Texas, San Antonio, USA

Acceptance Rates

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

93
Total Citations
View Citations
1,008
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XCui SZhu TChen LZhou FNing H(2024)CASL: Capturing Activity Semantics Through Location Information for Enhanced Activity RecognitionIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2023.323806421:4(1051-1059)Online publication date: Jul-2024
https://doi.org/10.1109/TCBB.2023.3238064
Jones MByun CJohnson NSeppi K(2022)Understanding the Roles of Video and Sensor Data in the Annotation of Human ActivitiesInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210158939:18(3634-3648)Online publication date: Aug-2022
https://doi.org/10.1080/10447318.2022.2101589
Dudley JJacques JKristensson PKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Crowdsourcing Design Guidance for Contextual Adaptation of Text Content in Augmented RealityProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445493(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445493
Agrawal AAhuja R(2021)Deep Learning Algorithms for Human Activity Recognition: A Comparative AnalysisCybernetics, Cognition and Machine Learning Applications10.1007/978-981-33-6691-6_43(391-402)Online publication date: 31-Mar-2021
https://doi.org/10.1007/978-981-33-6691-6_43
Jamil KRastogi DJohri PSabarwal M(2021)Brief Analysis on Human Activity RecognitionCybernetics, Cognition and Machine Learning Applications10.1007/978-981-33-6691-6_2(9-20)Online publication date: 31-Mar-2021
https://doi.org/10.1007/978-981-33-6691-6_2
Sathio ABrohi A(2021)The Imperative Role of Pervasive Data in HealthcarePervasive Healthcare10.1007/978-3-030-77746-3_2(17-29)Online publication date: 16-Nov-2021
https://doi.org/10.1007/978-3-030-77746-3_2
Chen YMonroy-Hernández AWehrman IOney SLasecki WVaish R(2020)Sifter: A Hybrid Workflow for Theme-based Video Curation at ScaleProceedings of the 2020 ACM International Conference on Interactive Media Experiences10.1145/3391614.3393657(65-73)Online publication date: 17-Jun-2020
https://dl.acm.org/doi/10.1145/3391614.3393657
Wu JHarrison CBigham JLaput GBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Automated Class Discovery and One-Shot Interactions for Acoustic Activity RecognitionProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376875(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376875
Woodward KKanjo EOikonomou AChamberlain A(2020)LabelSens: enabling real-time sensor data labelling at the point of collection using an artificial intelligence-based approachPersonal and Ubiquitous Computing10.1007/s00779-020-01427-xOnline publication date: 27-Jun-2020
https://doi.org/10.1007/s00779-020-01427-x
Cruz-Sandoval DBeltran-Marquez JGarcia-Constantino MGonzalez-Jasso LFavela JLopez-Nava ICleland IEnnis AHernandez-Cruz NRafferty JSynnott JNugent C(2019)Semi-Automated Data Labeling for Activity Recognition in Pervasive HealthcareSensors10.3390/s1914303519:14(3035)Online publication date: 10-Jul-2019
https://doi.org/10.3390/s19143035
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten