Abstract
In this study, we propose novel gamified active learning and inaccuracy detection for crowdsourced data labeling for an activity recognition system using mobile sensing (CrowdAct). First, we exploit active learning to address the lack of accurate information. Second, we present the integration of gamification into active learning to overcome the lack of motivation and sustained engagement. Finally, we introduce an inaccuracy detection algorithm to minimize inaccurate data. To demonstrate the capability and feasibility of the proposed model in realistic settings, we developed and deployed the CrowdAct system to a crowdsourcing platform. For our experimental setup, we recruited 120 diverse workers. Additionally, we gathered 6,549 activity labels from 19 activity classes by using smartphone sensors and user engagement information. We empirically evaluated the quality of CrowdAct by comparing it with a baseline using techniques such as machine learning and descriptive and inferential statistics. Our results indicate that CrowdAct was effective in improving activity accuracy recognition, increasing worker engagement, and reducing inaccurate data in crowdsourced data labeling. Based on our findings, we highlight critical and promising future research directions regarding the design of efficient activity data collection with crowdsourcing.
- Saeed Abdullah, Nicholas D Lane, and Tanzeem Choudhury. 2012. Towards population scale activity recognition: A framework for handling data diversity. In Twenty-Sixth AAAI Conference on Artificial Intelligence.Google Scholar
- Utku Günay Acer, Marc van den Broeck, Claudio Forlivesi, Florian Heller, and Fahim Kawsar. 2019. Scaling Crowdsourcing with Mobile Workforce: A Case Study with Belgian Postal Service. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 2 (2019), 1--32.Google ScholarDigital Library
- Hande Alemdar, Tim LM van Kasteren, and Cem Ersoy. 2011. Using active learning to allow activity recognition on a large scale. In International Joint Conference on Ambient Intelligence. Springer, 105--114.Google ScholarDigital Library
- Shahriyar Amini and Yang Li. 2013. CrowdLearner: rapidly creating mobile recognizers using crowdsourcing. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 163--172.Google ScholarDigital Library
- Dana Angluin. 1988. Queries and concept learning. Machine learning 2, 4 (1988), 319--342.Google Scholar
- Les E Atlas, David A Cohn, and Richard E Ladner. 1990. Training connectionist networks with queries and selective sampling. In Advances in neural information processing systems. 566--573.Google Scholar
- Salikh Bagaveyev and Diane J Cook. 2014. Designing and evaluating active learning methods for activity recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. 469--478.Google ScholarDigital Library
- Anahid Basiri, Muki Haklay, Giles Foody, and Peter Mooney. 2019. Crowdsourced geospatial data quality: Challenges and future directions.Google Scholar
- Martin Berchtold, Matthias Budde, Dawud Gordon, Hedda R Schmidtke, and Michael Beigl. 2010. Actiserv: Activity recognition service for mobile phones. In International Symposium on Wearable Computers (ISWC) 2010. IEEE, 1--8.Google ScholarCross Ref
- Dimitar Bounov, Anthony DeRossi, Massimiliano Menarini, William G Griswold, and Sorin Lerner. 2018. Inferring loop invariants through gamification. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Hennie Brugman, Albert Russel, and Xd Nijmegen. 2004. Annotating Multi-media/Multi-modal Resources with ELAN.. In LREC.Google Scholar
- Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 1--33.Google ScholarDigital Library
- Fabrizio Carcillo, Yann-Aël Le Borgne, Olivier Caelen, Yacine Kessaci, Frédéric Oblé, and Gianluca Bontempi. 2019. Combining unsupervised and supervised learning in credit card fraud detection. Information sciences (2019).Google Scholar
- Yung-Ju Chang, Gaurav Paruthi, and Mark W Newman. 2015. A field study comparing approaches to collecting annotated activity data in real-world settings. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 671--682.Google ScholarDigital Library
- Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.Google ScholarCross Ref
- Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, and Yuan Tang. 2015. Xgboost: extreme gradient boosting. R package version 0.4-2 (2015), 1--4.Google Scholar
- Allan H Church. 1993. Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public opinion quarterly 57, 1 (1993), 62--79.Google Scholar
- Federico Cruciani, Ian Cleland, Chris Nugent, Paul McCullagh, Kåre Synnes, and Josef Hallberg. 2018. Automatic annotation for human activity recognition in free living using a smartphone. Sensors 18, 7 (2018), 2203.Google ScholarCross Ref
- Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From game design elements to gamefulness: defining" gamification". In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments. 9--15.Google ScholarDigital Library
- Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O'Hara, and Dan Dixon. 2011. Gamification. using game-design elements in non-gaming contexts. In CHI'11 extended abstracts on human factors in computing systems. 2425--2428.Google ScholarDigital Library
- Zhiguo Ding and Minrui Fei. 2013. An anomaly detection approach based on isolation forest algorithm for streaming data using sliding window. IFAC Proceedings Volumes 46, 20 (2013), 12--17.Google ScholarCross Ref
- Souad Djelassi and Isabelle Decoopman. 2013. Customers' participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management 42, 5 (2013), 683--692.Google ScholarCross Ref
- Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American statistical association 56, 293 (1961), 52--64.Google ScholarCross Ref
- Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters 27, 8 (2006), 861--874.Google Scholar
- Zachary Fitz-Walter and Dian W Tjondronegoro. 2011. Exploring the opportunities and challenges of using mobile sensing for gamification and achievements. In UbiComp 11: Proceedings of the 2011 ACM Conference on Ubiquitous Computing. ACM Press, 1--5.Google Scholar
- Jesús Fontecha, Fco Javier Navarro, Ramón Hervás, and José Bravo. 2013. Elderly frailty detection by using accelerometer-enabled smartphones and clinical information records. Personal and ubiquitous computing 17, 6 (2013), 1073--1083.Google Scholar
- Juho Hamari, Jonna Koivisto, and Harri Sarsa. 2014. Does gamification work?-a literature review of empirical studies on gamification. In 2014 47th Hawaii international conference on system sciences. Ieee, 3025--3034.Google ScholarDigital Library
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers' earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--14.Google ScholarDigital Library
- Kim Hartman. 2011. How do intrinsic and extrinsic motivation correlate with each other in open source software development?Google Scholar
- Yu-chen Ho, Ching-hu Lu, I-han Chen, Shih-shinh Huang, Ching-yao Wang, Li-chen Fu, et al. 2009. Active-learning assisted self-reconfigurable activity recognition in a dynamic environment. In Proceedings of the 2009 IEEE international conference on Robotics and Automation. IEEE Press, 1567--1572.Google ScholarDigital Library
- HM Sajjad Hossain, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. 2017. Active learning enabled activity recognition. Pervasive and Mobile Computing 38 (2017), 312--330.Google ScholarCross Ref
- Sozo Inoue, Paula Lago, Tahera Hossain, Tittaya Mairittha, and Nattaya Mairittha. 2019. Integrating Activity Recognition and Nursing Care Records: The System, Deployment, and a Verification Study. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--24.Google ScholarDigital Library
- Sozo Inoue and Xincheng Pan. 2016. Supervised and unsupervised transfer learning for activity recognition from simple in-home sensors. In Proceedings of the 13th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. 20--27.Google ScholarDigital Library
- Eiichi Iwamoto, Masaki Matsubara, Chihiro Ota, Satoshi Nakamura, Tsutomu Terada, Hiroyuki Kitagawa, and Atsuyuki Morishima. 2018. Passerby Crowdsourcing: Workers' Behavior and Data Quality Management. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1--20.Google ScholarDigital Library
- Ashish Kapoor and Eric Horvitz. 2008. Experience sampling for building predictive user models: a comparative study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 657--666.Google ScholarDigital Library
- Michael Kipp. 2001. Anvil-a generic annotation tool for multimodal dialogue. In Seventh European Conference on Speech Communication and Technology.Google ScholarCross Ref
- William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association 47, 260 (1952), 583--621.Google ScholarCross Ref
- Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. 2011. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12, 2 (2011), 74--82.Google ScholarDigital Library
- Nicholas D Lane, Ye Xu, Hong Lu, Shaohan Hu, Tanzeem Choudhury, Andrew T Campbell, and Feng Zhao. 2011. Enabling large-scale human activity inference on smartphones using community similarity networks (csn). In Proceedings of the 13th international conference on Ubiquitous computing. 355--364.Google ScholarDigital Library
- Tak Yeon Lee, Casey Dugan, Werner Geyer, Tristan Ratchford, Jamie Rasmussen, N Sadat Shami, and Stela Lupushor. 2013. Experiments on motivational feedback for crowdsourced workers. In Seventh International AAAI Conference on Weblogs and Social Media.Google Scholar
- David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94. Springer, 3--12.Google ScholarCross Ref
- Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. 2014. Facing the cold start problem in recommender systems. Expert Systems with Applications 41, 4 (2014), 2065--2073.Google ScholarDigital Library
- Brent Longstaff, Sasank Reddy, and Deborah Estrin. 2010. Improving activity classification for health applications on mobile devices using active and semi-supervised learning. In 2010 4th International Conference on Pervasive Computing Technologies for Healthcare. IEEE, 1--7.Google ScholarCross Ref
- Nattaya Mairittha and Sozo Inoue. 2018. Gamification for High-Quality Dataset in Mobile Activity Recognition. In International Conference on Mobile Computing, Applications, and Services. Springer, 216--222.Google Scholar
- Nattaya Mairittha and Sozo Inoue. 2019. Crowdsourcing System Management for Activity Data with Mobile Sensors. In 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 85--90.Google Scholar
- Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. 2018. A Mobile App for Nursing Activity Recognition. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 400--403.Google ScholarDigital Library
- Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. 2019. On-Device Deep Learning Inference for Efficient Activity Data Collection. Sensors 19, 15 (2019), 3434.Google ScholarCross Ref
- Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. 2019. Optimizing activity data collection with gamification points using uncertainty based active learning. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 761--767.Google ScholarDigital Library
- Jane McGonigal. 2011. Reality is broken: Why games make us better and how they can change the world. Penguin.Google ScholarDigital Library
- Mohamed Musthag, Andrew Raij, Deepak Ganesan, Santosh Kumar, and Saul Shiffman. 2011. Exploring micro-incentive strategies for participant compensation in high-burden studies. In Proceedings of the 13th international conference on Ubiquitous computing. 435--444.Google ScholarDigital Library
- Heather L O'Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112 (2018), 28--39.Google ScholarCross Ref
- Maria V Palacin-Silva, Antti Knutas, Maria Angela Ferrario, Jari Porras, Jouni Ikonen, and Chandara Chea. 2018. The role of gamification in participatory environmental sensing: A study in the wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision making 5, 5 (2010), 411--419.Google Scholar
- Martin Pielot, Karen Church, and Rodrigo De Oliveira. 2014. An in-situ study of mobile phone notifications. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. 233--242.Google ScholarDigital Library
- Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. [n.d.]. Improving Worker Engagement Through Conversational Microtask Crowdsourcing. ([n. d.]).Google Scholar
- Sasank Reddy, Min Mun, Jeff Burke, Deborah Estrin, Mark Hansen, and Mani Srivastava. 2010. Using mobile phones to determine transportation modes. ACM Transactions on Sensor Networks (TOSN) 6, 2 (2010), 1--27.Google ScholarDigital Library
- Attila Reiss and Didier Stricker. 2012. Creating and benchmarking a new dataset for physical activity monitoring. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments. 1--8.Google ScholarDigital Library
- Guido Sautter and Klemens Böhm. 2013. High-throughput crowdsourcing mechanisms for complex tasks. Social Network Analysis and Mining 3, 4 (2013), 873--888.Google ScholarCross Ref
- Burr Settles. 2012. Active Learning. Vol. 18. Morgan & Claypool Publishers.Google Scholar
- Claude E Shannon. 1948. A mathematical theory of communication. Bell system technical journal 27, 3 (1948), 379--423.Google Scholar
- Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3/4 (1965), 591--611.Google ScholarCross Ref
- Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision. Springer, 510--526.Google ScholarCross Ref
- Maja Stikic, Kristof Van Laerhoven, and Bernt Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In 2008 12th IEEE International Symposium on Wearable Computers. IEEE, 81--88.Google ScholarDigital Library
- Nadeem Ahmed Syed, Syed Huan, Liu Kah, and Kay Sung. 1999. Incremental learning with support vector machines. (1999).Google Scholar
- Emma L Tonkin, Alison Burrows, Przemys&lstroke;aw R Woznowski, Pawel Laskowski, Kristina Y Yordanova, Niall Twomey, and Ian J Craddock. 2018. Talk, Text, Tag? Understanding Self-Annotation of Smart Home Data from a User's Perspective. Sensors 18, 7 (2018), 2365.Google ScholarCross Ref
- Niels Van Berkel, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2017. Gamification of mobile experience sampling improves data quality and quantity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--21.Google ScholarDigital Library
- Kristof Van Laerhoven, David Kilian, and Bernt Schiele. 2008. Using rhythm awareness in long-term activity recognition. In 2008 12th IEEE International Symposium on Wearable Computers. IEEE, 63--66.Google ScholarDigital Library
- Maja Vukovic, Rajarshi Das, and Soundar Kumara. 2013. From sensing to controlling: the state of the art in ubiquitous crowdsourcing. International Journal of Communication Networks and Distributed Systems 11, 1 (2013), 11--25.Google ScholarDigital Library
- Yufeng Wang, Xueyu Jia, Qun Jin, and Jianhua Ma. 2016. QuaCentive: a quality-aware incentive mechanism in mobile crowdsourced sensing (MCS). The Journal of Supercomputing 72, 8 (2016), 2924--2941.Google ScholarDigital Library
- Yu Xiao, Pieter Simoens, Padmanabhan Pillai, Kiryong Ha, and Mahadev Satyanarayanan. 2013. Lowering the barriers to large-scale mobile crowdsensing. In Proceedings of the 14th Workshop on Mobile Computing Systems and Applications. 1--6.Google ScholarDigital Library
- Mengwei Xu, Feng Qian, Qiaozhu Mei, Kang Huang, and Xuanzhe Liu. 2018. Deeptype: On-device deep learning for input personalization service with minimal privacy concern. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1--26.Google ScholarDigital Library
- Fei Yan, Josef Kittler, David Windridge, William Christmas, Krystian Mikolajczyk, Stephen Cox, and Qiang Huang. 2014. Automatic annotation of tennis games: An integration of audio, vision, and learning. Image and Vision Computing 32, 11 (2014), 896--903.Google ScholarDigital Library
- Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. 2011. A survey of crowdsourcing systems. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing. IEEE, 766--773.Google Scholar
- Liyue Zhao, Gita Sukthankar, and Rahul Sukthankar. 2011. Robust active learning using crowdsourced annotations for activity recognition. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.Google Scholar
- Chong Zhou and Randy C Paffenroth. 2017. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 665--674.Google ScholarDigital Library
- Mengdie Zhuang and Ujwal Gadiraju. 2019. In What Mood Are You Today? An Analysis of Crowd Workers' Mood, Performance and Engagement. In Proceedings of the 10th ACM Conference on Web Science. 373--382.Google ScholarDigital Library
Index Terms
- CrowdAct: Achieving High-Quality Crowdsourced Datasets in Mobile Activity Recognition
Recommendations
Annotating smart environment sensor data for activity learning
Smart Environments: Technology to Support HealthcareThe pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing health monitoring and assistance to individuals experiencing difficulties living independently at home. In order to monitor the functional health of ...
Ubiquitous reminders to manage timetable in a smart home
iiWAS '16: Proceedings of the 18th International Conference on Information Integration and Web-based Applications and ServicesThe aging of the population will bring changes on home care for elders and gerontechnologies provide alternatives for aging in place. This article presents a concept of a smart home that reminds the resident the activities he has planned on an ...
NuActiv: recognizing unseen new activities using semantic attribute-based learning
MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and servicesWe study the problem of how to recognize a new human activity when we have never seen any training example of that activity before. Recognizing human activities is an essential element for user-centric and context-aware applications. Previous studies ...
Comments