Alina Roitberg
ABSTRACT
We present an approach for monitoring and interpreting human activities based on a novel multimodal vision-based interface, aiming at improving the efficiency of human-robot interaction (HRI) in industrial environments. Multi-modality is an important concept in this design, where we combine inputs from several state-of-the-art sensors to provide a variety of information, e.g. skeleton and fingertip poses. Based on typical industrial workflows, we derived multiple levels of human activity labels, including large-scale activities (e.g. assembly) and simpler sub-activities (e.g. hand gestures), creating a duration- and complexity-based hierarchy. We train supervised generative classifiers for each activity level and combine the output of this stage with a trained Hierarchical Hidden Markov Model (HHMM), which models not only the temporal aspects between the activities on the same level, but also the hierarchical relationships between the levels.
- J. K. Aggarwal and M. S. Ryoo. Human activity analysis: A review. ACM Computing Surveys (CSUR), 43(3):16, 2011. Google Scholar
Digital Library
- O. Boiman, E. Shechtman, and M. Irani. In defense of nearest-neighbor based image classification. In Computer Vision and Pattern Recognition(CVPR) IEEE Conference on, pages 1--8. IEEE, 2008.Google ScholarCross Ref
- S. Fine, Y. Singer, and N. Tishby. The hierarchical hidden markov model: Analysis and applications. Machine learning, 32(1):41--62, 1998. Google ScholarDigital Library
- L. Gan and F. Chen. Human action recognition using apj3d and random forests. Journal of Software, 8(9):2238--2245, 2013.Google ScholarCross Ref
- B. Gleeson, K. MacLean, A. Haddadi, E. Croft, and J. Alcazar. Gestures for industry: Intuitive human-robot communication from human observation. In Proceedings of the 8th ACM/IEEE International Conference on Human-robot Interaction, HRI, pages 349--356. IEEE Press, 2013. Google ScholarDigital Library
- V. Kellokumpu, M. Pietikäinen, and J. Heikkilä. Human activity recognition using sequences of postures. In IAPR Conference on Machine Vision Applications, pages 570--573, 2005.Google Scholar
- H. S. Koppula, R. Gupta, and A. Saxena. Learning human activities and object affordances from rgb-d videos. The International Journal of Robotics Research, 32(8):951--970, 2013. Google ScholarDigital Library
- C. Lenz, A. Sotzek, T. Röder, H. Radrich, A. Knoll, M. Huber, and S. Glasauer. Human workflow analysis using 3D occupancy grid hand tracking in a human-robot collaboration scenario. In IROS, pages 3375--3380. IEEE, 2011.Google ScholarCross Ref
- B. Liang and L. Zheng. Multi-modal gesture recognition using skeletal joints and motion trail model. In Computer Vision-ECCV Workshops, pages 623--638. Springer, 2014.Google Scholar
- N. T. Nguyen, D. Q. Phung, S. Venkatesh, and H. Bui. Learning and detecting activities from movement trajectories using the hierarchical hidden markov model. In Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on, volume 2, pages 955--960. IEEE, 2005. Google ScholarDigital Library
- F. Offi, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy. Sequence of the most informative joints (smij): A new representation for human skeletal action recognition. Journal of Visual Communication and Image Representation, 25(1):24--38, 2014. Google ScholarDigital Library
- G. T. Papadopoulos, A. Axenopoulos, and P. Daras. Real-time skeleton-tracking-based human action recognition using kinect data. In MultiMedia Modeling, pages 473--483. Springer, 2014. Google ScholarDigital Library
- R. Poppe. A survey on vision-based human action recognition. Image and vision computing, 28(6):976--990, 2010. Google ScholarDigital Library
- L. R. Rabiner and B. H. Juang. An introduction to hidden markov models. ASSP Magazine, pages 4--16, Jan. 1986.Google ScholarCross Ref
- A. Roitberg, A. Perzylo, N. Somani, M. Giuliani, M. Rickert, and A. Knoll. Human activity recognition in the context of industrial human-robot interaction. In Asia-Pacific Signal and Information Processing Association, 2014 Annual Summit and Conference (APSIPA), pages 1--10. IEEE, 2014.Google ScholarCross Ref
- J. Sung, C. Ponce, B. Selman, and A. Saxena. Unstructured human activity detection from RGBD images. In IEEE International Conference on Robotics and Automation (ICRA), 2012.Google Scholar
- P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea. Machine recognition of human activities: A survey. Circuits and Systems for Video Technology, IEEE Transactions on, 18(11):1473--1488, 2008. Google ScholarDigital Library
- J. Yamato, J. Ohya, and K. Ishii. Recognizing human action in time-sequential images using hidden markov model. In CVPR, pages 379--385, 1992.Google ScholarCross Ref
- Y. Zhu, W. Chen, and G. Guo. Fusing spatiotemporal features and joints for 3d action recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE Conference on, pages 486--491. IEEE, 2013. Google ScholarDigital Library
Index Terms
- Multimodal Human Activity Recognition for Industrial Manufacturing Processes in Robotic Workcells
Recommendations
A spanning tree-based human activity prediction system using life logs from depth silhouette-based human activity recognition
CAIP'11: Proceedings of the 14th international conference on Computer analysis of images and patterns - Volume Part IIn this work, we propose a Human Activity Prediction (HAP) system using activity sequence spanning trees constructed from a life-log created by a video sensor-based daily Human Activity Recognition (HAR) system using time-sequential Independent ...
Human Activity Behavioural Pattern Recognition in Smart Home with Long-Hour Data Collection
AbstractThe research on human activity recognition has provided novel solutions to many applications like health care, sports, and user profiling. Considering the complex nature of human activities, it is still challenging even after effective and ...
Robot semantic mapping through human activity recognition
Semantic information can help robots understand unknown environments better. In order to obtain semantic information efficiently and link it to a metric map, we present a new robot semantic mapping approach through human activity recognition in a human-...
Comments