Skip to main content
SpringerLink
Log in

Multimodal Prediction-Based Robot Abnormal Movement Identification Under Variable Time-length Experiences

  • Short paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Robots will eventually make part of our daily lives, helping us at home, taking care of the elderly, and collaborating at work. In such Human-Robot Collaboration (HRC) scenarios, achieving abnormal movement identification can effectively deal with unexpected anomalies such as human collisions, external perturbations, and unexpected changes in the environment. To this end, Long-short Term Memory (LSTMs) based prediction methods are widely proposed for abnormal identification, which typically has an implicit requirement of fixed-length input signals such that the identification thresholds are calculated from the prediction-error sequences with the same length. However, in robotics, this is rarely the case, generalization in HRC is a desirable characteristic that indicates the recorded executions would have different lengths for a specific movement. To address this problem, we first extend the concept of stacked LSTMs to predict anomalies by admitting the input multivariate time series of varying lengths. Subsequently, prediction errors with different lengths are modeled using a probabilistic model for tackling the temporal uncertainty. Consequently, dynamic threshold representation is learned from the trained probabilistic model for abnormal movement identification. A self-designed robot manipulation task consisting of six individual movements is used to evaluate the proposed approach and compared it to baselines. Experimental results indicate that the proposed method with an average anomaly identification accuracy of 94% outperforms the baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Park, D., Hoshi, Y., Kemp, C.C.: A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robot. Autom. Lett. 3(3), 1544–1551 (2018)

    Article  Google Scholar 

  2. Heo, Y.J., Kim, D., Lee, W., Kim, H., Park, J., Chung, W.K.: Collision detection for industrial collaborative robots: a deep learning approach. IEEE Robot. Autom. Lett. 4(2), 740–746 (2019)

    Article  Google Scholar 

  3. Husein, A.M., Arsyal, M., Sinaga, S., Syahputa, H.: Generative adversarial networks time series models to forecast medicine daily sales in hospital. SinkrOn 3(2), 112–118 (2019)

    Article  Google Scholar 

  4. Cao, P., Gan, Y., Dai, X.: Model-based sensorless robot collision detection under model uncertainties with a fast dynamics identification. Int. J. Adv. Robot. Syst. 16(3), 1729881419853713 (2019)

    Article  Google Scholar 

  5. Birjandi, S.A.B., Kühn, J., Haddadin, S.: Observer-extended direct method for collision monitoring in robot manipulators using proprioception and imu sensing. IEEE Robot. Autom. Lett. 5(2), 954–961 (2020)

    Article  Google Scholar 

  6. Kaur, U., Khan, U., Chauhan, N.R., Mukherjee, S.: Collision detection and inverse dynamics control of kuka lbr iiwa robot. Int. J. Mechatron. Autom. 8(1), 9–21 (2021)

    Article  Google Scholar 

  7. Zhang, Y., Zhu, W., Rosendo, A.: Qr code-based self-calibration for a fault-tolerant industrial robot arm. IEEE Access 7, 73349–73356 (2019)

    Article  Google Scholar 

  8. Haddadin, S., De Luca, A., Albu-Schäffer, A.: Robot collisions: a survey on detection, isolation, and identification. IEEE Trans. Robot. 33(6), 1292–1312 (2017)

    Article  Google Scholar 

  9. Zanchettin, A.M., Ceriani, N.M., Rocco, P., Ding, H., Matthias, B.: Safety in human-robot collaborative manufacturing environments: Metrics and control. IEEE Trans. Autom. Sci. Eng. 13(2), 882–893 (2015)

    Article  Google Scholar 

  10. Zacharaki, A., Kostavelis, I., Gasteratos, A., Dokas, I.: Safety bounds in human robot interaction: A survey. Safe. Sci. 127, 104667 (2020)

    Article  Google Scholar 

  11. Buda, T.S., Caglayan, B., Assem, H.: Deepad: A generic framework based on deep learning for time series anomaly detection. In: Pacific-Asia conference on knowledge discovery and data mining, pp 577–588. Springer (2018)

  12. Ijspeert, A.J., Nakanishi, J., Hoffmann, H., Pastor, P., Schaal, S.: Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25(2), 328–373 (2013)

    Article  MathSciNet  Google Scholar 

  13. Calinon, S.: Learning from demonstration (programming by demonstration), Encyclopedia of Robotics (2009)

  14. Huang, Y., Rozo, L., Silvério, J., Caldwell, D.G.: Kernelized movement primitives. Int. J. Robot. Res. 38(7), 833–852 (2019)

    Article  Google Scholar 

  15. Munir, M., Siddiqui, S.A., Dengel, A., Ahmed, S.: Deepant: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access 7, 1991–2005 (2018)

    Article  Google Scholar 

  16. Di Lello, E., De Laet, T., Bruyninckx, H.: Hierarchical Dirichlet Process Hidden Markov Models for Abnormality Detection in Robotic Assembly. In: Workshop on Bayesian Nonparametric Models (BNPM) for Reliable Planning and Decision-Making under Uncertainty, NIPS, vol. 2012 (2012)

  17. Di Lello, E., Klotzbücher, M., De Laet, T., Bruyninckx, H.: Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In: 2013 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 5827–5833 (2013)

  18. Nguyen, H., Tran, K.P., Thomassey, S., Hamad, M.: Forecasting and anomaly detection approaches using lstm and lstm autoencoder techniques with the applications in supply chain management. Int. J. Inform. Manag. 57, 102282 (2021)

    Article  Google Scholar 

  19. Taylor, A., Leblanc, S., Japkowicz, N.. In: 2016 IEEE international conference on data science and advanced analytics (DSAA), pp 130–139. IEEE (2016)

  20. Ergen, T., Kozat, S.S.: Unsupervised anomaly detection with lstm neural networks. IEEE Trans. Neural Netw. Learn. Syst. 31(8), 3127–3141 (2019)

    Article  MathSciNet  Google Scholar 

  21. Shewalkar, A., Nyavanandi, D., Ludwig, S.A.: Performance evaluation of deep neural networks applied to speech recognition: rnn, lstm and gru. J. Artif. Intell. Soft Comput. Res. 9(4), 235–245 (2019)

    Article  Google Scholar 

  22. Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Fifteenth annual conference of the international speech communication association (2014)

  23. Wang, R., Nie, K., Wang, T., Yang, Y., Long, B.: Deep learning for anomaly detection. In: Proceedings of the 13th international conference on web search and data mining, pp 894–896 (2020)

  24. An, J., Cho, S.: Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE 2(1), 1–18 (2015)

    Google Scholar 

  25. Li, D., Chen, D., Jin, B., Shi, L., Goh, J., Ng, S.-K.: Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In: International conference on artificial neural networks, pp 703–716. Springer (2019)

  26. Shao, H., Soong, B.-H.: Traffic Flow Prediction with Long Short-Term Memory Networks (Lstms). In: 2016 IEEE Region 10 Conference (TENCON), pp 2986–2989. IEEE (2016)

  27. Gangopadhyay, T., Y. Tan, S., Huang, G., Sarkar, S.: Temporal attention and stacked lstms for multivariate time series prediction (2018)

  28. Hundman, K., Constantinou, V., Laporte, C., Colwell, I., Soderstrom, T.: Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 387–395. ACM (2018)

  29. Bontemps, L., McDermott, J., Le-Khac, N.-A., et al.: Collective anomaly detection based on long short-term memory recurrent neural networks. In: International conference on future data and security engineering, pp 141–152. Springer (2016)

  30. Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings, Presses universitaires de Louvain, p 89 (2015)

  31. Chauhan, S., Vig, L. IEEE, DSAA (2015)

  32. Wu, H., Guan, Y., Rojas, J.: Analysis of multimodal bayesian nonparametric autoregressive hidden markov models for process monitoring in robotic contact tasks. Int. J. Adv. Robot. Syst. 16(2), 1729881419834840 (2019)

    Article  Google Scholar 

  33. Wu, H., Guan, Y., Rojas, J.: A latent state-based multimodal execution monitor with anomaly detection and classification for robot introspection. Appl. Sci. 9(6), 1072 (2019)

    Article  Google Scholar 

  34. Ralaivola, L., D’Alché-Buc, F.: Time series filtering, smoothing and learning using the kernel kalman filter. In: Proceedings. 2005 IEEE international joint conference on neural networks, 2005, vol. 3, pp 1449–1454. IEEE (2005)

  35. Maeda, G., Ewerton, M., Neumann, G., Lioutikov, R., Peters, J.: Phase estimation for fast action recognition and trajectory generation in human–robot collaboration. Int. J. Robot. Res. 36(13-14), 1579–1594 (2017)

    Article  Google Scholar 

  36. Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. J. Machine Learn. Res. 9(Nov), 2579–2605 (2008)

    MATH  Google Scholar 

Download references

Acknowledgments

We thank Shuangda Duan and Dong Liu for their assistance on system development, and Prof. Juan Rojas for his revision on English writing throughout this paper.

Funding

This work is supported by Guangdong Province Key Areas R&D Program (Grant No. 2019B090919002), Basic and Applied Basic Research Project of Guangzhou (Grant No. 202002030237), GDAS’ Project of Thousand doctors(post-doctors) Introduction (2020GDASYL-20200103128), Foshan Key Technology Research Project(Grant No. 1920001001148), Guangdong Province International Cooperation Project of Science and Technology (Grant No. 2019A050510040), National Science Foundation of China (Grant No. 61950410758).

Author information

Authors and Affiliations

  1. Institute of Intelligent Manufacturing, Guangdong Academy of Sciences, and Guangdong Key Laboratory of Modern Control Technology, Guangzhou, China

    Hongmin Wu, Wu Yan, Zhihao Xu, Taobo Cheng & Xuefeng Zhou

  2. Foshan Tri-Co Intelligent Robot Technology Co., Ltd, Foshan, China

    Zhihao Xu

  3. School of Engineering, Swansea University, Swansea, United Kingdom

    Shuai Li

Authors
  1. Hongmin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wu Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhihao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Taobo Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xuefeng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

This paper has six authors. Authors made most of the contributions on conceptualization, development of theory, validation, verification of the analytical methods, discussion of results, and contributed to the final manuscript. Individual contributions follow: the draft writing, methodology, and development of theory finished by Hongmin Wu; the code development, experimentation, and verification of the analytical methods did by Wu Yan and Zhihao Xu; the project administration and discussion of results did by Shuai Li and Taobo Cheng; Xuefeng Zhou provided the formal analysis and investigation of this study.

Corresponding author

Correspondence to Xuefeng Zhou.

Ethics declarations

Ethics approval

This study did not require ethics approval as the information consists of naturalistic observations regarding choice. Choices remained anonymous. There is not any identifier information that would allow attribution of private information to an individual.

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Consent to participate

Informed consent was obtained from all participants in the study.

Consent to publish

Participant consented to the submission of this article to the journal.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Yan, W., Xu, Z. et al. Multimodal Prediction-Based Robot Abnormal Movement Identification Under Variable Time-length Experiences. J Intell Robot Syst 104, 8 (2022). https://doi.org/10.1007/s10846-021-01496-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01496-x

Keywords

Search

Navigation