Skip to main content
SpringerLink
Log in

Automatic robot Manoeuvres detection using computer vision and deep learning techniques: a perspective of internet of robotics things (IoRT)

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

To minimize any impediments in real-time Internet of Things (IoT)-enabled robotics applications, this study demonstrated how to build and deploy a revolutionary framework using computer vision and deep learning. In contrast to robotic path planning algorithms based on geolocation. We focus on sensor-captured streams/images and geographical information to enable the Internet of Robotic Things (IoRT) to evolve. The application will collect real-time data from moving robotics at various situations and intervals and use it for research projects. The data collected in videos/image forms are delivered in the robotics application using visual sensor nodes. In this study, anticipating moving robot moves automatically early on can aid in issuing commands to monitor and regulate robots’ future activities before they occur. To do so, we propose the framework using efficient computer vision techniques and a deep learning classifier. The computer vision methods are designed for frame quality improvement, object segmentation, and feature estimation. The Long-Term Short Memory (LSTM) classifier detects robot motions automatically from initial sequential features. We mainly designed the proposed model using an LSTM classifier to perform the earlier prediction from the initial sequential features of partial video frames and to overcome the problems of exploding and vanishing gradients. LSTM helps to reduce the prediction duration with higher accuracy. It also enables the central system of a certain robotic application to prevent collisions caused by impediments in the interior or outdoor situation. The simulation results utilizing publicly available research datasets demonstrate the proposed model’s efficiency and robustness compared to state-of-the-art approaches. The overall accuracy of the proposed model has improved approximately by 5% and reduced computational complexity by 84% approximately.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Alcácer, V, Cruz-Machado, V (2019) Scanning the industry 4.0: a literature review on Technologies for Manufacturing Systems. Eng Sci Technol Int J

  2. Alhayani B, Abbas ST, Mohammed HJ, Mahajan HB (2021) Intelligent secured two-way image transmission using Corvus Corone module over WSN. Wirel Pers Commun 120:665–700. https://doi.org/10.1007/s11277-021-08484-2

    Article  Google Scholar 

  3. Bar-Hillel A, Lerner R, Levi D, Raz G (2011) Recent progress in road and lane detection: a survey. Mach Vis Appl 25:727–745

    Article  Google Scholar 

  4. Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A, Di W (2017) Recommendation system for immunization coverage and monitoring. Human Vaccines Immunotherapeutics 14(1):165–171. https://doi.org/10.1080/21645515.2017.1379639

    Article  Google Scholar 

  5. Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2018) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13:1–23. https://doi.org/10.1080/17517575.2018.1557256

    Article  Google Scholar 

  6. Bhatti, U, Yu, Z, Yuan, L, Nawaz, SA, Bhatti, M, Mehmood, A, Ain, QUl, Zeehsan, Z, Wen, L. (2020) Geometric algebra applications in geospatial artificial intelligence and remote sensing image processing. IEEE Access. PP. 1–1. https://doi.org/10.1109/ACCESS.2020.3018544.

  7. Bhatti, U, Yu, Z, Chanussot, J, Zeeshan Z, Yuan, L, Luo, W, Nawaz, SA, Bhatti, M, Ain, QUl, Mehmood, A. (2021) Local Similarity-Based Spatial-Spectral Fusion Hyperspectral Image Classification With Deep CNN and Gabor Filtering. IEEE Trans Geosci Remote Sensing. PP. 1–15. https://doi.org/10.1109/TGRS.2021.3090410.

  8. Bhatti UA, Yan Y, Zhou M, Ali S, Hussain A, Qingsong H, Yu Z, Yuan L (2021) Time series analysis and forecasting of air pollution particulate matter (PM2.5): An SARIMA and factor analysis approach. IEEE Access 9:41019–41031. https://doi.org/10.1109/access.2021.3060744

    Article  Google Scholar 

  9. Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z, Yuan L (2022) Assessing the change of ambient air quality patterns in Jiangsu Province of China pre-to post-COVID-19. Chemosphere 288(Pt 2):132569. https://doi.org/10.1016/j.chemosphere.2021.132569

    Article  Google Scholar 

  10. Calli B, Singh A, Bruce J, Walsman A, Konolige K, Srinivasa S, Abbeel P, Dollar AM (2017) Yale-CMU-Berkeley dataset for robotic manipulation research. Int J Robot Res 36(3):261–268. https://doi.org/10.1177/0278364917700714

    Article  Google Scholar 

  11. Caruso, D, Engel, J, Cremers, D (2015) Large-scale direct SLAM for omnidirectional cameras. 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), 141–148

  12. Chang, C, Siagian, C, Itti, L (2010) Mobile robot vision navigation & localization using gist and saliency. 2010 IEEE/RSJ international conference on intelligent robots and systems, 4147–4154

  13. Chauhan A, Jakhar SK, Chauhan C (2020) The interplay of circular economy with industry 4.0 enabled smart city drivers of healthcare waste disposal. J Clean Prod 279:123854–123854

    Article  Google Scholar 

  14. Chen H, Sun D (2012) Moving groups of microparticles into Array with a robot–tweezers manipulation system. IEEE Trans Robot 28:1069–1080

    Article  Google Scholar 

  15. Chen M, Sinha A, Hu K, Shah MI (2021) Impact of technological innovation on energy efficiency in industry 4.0 era: moderation of shadow economy in sustainable development. Technol Forecast Soc Change 164:120521

    Article  Google Scholar 

  16. Cummins M, Newman P (2008) FAB-MAP: probabilistic localization and mapping in the space of appearance. Int J Robot Res 27(6):647–665. https://doi.org/10.1177/0278364908090961

    Article  Google Scholar 

  17. Duan, Y, Huang, X, Yu, X (2016) Multi-robot dynamic virtual potential point hunting strategy based on FIS. 2016 IEEE Chinese guidance, Navigation and Control Conference (CGNCC), 332–335

  18. Fang Y, Liu J, Li J, Cheng J, Hu J, Yi D, Xiao X, Bhatti UA (2022) Robust zero-watermarking algorithm for medical images based on SIFT and Bandelet-DCT. Multimed Tools Appl 81:16863–16879. https://doi.org/10.1007/s11042-022-12592-x

    Article  Google Scholar 

  19. Gil-Martin M, San-Segundo R, Fernandez-Martinez F, de Cordoba R (2020) Human activity recognition adapted to the type of movement. Comput Electr Eng 88:106822. https://doi.org/10.1016/j.compeleceng.2020.106822

    Article  Google Scholar 

  20. Gualtieri L, Rauch E, Vidoni R (2021) Emerging research fields in safety and ergonomics in industrial collaborative robotics: A systematic literature review. Rob Comput Integr Manuf 67:Article 101998

    Article  Google Scholar 

  21. Ha J, Shin J, Park H, Paik J (2021) Action recognition network using stacked short-term deep features and bidirectional moving average. Appl Sci 11(12):5563. https://doi.org/10.3390/app11125563

    Article  Google Scholar 

  22. Hari Pavan A, Anvitha P, Prem Sai A, Sunil I, Maruthi Y, Radhesyam V (2022) Human action recognition in videos using deep neural network. In: Chowdary PSR, Anguera J, Satapathy SC, Bhateja V (eds) Evolution in signal processing and telecommunication networks. Lecture notes in electrical engineering, vol 839. Springer, Singapore. https://doi.org/10.1007/978-981-16-8554-5_31

    Chapter  Google Scholar 

  23. Hasnain A, Sheng Y, Hashmi et al (2022) Time Series Analysis and Forecasting of Air Pollutants Based on Prophet Forecasting Model in Jiangsu Province, China Citation. Front Environ Sci 10:1. https://doi.org/10.3389/fenvs.2022.945628

    Article  Google Scholar 

  24. Hu, S, Cao, C, Pan, J (2017) Deep-learned pedestrian avoidance policy for robot navigation. 2017 IEEE international conference on robotics and biomimetics (ROBIO), 338–343

  25. Hussain A, Hussain T, Ullah W, Baik SW (2022) Vision transformer and deep sequence learning for human activity recognition in surveillance videos. Comput Intell Neurosci 2022:3454167. https://doi.org/10.1155/2022/3454167

    Article  Google Scholar 

  26. Ijaz MF, Attique M, Son Y (2020) Data-driven cervical Cancer prediction model with outlier detection and over-sampling methods. Sensors 20(10):2809. https://doi.org/10.3390/s20102809

    Article  Google Scholar 

  27. Jain, A, Koppula, HS, Soh, S, Raghavan, B, Singh, A, Saxena, A (2016) Brain4Cars: Car that knows before you do via sensory-fusion deep learning architecture. ArXiv, abs/1601.00740

  28. Jama, M; Schinstock, D (2011) Parallel Tracking and Mapping for Controlling VTOL Airframe. J Control Sci Eng, 2011(), 1–10. https://doi.org/10.1155/2011/413074

  29. Khan IU, Afzal S, Lee JW (2022) Human activity recognition via hybrid deep learning based model. Sensors (Basel, Switzerland) 22(1):323. https://doi.org/10.3390/s22010323

    Article  Google Scholar 

  30. Kim J-H, Won CS (2020) Action recognition in videos using pre-trained 2D convolutional neural networks. IEEE Access 8:60179–60188. https://doi.org/10.1109/access.2020.2983427

    Article  Google Scholar 

  31. Kipper LM, Iepsen S, Dal Forno AJ, Frozza R, Furstenau L, Agnes J, Cossul D (2021) Scientific mapping to identify competencies required through industry 4.0 Technol. Soc 64:Article 101454

    Google Scholar 

  32. Kose, N, Kopuklu, O, Unnervik, A, Rigoll, G (2019) Real-time driver state monitoring using a CNN based Spatio-temporal approach*. 2019 IEEE intelligent transportation systems conference (ITSC). https://doi.org/10.1109/itsc.2019.8917460.

  33. Kruijff, GM, Colas, F, Svoboda, T, Diggelen, JV, Balmer, P, Pirri, F, Worst, R (2012) Designing intelligent robots for human-robot teaming in Urban search and rescue. AAAI Spring Symposium: Designing Intelligent Robots

  34. Kumar P, Singh RK, Kumar V (2021) Managing supply chains for sustainable operations in the era of industry 4.0 and circular economy: analysis of barriers Resour. Conserv Recycl 164:Article 105215

    Article  Google Scholar 

  35. Lee, S-H, Lee, B. (2015). Kalman Consensus Based Multi-Robot SLAM with a Rao-Blackwellized Particle Filter. J Autom Control Eng. 368–372. https://doi.org/10.12720/joace.3.5.368-372.

  36. Liang X, Wang H, Chen W, Guo D, Liu T (2015) Adaptive image-based trajectory tracking control of wheeled Mobile robots with an uncalibrated fixed camera. IEEE Trans Control Syst Technol 23:2266–2282

    Article  Google Scholar 

  37. Liu M (2016) Robotic online path planning on point cloud. IEEE Trans Cybern 46:1217–1228

    Article  Google Scholar 

  38. Liu, M, Colas, F, Siegwart, RY (2011) Regional topological segmentation based on mutual information graphs. 2011 IEEE international conference on robotics and automation, 3269–3274

  39. Liu M, Pradalier C, Siegwart R (2013) Visual homing from scale with an uncelebrated omnidirectional camera. IEEE Trans Robot 29(6):1353–1365

    Article  Google Scholar 

  40. Lu K, Li J, An X, He H (2015) Vision Sensor-Based Road Detection for Field Robot Navigation. Sensors (Basel, Switzerland) 15:29594–29617

    Article  Google Scholar 

  41. Ma C, Hu X, Xiao J, Du H, Zhang G (2020) Improved ORB algorithm using three-patch method and local gray difference. Sensors 20(4):975. https://doi.org/10.3390/s20040975

    Article  Google Scholar 

  42. Mahajan, HB, Badarla, A (2018) Application of internet of things for smart precision farming: solutions and challenges. Int J Adv Sci Technol, Vol Dec 2018, PP. 37–45

  43. Mahajan HB, Badarla A (2019) Experimental analysis of recent clustering algorithms for wireless sensor network: application of IoT based smart precision farming. J Adv Res Dyn Control Syst 11(9):116–125. https://doi.org/10.5373/JARDCS/V11I9/20193162

    Article  Google Scholar 

  44. Mahajan HB, Badarla A (2020) Detecting HTTP vulnerabilities in IoT-based precision farming connected with cloud environment using artificial intelligence. Int J Adv Sci Technol 29(3):214–226

    Google Scholar 

  45. Mahajan HB, Badarla A (2021) Cross-layer protocol for WSN-assisted IoT smart farming applications using nature inspired algorithm. Wirel Pers Commun 121:3125–3149. https://doi.org/10.1007/s11277-021-08866-6

    Article  Google Scholar 

  46. Mahajan HB, Badarla A, Junnarkar AA (2021) CL-IoT: cross-layer internet of things protocol for intelligent manufacturing of smart farming. J Ambient Intell Humaniz Comput 12:7777–7791. https://doi.org/10.1007/s12652-020-02502-0

    Article  Google Scholar 

  47. Mahajan, HB, Rashid, AS, Junnarkar, AA et al. (2022) Integration of Healthcare 4.0 and blockchain into secure cloud-based electronic health records systems. Appl Nanosci, https://doi.org/10.1007/s13204-021-02164-0.

  48. Mandal M, Singh PK, Ijaz MF, Shafi J, Sarkar R (2021) A tri-stage wrapper-filter feature selection framework for disease classification. Sensors 21(16):5571. https://doi.org/10.3390/s21165571

    Article  Google Scholar 

  49. Mikhail, A, Kamil, IA, Mahajan, H (2017) Increasing SCADA System Availability by Fault Tolerance Techniques. 2017 International conference on computing, communication, Control and Automation (ICCUBEA) https://doi.org/10.1109/iccubea.2017.8463911

  50. Mikhail, A, Kareem, HH, Mahajan, H (2017) Fault Tolerance to Balance for Messaging Layers in Communication Society. 2017 International conference on computing, communication, Control and Automation (ICCUBEA) https://doi.org/10.1109/iccubea.2017.8463871

  51. Momen, S, Lima, TI, Siddika, R (2016) Group decision by house-hunting agents in multi-robot systems. 2016 2nd international symposium on agent, multi-agent systems and robotics (ISAMSR), 73-78

  52. Newcombe, RA, Lovegrove, SJ, Davison, AJ (2011) 2011 International Conference on Computer Vision - DTAM: Dense tracking and mapping in real-time. (0), 2320–2327. https://doi.org/10.1109/iccv.2011.6126513.

  53. Newcombe, RA, Izadi, S, Hilliges, O, Molyneaux, D, Kim, D, Davison, AJ, Kohli, P, Shotton, J, Hodges, S, Fitzgibbon, AW (2011) KinectFusion: real-time dense surface mapping and tracking. 2011 10th IEEE international symposium on mixed and augmented reality, 127-136

  54. Ran, L, Zhang, Y, Zhang, Q, Yang, T (2017) Convolutional neural network-based robot navigation using uncalibrated spherical images †. Sensors (Basel, Switzerland), 17

  55. Ruiz-Sarmiento JR, Galindo C, Gonzalez-Jimenez J (2017) Robot@home, a robotic dataset for semantic mapping of home environments. Int J Robot Res 36(2):131–141. https://doi.org/10.1177/0278364917695640

    Article  Google Scholar 

  56. Serpush F, Rezaei M (2021) Complex human action recognition using a hierarchical feature reduction and deep learning-based method. SN Comput Sci 2:94. https://doi.org/10.1007/s42979-021-00484-0

    Article  Google Scholar 

  57. Shen, H, Li, N, Rojas, S, Zhang, L (2016) Multi-robot cooperative hunting. 2016 international conference on collaboration technologies and systems (CTS), 349–353

  58. Shi Q, Zhang HB, Ren HT, du JX, Lei Q (2020) Consistent constraint-based video-level learning for action recognition. J Image Video Proc 2020(35). https://doi.org/10.1186/s13640-020-00519-1

  59. Srinivasu PN, SivaSai JG, Ijaz MF, Bhoi AK, Kim W, Kang JJ (2021) Classification of skin disease using deep learning neural networks with MobileNet V2 and LSTM. Sensors 21(8):2852. https://doi.org/10.3390/s21082852

    Article  Google Scholar 

  60. Srinivasu PN, Ahmed S, Alhumam A, Bhoi AK, Ijaz MF (2021) An AW-HARIS Based Automated Segmentation of Human Liver Using CT Images. Comput Mater Continua 69:3303–3319. https://doi.org/10.32604/cmc.2021.018472

    Article  Google Scholar 

  61. Tai, L, Liu, M (2016) Deep-learning in Mobile robotics - from perception to control systems: a survey on why and why not. ArXiv, abs/1612.07139

  62. Tai, L, Li, S, Liu, M (2017) Autonomous exploration of mobile robots through deep neural networks. Int J Adv Robot Syst, 14

  63. Tai, L, Li, S, Liu, M (2017) Autonomous exploration of mobile robots through deep neural networks. Int J Adv Robot Syst, 14

  64. Uke, N, Pise, P, Mahajan, HB, et.al. (2021) Healthcare 4.0 Enabled Lightweight Security Provisions for Medical Data Processing. Turkish Journal of Computer and Mathematics (2021), Vol. 12, No. 11. https://doi.org/10.17762/turcomat.v12i11.5858.

  65. Ullah A, Muhammad K, Ding W, Palade V, Haq IU, Baik SW (2021) Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications. Appl Soft Comput 103:107102. https://doi.org/10.1016/j.asoc.2021.107102

    Article  Google Scholar 

  66. Urban, S, Hinz, S (2016) MultiCol-SLAM - A Modular Real-Time Multi-Camera SLAM System. ArXiv, abs/1610.07336

  67. Vulli A, Srinivasu PN, Sashank M, Shafi J, Choi J, Ijaz MF (2022) Fine-tuned DenseNet-169 for breast Cancer metastasis prediction using FastAI and 1-cycle policy. Sensors (Basel, Switzerland) 22(8):2988. https://doi.org/10.3390/s22082988

    Article  Google Scholar 

  68. Wang H, Guo D, Liang X, Chen W, Hu G, Leang KK (2017) Adaptive vision-based leader–follower formation control of Mobile robots. IEEE Trans Ind Electron 64:2893–2902

    Article  Google Scholar 

  69. Werthen-Brabants L, Bhavanasi G, Couckuyt I, Dhaene T, Deschrijver D (2022) Split BiRNN for real-time activity recognition using radar and deep learning. Sci Rep 12:7436. https://doi.org/10.1038/s41598-022-08240-x

    Article  Google Scholar 

  70. Xia, C (2015) Intelligent Mobile robot learning in autonomous navigation

  71. Yoneda K, Kato C, Sugawara T (2013) Autonomous learning of target decision strategies without Communications for Continuous Coordinated Cleaning Tasks. 2013 IEEE/WIC/ACM Int Joint Conf Web Intell (WI) Intell Agent Technol (IAT) 2:216–223

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Analysis and Data Scientist, Godwit Technologies, Pune, India

    Hemant B. Mahajan

  2. Trinity Academy of Engineering, Pune, India

    Nilesh Uke

  3. Indira College of Engineering and Management, Pune, India

    Priya Pise

  4. SVKM’s Institute of Technology, Mumbai Agra Highway, Behind Gurudwara, Dhule, India

    Makarand Shahade

  5. PES Modern College of Engineering, Pune, India

    Vandana G. Dixit, Swapna Bhavsar & Sarita D. Deshpande

Authors
  1. Hemant B. Mahajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nilesh Uke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Priya Pise
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Makarand Shahade
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vandana G. Dixit
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Swapna Bhavsar
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sarita D. Deshpande
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hemant B. Mahajan.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Conflict of interest

All authors declares that they has no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahajan, H.B., Uke, N., Pise, P. et al. Automatic robot Manoeuvres detection using computer vision and deep learning techniques: a perspective of internet of robotics things (IoRT). Multimed Tools Appl 82, 23251–23276 (2023). https://doi.org/10.1007/s11042-022-14253-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14253-5

Keywords

Search

Navigation