Skip to main content
Springer Nature Link
Log in

Dynamic Analysis and Modeling of DNN-Based Visual Servoing Systems

  • Conference paper
  • First Online:
Intelligent Computing (SAI 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 507))

Included in the following conference series:

  • 1235 Accesses

Abstract

The integration of deep learning and control techniques has created robotic systems capable of implementing visual servoing and navigating autonomously in unknown environments. However, analyzing the effect that timing interactions between controllers and deep neural networks have, has received little attention in the literature. In this paper we describe a novel model that includes the effects that detection loss and inference latency have on the controller of a visual servoing system. To test our model we created a target tracking system consisting of a video camera mounted on a moving platform that tracks objects using deep neural networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Finddelay: Estimate delay(s) between signals. https://se.mathworks.com/help/signal/ref/finddelay.htm. Accessed 24 Nov 2021

  2. Bojarski, M., et al.: End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016)

  3. Geiger, W., et al.: MEMS IMU for AHRS applications. In: 2008 IEEE/ION Position, Location and Navigation Symposium, pp. 225–231. IEEE (2008)

    Google Scholar 

  4. Gemerek, J., Ferrari, S., Wang, B.H., Campbell, M.E.: Video-guided camera control for target tracking and following. IFAC-PapersOnLine 51(34), 176–183 (2019). 2nd IFAC Conference on Cyber-Physical and Human Systems CPHS 2018

    Google Scholar 

  5. Giusti, A., et al.: A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robot. Autom. Lett. 1(2), 661–667 (2016)

    Article  Google Scholar 

  6. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)

    Google Scholar 

  7. Hadidi, R., Cao, J., Xie, Y., Asgari, B., Krishna, T., Kim, H.: Characterizing the deployment of deep neural networks on commercial edge devices. In: 2019 IEEE International Symposium on Workload Characterization (IISWC), pp. 35–48. IEEE (2019)

    Google Scholar 

  8. Howard, A., et al.: Searching for MobileNetV3. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  9. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017)

    Google Scholar 

  10. Hutchinson, S., Hager, G.D., Corke, P.I.: A tutorial on visual servo control. IEEE Trans. Robot. Autom. 12(5), 651–670 (1996)

    Article  Google Scholar 

  11. Jia, Z., Balasuriya, A., Challa, S.: Vision based data fusion for autonomous vehicles target tracking using interacting multiple dynamic models. Comput. Vis. Image Underst. 109(1), 1–21 (2008)

    Article  Google Scholar 

  12. Kragic, D., Christensen, H.I., et al.: Survey on visual servoing for manipulation. Computational Vision and Active Perception Laboratory, Fiskartorpsv, 15:2002 (2002)

    Google Scholar 

  13. Larsen, T.D., Andersen, N.A., Ravn, O., Poulsen, N.K.: Incorporation of time delayed measurements in a discrete-time Kalman filter. In: Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No. 98CH36171), vol. 4, pp. 3972–3977. IEEE (1998)

    Google Scholar 

  14. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  15. LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989). https://doi.org/10.1162/neco.1989.1.4.541

    Article  Google Scholar 

  16. Liu, W., et al.: SSD: single shot multibox detector. CoRR, abs/1512.02325 (2015)

    Google Scholar 

  17. Loquercio, A., Maqueda, A.I., del Blanco, C.R., Scaramuzza, D.: DroNet: learning to fly by driving. IEEE Robot. Autom. Lett. 3(2), 1088–1095 (2018)

    Article  Google Scholar 

  18. Machkour, Z., Ortiz-Arroyo, D., Durdevic, P.: Classical and deep learning based visual servoing systems: a survey on state of the art. J. Intell. Robot. Syst. 104(1), 1–27 (2022). https://doi.org/10.1007/s10846-021-01540-w

    Article  Google Scholar 

  19. Nilsson, J., et al.: Real-Time Control Systems with Delays (1998)

    Google Scholar 

  20. Petar, D., Ortiz-Arroyo, D., Li, S., Yang, Z.: Vision aided navigation of a quad-rotor for autonomous wind-farm inspection. IFAC-PapersOnLine 52, 61–66 (2019)

    Google Scholar 

  21. Ramakoti, N., Vinay, A., Jatoth, R.K.: Particle swarm optimization aided kalman filter for object tracking. In: 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies, pp. 531–533 (2009)

    Google Scholar 

  22. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  23. Sariyildiz, E., Oboe, R., Ohnishi, K.: Disturbance observer-based robust control and its applications: 35th anniversary overview. IEEE Trans. Industr. Electron. 67(3), 2042–2053 (2019)

    Article  Google Scholar 

  24. Schenato, L.: Kalman filtering for networked control systems with random delay and packet loss. In: Conference of Mathematical Theory of Networks and Systems. MTNS 2006. Citeseer (2006)

    Google Scholar 

  25. Schenato, L.: Optimal estimation in networked control systems subject to random delay and packet drop. IEEE Trans. Autom. Control 53(5), 1311–1317 (2008)

    Article  MathSciNet  Google Scholar 

  26. Schenato, L., Sinopoli, B., Franceschetti, M., Poolla, K., Sastry, S.S.: Foundations of control and estimation over lossy networks. Proc. IEEE 95(1), 163–187 (2007)

    Article  Google Scholar 

  27. Shi, L., Epstein, M., Murray, R.M.: Kalman filtering over a packet-dropping network: a probabilistic perspective. IEEE Trans. Autom. Control 55(3), 594–604 (2010)

    Article  MathSciNet  Google Scholar 

  28. Sinopoli, B., Schenato, L., Franceschetti, M., Poolla, K., Jordan, M.I., Sastry, S.S.: Kalman filtering with intermittent observations. IEEE Trans. Autom. Control 49(9), 1453–1464 (2004). https://doi.org/10.1109/TAC.2004.834121

    Article  MathSciNet  MATH  Google Scholar 

  29. Tsai, C.-Y., Dutoit, X., Song, K.-T., Van Brussel, H., Nuttin, M.: Robust face tracking control of a mobile robot using self-tuning kalman filter and echo state network. Asian J. Control 12(4), 488–509 (2010)

    MathSciNet  Google Scholar 

  30. Zhang, W., Branicky, M.S., Phillips, S.M.: Stability of networked control systems. IEEE Control. Syst. 21(1), 84–99 (2001). https://doi.org/10.1109/37.898794

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Aalborg University, Esbjerg, 6700, Denmark

    Petar Durdevic & Daniel Ortiz-Arroyo

Authors
  1. Petar Durdevic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Ortiz-Arroyo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petar Durdevic .

Editor information

Editors and Affiliations

  1. Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Durdevic, P., Ortiz-Arroyo, D. (2022). Dynamic Analysis and Modeling of DNN-Based Visual Servoing Systems. In: Arai, K. (eds) Intelligent Computing. SAI 2022. Lecture Notes in Networks and Systems, vol 507. Springer, Cham. https://doi.org/10.1007/978-3-031-10464-0_59

Download citation

Publish with us

Policies and ethics