Skip to main content
SpringerLink
Log in

Real-Time Performance Evaluation for Robotics

An Approach using the Robotstone Benchmark

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper discusses a novel approach to evaluate the real-time performance of computing platforms embedded in robotic systems. The motivation behind this work is the need for the robotic systems to meet their timing constraints, thus requiring time predictable real-time systems. We proposed a benchmark, named the Robotstone, which is an adaptation of a traditional real-time performance benchmark widely known in the real-time systems community. Thus, the presented work makes it easier for a robotic engineer to apply the benchmark in the modern robotics context by filtering out issues that do not matter on the robotics applications and adapting the benchmark’s relevant portions. The Robotstone has a set of experiments related to time-constrained application scenarios usually found in robotic systems. Each experiment defines an application-specific parameter that increases at every iteration until the system overloads. The real-time performance is then evaluated through the breakdown point, i.e., the system configuration in the application scenario when any functional or timing constraint is not met. The proposed toolset has been evaluated on two distinct platforms representing a class of embedded computing systems usually employed in robotic systems. Obtained results demonstrate the applicability of the Robotstone benchmark for a quick assessment of computing systems’ real-time performance often required on initial development stages of a robotic system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abella, J., Hernandez, C., Quiṅones, E., Cazorla, F.J., Conmy, P.R., Azkarate-Askasua, M., Perez, J., Mezzetti, E., Vardanega, T.: WCET analysis methods: Pitfalls and challenges on their trustworthiness. In: 10th IEEE International Symposium on Industrial Embedded Systems (SIES), pp 1–10. IEEE, Siegen (2015)

  2. Amazon Web Services: The FreeRTOSTM Reference Manual. https://www.freertos.org/wp-content/uploads/2018/07/FreeRTOS Reference Manual_V10.0.0.pdf (2017). Accessed 23 July 2020

  3. Ando, N., Suehiro, T., Kitagaki, K., Kotoku, T., Yoon, W.K.: RT-Middleware: Distributed component middleware for RT (Robot Technology). In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 3933–3938. IEEE, Edmonton (2005)

  4. Anh, T.N.B., Tan, S.L.: Real-time operating systems for small microcontrollers. IEEE Micro 29(5), 30–45 (2009)

    Article  Google Scholar 

  5. Arad, B., Kurtser, P., Barnea, E., Harel, B., Edan, Y., Ben-Shahar, O.: Controlled lighting and illumination-independent target detection for real-time cost-efficient applications. the case study of sweet pepper robotic harvesting. Sensors 19(6), 1390 (2019)

    Article  Google Scholar 

  6. Asadi, K., Ramshankar, H., Pullagurla, H., Bhandare, A., Shanbhag, S., Mehta, P., Kundu, S., Han, K., Lobaton, E., Wu, T.: Vision-based integrated mobile robotic system for real-time applications in construction. Autom. Constr. 96, 470–482 (2018)

    Article  Google Scholar 

  7. Axer, P., et al.: Building timing predictable embedded systems. ACM Trans. Embedded Comput. Syst. 13(4) (2014)

  8. Barbalace, A., Luchetta, A., Manduchi, G., Moro, M., Soppelsa, A., Taliercio, C.: Performance comparison of VxWorks, Linux, RTAI, and Xenomai in a hard real-time application. IEEE Trans. Nucl. Sci. 55(1), 435–439 (2008)

    Article  Google Scholar 

  9. Bardaro, G., Semprebon, A., Matteucci, M.: A use case in model-based robot development using Aadl and Ros. In: Proceedings of the International Workshop on Robotics Software Engineering, pp 9–16. ACM, New York (2018)

  10. Barry, R.: Mastering the FreeRTOSTM Real Time Kernel – A Hands-On Tutorial Guide. https://www.freertos.org/wp-content/uploads/2018/07/161204%_Mastering_the_FreeRTOS_Real_Time_Kernel-A_Hands-On_Tutorial_Guide.pdf (2016). Accessed 23 July 2020

  11. Bruyninckx, H.: Open robot control software: The OROCOS project. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, vol. 3, pp 2523–2528. IEEE, Korea (2001)

  12. Bruyninckx, H.: Open robot control software: The OROCOS project. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 3, pp. 2523–2528 vol.3 (2001)

  13. Bruyninckx, H.: OROCOS: Design and implementation of a robot control software framework. In: Proceedings of IEEE International Conference on Robotics and Automation. IEEE (2002)

  14. Bucci, G., Carnevali, L., Ridi, L., Vicario, E.: Oris: A tool for modeling, verification and evaluation of real-time systems. Int J Softw Tools Technol Trans 12(5), 391–403 (2010)

    Article  Google Scholar 

  15. Bäuml, B., Hirzinger, G.: When hard realtime matters: Software for complex mechatronic systems. Robot. Autonomous Syst. 56(1), 5–13 (2008). Human Technologies: ”Know-how”

    Article  Google Scholar 

  16. Buschmann, F.: Pattern-oriented Software Architecture, a System of Patterns, vol. 1. Wiley, New Jersey (1996)

    Google Scholar 

  17. Butazzo, G.C.: Hard Real-Time Computing Systems: Predictable Scheduling Algorithms and Applications, 3rd edn. Springer, New York (2011)

    Book  Google Scholar 

  18. Carpenter, J., Funk, S., Holman, P., Srinivasan, A., Anderson, J.H., Baruah, S.K.: A Categorization of real-time multiprocessor scheduling problems and algorithms. In: Handbook of Scheduling, pp 676–695. Chapman and Hall/CRC, Boca Raton (2004)

  19. Curnow, H.J., Wichmann, B.A.: A synthetic benchmark. Comput. J. 19(1), 43–49 (1976)

    Article  Google Scholar 

  20. Donohoe, P., Shapiro, R., Weiderman, N.: Hartstone Benchmark User’s Guide. Tech. Rep., Software Engineering Institute, Carnegie Mellon University Technical Report CMU-SEI- 90-TR-1 (1990)

  21. Dudek, G., Jenkin, M.: Computational Principles of Mobile Robotics. Cambridge university press, Cambridge (2010)

    Book  Google Scholar 

  22. Elkady, A., Sobh, T.: Robotics middleware: A comprehensive literature survey and attribute-based bibliography. J. Robot. 2012, 01–15 (2012)

    Article  Google Scholar 

  23. Embedded: How to make C++ more real-time friendly. https://www.embedded.com/how-to-make-c-more-real-time-friendly/ (2014). Accessed 23 July 2020

  24. Ferdinand, C., Heckmann, R., Langenbach, M., Martin, F., Schmidt, M., Theiling, H., Thesing, S., Wilhelm, R.: Reliable and Precise Wcet determination for a real-life processor. In: Henzinger, T.A., Kirsch, C.M. (eds.) Embedded Software, pp 469–485. Springer, Berlin (2001)

  25. Flores, G., Zhou, S., Lozano, R., Castillo, P.: A vision and gps-based real-time trajectory planning for a mav in unknown and low-sunlight environments. J. Intell. Robot. Syst. 74(1-2), 59–67 (2014)

    Article  Google Scholar 

  26. Gerum, P.: Xenomai - Implementing a RTOS emulation framework on GNU/Linux. https://design.ros2.org/articles/realtime_background.html (2004). Accessed 23 July 2020

  27. Hammer, T., Bäuml, B.: The communication layer of the ardx software framework: Highly performant and realtime deterministic. J. Intell. Robot. Syst. 77(1), 171–185 (2015)

    Article  Google Scholar 

  28. Heckmann, R., Langenbach, M., Thesing, S., Wilhelm, R.: The influence of processor architecture on the design and the results of wcet tools. Proc. IEEE 91(7), 1038–1054 (2003)

    Article  Google Scholar 

  29. Jo, K., Kim, J., Kim, D., Jang, C., Sunwoo, M.: Development of autonomous car – Part II: A Case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Trans. Ind. Electron. 62(8), 5119–5132 (2015)

    Article  Google Scholar 

  30. Joseph, M., Pandya, P.: Finding response times in a Real-Time system. Comput. J. 29(5), 390–395 (1986)

    Article  MathSciNet  Google Scholar 

  31. Kamal, R.: Embedded Systems: Architecture, Programming and Design. Tata McGraw-Hill Education, New York (2011)

  32. Kamenoff, N.I., Weiderman, N.H.: Hartstone distributed benchmark: Requirements and definitions. In: Real-Time Systems Symposium, 1991. Proceedings. Twelfth, pp 199–208. IEEE (1991)

  33. Kerstens, R., Laurijssen, D., Steckel, J.: Ertis: A fully embedded real time 3D imaging sonar sensor for robotic applications. In: 2019 International Conference on Robotics and Automation (ICRA), pp 1438–1443. IEEE (2019)

  34. Kiszka, J.: The real-time driver model and first applications. In: 7Th Real-Time Linux Workshop, Lille, France (2005)

  35. Klein, M.H., Ralya, T., Pollak, B., Obenza, R., Harbour, M.G.: A Practioner’s Handbook for Real-Time Analysis: Guide to Rate Monotonic Analysis for Real-Time Systems. Kluwer Academic Publishers, Dordrecht (1993)

    Book  Google Scholar 

  36. Kopetz, H.: Real-Time Systems: Design principles for distributed embedded applications. Springer Science & Business Media, Berlin (2011)

    Book  Google Scholar 

  37. Krten, R.: Getting Started with QNX Neutrino 2: A Guide for Realtime Programmers. PARSE Software Devices, Kanata (1999)

    Google Scholar 

  38. Lee, E.A., Seshia, S.A.: Introduction to Embedded Systems - A Cyber-Physical Systems Approach, 2nd edn. MIT Press, Cambridge (2017)

    MATH  Google Scholar 

  39. Liu, C.L., Layland, J.W.: Scheduling algorithms for multiprogramming in a hard-real-time environment. J. ACM (JACM) 20(1), 46–61 (1973)

    Article  MathSciNet  Google Scholar 

  40. Liu, J.W.S.: Real-Time Systems. Prentice Hall, Upper Saddle River (2000)

    Google Scholar 

  41. Magyar, G., Sinčák, P., KrizsáN, Z.: Comparison study of robotic Middleware for robotic applications. In: Sinčák, P., Hartono, P., Virčíková, M., Vaščák, J., Jakša, R. (eds.) Emergent Trends in Robotics and Intelligent Systems, pp 121–128. Springer International Publishing, Cham (2015)

  42. Maruyama, Y., Kato, S., Azumi, T.: Exploring the performance of ROS2. In: Proceedings of the 13th International Conference on Embedded Software, p 5. ACM (2016)

  43. Metta, G., Fitzpatrick, P., Natale, L.: Yarp: Yet another robot platform. Int. J. Adv. Robot. Syst. 3(1), 8 (2006)

    Article  Google Scholar 

  44. Muratore, L., Laurenzi, A., Hoffman, E.M., Rocchi, A., Caldwell, D.G., Tsagarakis, N.G.: Xbotcore: A real-time cross-robot software platform. In: 2017 First IEEE International Conference on Robotic Computing (IRC), pp 77–80. IEEE (2017)

  45. Nise, N.S.: Control Systems Engineering, 6th edn. Wiley, New Jersey (2011)

    MATH  Google Scholar 

  46. Open Source Robotics Foundation: Introduction to real-time systems. https://design.ros2.org/articles/realtime_background.html (2019). Accessed 23 July 2020

  47. Perathoner, S., Wandeler, E., Thiele, L., Hamann, A., Schliecker, S., Henia, R., Racu, R., Ernst, R., Harbour, M.G.: Influence of different system abstractions on the performance analysis of distributed real-time systems. In: Proceedings of the 7th ACM & IEEE International Conference on Embedded Software, pp 193–202 (2007)

  48. Pinto, M.L., de Oliveira, A.S., Wehrmeister, M.A.: Real-time systems evaluation for robotics using the Hart-ROS benchmark. In: 2019 19th International Conference on Advanced Robotics (ICAR), pp 296–301. IEEE, Belo Horizonte (2019)

  49. Quigley, M., Faust, J., Foote, T., Leibs, J.: Ros: An open-source robot operating system. In: International Conference on Robotics and Automation (ICRA) - Workshop on Open Source Software, vol. 3, p 5P (2009)

  50. Rajkumar, R., Gagliardi, M., Sha, L.: The real-time Publisher/Subscriber Inter-Process communication model for distributed real-time systems: Design and implementation. In: 1st IEEE Real-Time Technology and Applications Symposium, pp 66–75. IEEE, United States (1995)

  51. Redmon, J., Angelova, A.: Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp 131–1322. IEEE (2015)

  52. RobMoSys EU H2020 Project (2017–2020). RobMoSys: Composable Models and Software for Robtics Systems - Towards an EU Digital Industrial Platform for Robotics. https://robmosys.eu (2020). Accessed 23 July 2020

  53. Sha, L., Rajkumar, R., Lehoczky, J.P.: Priority inheritance protocols: an approach to real-time synchronization. IEEE Trans. Comput. 39(9), 1175–1185 (1990)

    Article  MathSciNet  Google Scholar 

  54. Siciliano, B., Khatib, O.: Springer Handbook of Robotics. Springer, Berlin (2016)

    Book  Google Scholar 

  55. Siegwart, R., Nourbakhsh, I.R., Scaramuzza, D., Arkin, R.C.: Introduction to Autonomous Mobile Robots, 2nd edn. MIT press, Massachusetts (2011)

    Google Scholar 

  56. Stankovic, J.A.: Misconceptions about real-time computing: A serious problem for next-generation systems. Computer 21(10), 10–19 (1988)

    Article  Google Scholar 

  57. Stankovic, J.A., Rajkumar, R.: Real-time operating systems. Real-Time Syst. 28(2-3), 237–253 (2004)

    Article  Google Scholar 

  58. Subosits, J.K., Gerdes, J.C.: From the racetrack to the road: Real-time trajectory replanning for autonomous driving. IEEE Trans. Intell. Vehicles 4(2), 309–320 (2019)

    Article  Google Scholar 

  59. Tanenbaum, A.S., Van Steen, M.: Distributed Systems: Principles and Paradigms. Prentice-Hall, New Jersey (2007)

    MATH  Google Scholar 

  60. Wang, M., Liu, J.N.K.: Fuzzy logic-based real-time robot navigation in unknown environment with dead ends. Robot. Auton. Syst. 56(7), 625–643 (2008)

    Article  Google Scholar 

  61. Wehrmeister, M.A., de Freitas, E.P., Binotto, A.P.D., Pereira, C.E.: Combining aspects and object-orientation in model-driven engineering for distributed industrial mechatronics systems. Mechatronics 24(7), 844–865 (2014). https://doi.org/10.1016/j.mechatronics.201312.008

    Article  Google Scholar 

  62. Weiderman, N.H., Kamenoff, N.I.: Hartstone uniprocessor benchmark: Definitions and experiments for real-time systems. Real-Time Syst. 4(4), 353–382 (1992)

    Article  Google Scholar 

  63. Wilhelm, R., Engblom, J., Ermedahl, A., Holsti, N., Thesing, S., Whalley, D., Bernat, G., Ferdinand, C., Heckmann, R., Mitra, T., et al.: The worst-case execution-time problem – overview of methods and survey of tools. ACM Trans. Embedded Comput. Syst. (TECS) 7(3), 1–53 (2008)

    Article  Google Scholar 

  64. Xu, Y., Fang, G., Chen, S., Zou, J.J., Ye, Z.: Real-time image processing for vision-based weld seam tracking in robotic gmaw. Int. J. Adv. Manufact. Technol. 73(9–12), 1413–1425 (2014)

    Article  Google Scholar 

  65. Yaghmour, K., Masters, J., Ben-Yossef, G., Gerum, P.: Building Embedded Linux Systems, 2nd edn. O’Reilly, Sebastopol (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento Acadêmico de Eletrônica, Instituto Federal de Santa Catarina, Florianópolis, 88020-300, Brazil

    Matheus Leitzke Pinto

  2. Programa de Pós-Graduação em Engenharia Elétrica e Informática Industrial, Universidade Tecnológica Federal do Paraná, Curitiba, 80230-901, Brazil

    Marco Aurélio Wehrmeister & André Schneider de Oliveira

Authors
  1. Matheus Leitzke Pinto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Aurélio Wehrmeister
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. André Schneider de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matheus Leitzke Pinto.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Topical Collection on ICAR 2019 Special Issue

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pinto, M.L., Wehrmeister, M.A. & de Oliveira, A.S. Real-Time Performance Evaluation for Robotics. J Intell Robot Syst 101, 37 (2021). https://doi.org/10.1007/s10846-020-01301-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-020-01301-1

Keywords

Search

Navigation