Abstract
This paper introduces Jubileo, an open-source simulated humanoid robot as a framework for the development of human-robot interaction applications. By leveraging the power of the Robot Operating System (ROS) and Unity in a virtual reality environment, this simulation establishes a strong connection to real robotics, faithfully replicating the robot’s physical components down to its motors and enabling communication with servo-actuators to control both the animatronic face and the joints of a real humanoid robot. To validate the capabilities of the framework, we propose English teaching games that integrate Virtual Reality (VR), game-based Human-Robot Interaction (HRI), and advanced large language models such as Generative Pre-trained Transformer (GPT). These games aim to foster linguistic competence within dynamic and interactive virtual environments. The incorporation of large language models bolsters the robot’s capability to generate human-like responses, thus facilitating a more realistic conversational experience. Moreover, the simulation framework reduces real-world testing risks and offers a cost-effective, efficient, and scalable platform for developing new HRI applications. The paper underscores the transformative potential of converging VR, large language models, and HRI, particularly in educational applications.
Similar content being viewed by others
Available of data and material
Available in GitHub https://github.com/jajaguto/jubileo.
References
Goodrich, M., Schultz, A.: Human-robot interaction: a survey. Foundations and Trends in Human-Computer Interaction. 1, 203–275 (2007). https://doi.org/10.1561/1100000005
Terzioğlu, Y., Mutlu, B., Şhin, E.: Designing social cues for collaborative robots: the role of gaze and breathing in human-robot collaboration. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-robot Interaction, pp. 343–357 (2020)
Cao, H.-L., Scholz, C., De Winter, J., Makrini, I.E., Vanderborght, B.: Investigating the role of multi-modal social cues in human-robot collaboration in industrial settings. International Journal of Social Robotics, 1–11 (2023)
Pennisi, P., Tonacci, A., Tartarisco, G., Billeci, L., Ruta, L., Gangemi, S., Pioggia, G.: Autism and social robotics: a systematic review. Autism Res. 9(2), 165–183 (2016)
Breazeal, C., Dautenhahn, K., Kanda, T.: Social robotics. Springer handbook of robotics. 1935–1972 (2016)
Ishiguro, H.: Android science. Cognitive neuroscience robotics: a synthetic approaches to human understanding. 193–234 (2016)
Asada, M., Hosoda, K., Kuniyoshi, Y., Ishiguro, H., Inui, T., Yoshikawa, Y., Ogino, M., Yoshida, C.: Cognitive developmental robotics: A survey. IEEE Trans. Auton. Ment. Dev. 1(1), 12–34 (2009)
Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al.: Solving rubik’s cube with a robot hand (2019). arXiv:1910.07113
Monteiro, F.F., Vieira, A.L.B., Teixeira, J.M.X.N., Teichrieb, V., et al.: Simulating real robots in virtual environments using nvidia’s isaac sdk. In: Anais Estendidos do XXI Simpósio de Realidade Virtual e Aumentada. pp. 47–48 (2019). SBC
Tan, J.T.C., Inamura, T.: Sigverse-a cloud computing architecture simulation platform for social human-robot interaction. In: IEEE International Conference on Robotics and Automation. pp. 1310–1315 (2012). IEEE
Liu, O., Rakita, D., Mutlu, B., Gleicher, M.: Understanding human-robot interaction in virtual reality. In: 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). pp. 751–757 (2017). IEEE
Bottega, J.A., Kich, V.A., Kolling, A.H., Dyonisio, J.D., Corçaque, P.L., Guerra, R.d.S., Gamarra, D.F.: Jubileo: An open-source robot and framework for research in human robot social interaction. In: 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids). pp. 149–154 (2022). IEEE
Bottega, J.A., Steinmetz, R., Kolling, A.H., Kich, V.A., De Jesus, J.C., Grando, R.B., Gamarra, D.F.T.: Virtual reality platform to develop and test applications on human robot social interaction. In: 2022 Latin American Robotics Symposium (LARS), 2022 Brazilian Symposium on Robotics (SBR), and 2022 Workshop on Robotics in Education (WRE), pp. 1–6 (2022). IEEE
Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. (2023). arXiv:2303.18223
Wang, B., Li, G., Li, Y.: Enabling conversational interaction with mobile ui using large language models. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. pp. 1–17 (2023)
Ross, S.I., Martinez, F., Houde, S., Muller, M., Weisz, J.D.: The programmer’s assistant: Conversational interaction with a large language model for software development. In: Proceedings of the 28th International Conference on Intelligent User Interfaces. pp. 491–514 (2023)
Godwin-Jones, R.: Games in language learning: opportunities and challenges (2014)
Klimova, B., Kacet, J.: Efficacy of computer games on language learning. Turkish Online Journal of Educational Technology-TOJET. 16(4), 19–26 (2017)
al, M.: The uncanny valley [from the field]. IEEE Robotics & Automation Magazine. 19(2), 98–100 (2012)
Sherwani, F., Asad, M.M., Ibrahim, B.S.K.K.: Collaborative robots and industrial revolution 4.0 (ir 4.0). In: 2020 International Conference on Emerging Trends in Smart Technologies (ICETST). pp. 1–5 (2020). IEEE
Goodrich, M.A., Schultz, A.C., et al.: Human–robot interaction: a survey. Foundations and Trends® in Human–Computer Interaction. 1(3), 203–275 (2008)
Sheridan, T.B.: Human–robot interaction: status and challenges. Hum. Factors 58(4), 525–532 (2016)
Yanco, H.A., Drury, J.L.: A taxonomy for human-robot interaction. In: Proceedings of the AAAI Fall Symposium on Human-robot Interaction. pp. 111–119 (2002)
Faraj, Z., Selamet, M., Morales, C., Torres, P., Hossain, M., Chen, B., Lipson, H.: Facially expressive humanoid robotic face. HardwareX. 9, 00117 (2021)
Walters, M.L., Syrdal, D.S., Dautenhahn, K., Te Boekhorst, R., Koay, K.L.: Avoiding the uncanny valley: robot appearance, personality and consistency of behavior in an attention-seeking home scenario for a robot companion. Auton. Robot. 24, 159–178 (2008)
Złotowski, J.A., Sumioka, H., Nishio, S., Glas, D.F., Bartneck, C., Ishiguro, H.: Persistence of the uncanny valley: the influence of repeated interactions and a robot’s attitude on its perception. Front. Psychol. 6, 883 (2015)
Žlajpah, L.: Simulation in robotics. Math. Comput. Simul. 79(4), 879–897 (2008)
Datteri, E., Schiaffonati, V.: Robotic simulations, simulations of robots. Mind. Mach. 29, 109–125 (2019)
Nascimento, L.M.d., Neri, D.S., Ferreira, T.d.N., Pereira, F.d.A., Albuquerque, E.A.Y., Gonçalves, L.M.G., Sá, S.T.d.L.: Sbotics-gamified framework for educational robotics. Journal of Intelligent & Robotic Systems. 102(1), 17 (2021)
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., Tanaka, F.: Social robots for education: a review. Sci. Rob. 3(21) (2018)
Toh, L.P.E., Causo, A., Tzuo, P.-W., Chen, I.-M., Yeo, S.H.: A review on the use of robots in education and young children. Journal of Educational Technology & Society. 19(2), 148–163 (2016)
Kim, J.-O., Kim, J.: Development and application of art based steam education program using educational robot. In: Robotic Systems: Concepts, Methodologies, Tools, and Applications. pp. 1675–1687. IGI Global, ??? (2020)
Zheng, J., Chan, K., Gibson, I.: Virtual reality. Ieee. Potentials 17(2), 20–23 (1998)
Chen, Y.-L., Hsu, C.-C.: Self-regulated mobile game-based english learning in a virtual reality environment. Computers & Education. 154, 103910 (2020)
Shim, K.-C., Park, J.-S., Kim, H.-S., Kim, J.-H., Park, Y.-C., Ryu, H.-I.: Application of virtual reality technology in biology education. J. Biol. Educ. 37(2), 71–74 (2003)
Bogusevschi, D., Muntean, C., Muntean, G.-M.: Teaching and learning physics using 3d virtual learning environment: a case study of combined virtual reality and virtual laboratory in secondary school. J. Comput. Math. Sci. Teach. 39(1), 5–18 (2020)
Shahab, M., Taheri, A., Mokhtari, M., Shariati, A., Heidari, R., Meghdari, A., Alemi, M.: Utilizing social virtual reality robot (v2r) for music education to children with high-functioning autism. Education and Information Technologies, 1–25 (2022)
Ye, Y., You, H., Du, J.: Improved trust in human-robot collaboration with chatgpt (2023). arXiv:2304.12529
Christmann, G.H.G.: Um Jogo de Interação Humano-Robô para o Ensino de Língua Inglesa. Bachelor Thesis, UFSM. Advisor: Prof. Dr. Rodrigo S. Guerra (2019). http://rodrigoguerra.com/wp-content/uploads/2019/02/tcc revisado.pdf
Quigley, M., Faust, J., Foote, T., Leibs, J., et al.: Ros: an open-source robot operating system (2009)
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., Lange, D.: Unity: A General Platform for Intelligent Agents (2020)
Jocher, G., Stoken, A., Borovec, J., NanoCode012, ChristopherSTAN, Changyu, L., Laughing, tkianai, Hogan, A., lorenzomammana, yxNONG, AlexWang1900, Diaconu, L., Marc, wanghaoyang0106, ml5ah, Doug, Ingham, F., Frederik, Guilhen, Hatovix, Poznanski, J., Fang, J., Yu, L., changyu98, Wang, M., Gupta, N., Akhtar, O., PetrDvoracek, Rai, P.: ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements. Zenodo (2020). https://doi.org/10.5281/zenodo.4154370
OpenAI: GPT-4 Technical Report (2023)
Google: Google Cloud Speech-to-Text API. https://cloud.google.com/speech-to-text (2023)
Google: Google Cloud Text-to-Speech API. https://cloud.google.com/text-to-speech (2023)
Acknowledgements
The authors would like to thank the VersusAI team. This work was partly supported by the University of Tsukuba.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
Jair Augusto Bottega conceived the research, writing of the article, designed and program the experiments, collected and processed the test data. Victor Augusto Kich write the article, program the experiments, collected and processed the test data. Junior Costa de Jesus write the article, program the experiments, collected and processed the test data. Raul Steinmetz write the article, program the experiments, collected and processed the test data. Alisson Henrique Kolling write the article, program the experiments, collected and processed the test data. Ricardo Bedin Grando write the article, collected and processed the test data. Rodrigo da Silva Guerra discussion and conception of the main ideas of the article, provided valuable comments. Daniel Fernando Tello Gamarra conceived the research, writing of the article and discussion of the main ideas of the article.
Corresponding author
Ethics declarations
Ethical Approval
The article has the approval of all the authors.
Consent to Participate
All the authors gave their consent to participate in this article
Consent to Publish
The authors gave their authorization for the publishing of this article.
Competing interest
There are not conflict of interest or competing interest
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bottega, J.A., Kich, V.A., Jesus, J.C.d. et al. Jubileo: An Immersive Simulation Framework for Social Robot Design. J Intell Robot Syst 109, 91 (2023). https://doi.org/10.1007/s10846-023-01991-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-01991-3