Abstract
This paper introduces the development of an autonomous seaman system, leveraging sound-to-text processing, intent detection, slot filling and action control. The Whisper model was employed to process sound commands and transcribe them, while JointBert was used to extract intentions and fill relevant slots. For enhanced model performance, Whisper model was fine-tuned using real voice data in the Portuguese language, while JointBert benefited from data generated by Chat GPT-3. To ensure effective interaction management and action execution, a directed graph structure was used as abstraction. The system’s performance was evaluated based on word error rate, intent accuracy, F1 score for slot filling and task accomplished. Experimental results showcased the effectiveness of our proposed approach, demonstrating accurate comprehension of sound commands and efficient action control. As a result, the autonomous seaman robot holds great promise for practical applications in automating diverse seafaring tasks. The improvement in the man-machine interface is very relevant for manned systems, but even more for unmanned robotic systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The Action Control block does not encompass the navigational dynamics wherein the ship is influenced by various forces necessitating distinct rudder adjustments. Instead, we have approached it as a linear problem \(\theta = x*t_{s}\), where \(\theta \) corresponds to change in direction, x to rudder degree and \(t_{s}\) to time in seconds.
- 2.
https://huggingface.co/openai/whisper-base (Apache License 2.0).
- 3.
https://github.com/monologg/JointBERT (Apache License 2.0).
- 4.
References
Veitch, E., Andreas Alsos, O.: A systematic review of human-AI interaction in autonomous ship systems. Saf. Sci. 152, 105778 (2022). https://www.sciencedirect.com/science/article/pii/S0925753522001175
Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision (2022)
Chen, Q., Zhuo, Z., Wang, W.: BERT for joint intent classification and slot filling (2019)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
French, M., Handy, R.: Spectrograms: turning signals into pictures. J. Eng. Technol. 24, 32–35 (2007)
Abdul, Z.K., Al-Talabani, A.K.: Mel frequency cepstral coefficient and its applications: a review. IEEE Access 10, 122136–122158 (2022)
Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017). http://arxiv.org/abs/1706.03762
Gulati, A., et al.: Conformer: convolution-augmented transformer for speech recognition (2020)
Chung, Y., et al.: W2V-BERT: combining contrastive learning and masked language modeling for self-supervised speech pre-training. CoRR abs/2108.06209 (2021). https://arxiv.org/abs/2108.06209
Tüske, Z., Saon, G., Kingsbury, B.: On the limit of English conversational speech recognition. CoRR abs/2105.00982 (2021). https://arxiv.org/abs/2105.00982
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. CoRR abs/1506.07503 (2015). http://arxiv.org/abs/1506.07503
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate (2016)
Latif, S., Zaidi, A., Cuayahuitl, H., Shamshad, F., Shoukat, M., Qadir, J.: Transformers in speech processing: a survey (2023)
Jurafsky, D., Martin, J.H.: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Pearson Prentice Hall, Upper Saddle River (2023)
Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: pretrained BERT models for Brazilian Portuguese. In: 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, 20–23 October (2020, to appear)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rodrigues, V.B., Lobo, V. (2024). The Automatic Seaman: From Speech2text to Text2Task. In: Marques, L., Santos, C., Lima, J.L., Tardioli, D., Ferre, M. (eds) Robot 2023: Sixth Iberian Robotics Conference. ROBOT 2023. Lecture Notes in Networks and Systems, vol 978. Springer, Cham. https://doi.org/10.1007/978-3-031-59167-9_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-59167-9_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-59166-2
Online ISBN: 978-3-031-59167-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)