Abstract
Large Language Models (LLMs) have transformed society, but modifying their internal knowledge remains challenging. Here, we focus on interference in in-context learning, examining how new knowledge affects performance in self-aware robots. We propose an evaluation benchmark based on the bAbI dataset to assess the robot’s ability to manage interference, maintain stability, ensure flexible information routing, and facilitate task performance. Addressing these challenges is crucial for improving LLMs’ effectiveness in developing self-aware robots.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bulatov, A., Kuratov, Y., Burtsev, M.S.: Scaling transformer to 1m tokens and beyond with RMT. arXiv preprint arXiv:2304.11062 (2023)
Chase, H.: Langchain (2022). https://github.com/hwchase17/langchain
Chiang, W.L., et al.: Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality (2023). https://lmsys.org/blog/2023-03-30-vicuna/
Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J.X., Schulz, E.: Meta-in-context learning in large language models (2023)
DeChant, C., Akinola, I., Bauer, D.: Learning to summarize and answer questions about a virtual robot’s past actions (2023)
Du, Y., Li, J., Tang, T., Zhao, W.X., Wen, J.R.: Zero-shot visual question answering with language model feedback (2023)
Fan, H., Liu, X., Fuh, J., Lu, W.F., Li, B.: Embodied intelligence in manufacturing: leveraging large language models for autonomous industrial robotics. J. Intell. Manuf. 1–17 (2024). https://doi.org/10.1007/s10845-023-02294-y
Garg, S., Tsipras, D., Liang, P., Valiant, G.: What can transformers learn in-context? A case study of simple function classes (2023)
Hu, Z., et al.: Deploying and evaluating LLMs to program service mobile robots (2023)
Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., Liu, B.: Continual pre-training of language models. In: The Eleventh International Conference on Learning Representations (2023)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989)
Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., Almahairi, A.: Progressive prompts: continual learning for language models (2023)
Tan, C., Chen, Y., Shao, W., Chen, W.: Make a choice! Knowledge base question answering with in-context learning (2023)
Tan, Q., Ng, H.T., Bing, L.: Towards benchmarking and improving the temporal reasoning capability of large language models (2023)
Tan, Y., et al.: Evaluation of ChatGPT as a question answering system for answering complex questions (2023)
Wang, J., et al.: Large language models for robotics: opportunities, challenges, and perspectives (2024)
Weston, J., Bordes, A., Chopra, S., Mikolov, T.: Towards AI-complete question answering: a set of prerequisite toy tasks. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1502.05698
Yuan, J., Tang, R., Jiang, X., Hu, X.: Large language models for healthcare data augmentation: an example on patient-trial matching (2023)
Acknowledgements
This research was supported by Leonardo Labs and the EMERGE Project (Grant Agreement ID: 101070918). We extend our gratitude to both entities for their generous support, feedback, and guidance, significantly contributing to the success of this project.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Coleman, E.N., Hurtado, J., Lomonaco, V. (2024). In-Context Interference In Chat-Based Large Language Models. In: Secchi, C., Marconi, L. (eds) European Robotics Forum 2024. ERF 2024. Springer Proceedings in Advanced Robotics, vol 32. Springer, Cham. https://doi.org/10.1007/978-3-031-76424-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-76424-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-76423-3
Online ISBN: 978-3-031-76424-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)