Skip to main content

In-Context Interference In Chat-Based Large Language Models

  • Conference paper
  • First Online:
European Robotics Forum 2024 (ERF 2024)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 32))

Included in the following conference series:

  • 69 Accesses

Abstract

Large Language Models (LLMs) have transformed society, but modifying their internal knowledge remains challenging. Here, we focus on interference in in-context learning, examining how new knowledge affects performance in self-aware robots. We propose an evaluation benchmark based on the bAbI dataset to assess the robot’s ability to manage interference, maintain stability, ensure flexible information routing, and facilitate task performance. Addressing these challenges is crucial for improving LLMs’ effectiveness in developing self-aware robots.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bulatov, A., Kuratov, Y., Burtsev, M.S.: Scaling transformer to 1m tokens and beyond with RMT. arXiv preprint arXiv:2304.11062 (2023)

  2. Chase, H.: Langchain (2022). https://github.com/hwchase17/langchain

  3. Chiang, W.L., et al.: Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality (2023). https://lmsys.org/blog/2023-03-30-vicuna/

  4. Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J.X., Schulz, E.: Meta-in-context learning in large language models (2023)

    Google Scholar 

  5. DeChant, C., Akinola, I., Bauer, D.: Learning to summarize and answer questions about a virtual robot’s past actions (2023)

    Google Scholar 

  6. Du, Y., Li, J., Tang, T., Zhao, W.X., Wen, J.R.: Zero-shot visual question answering with language model feedback (2023)

    Google Scholar 

  7. Fan, H., Liu, X., Fuh, J., Lu, W.F., Li, B.: Embodied intelligence in manufacturing: leveraging large language models for autonomous industrial robotics. J. Intell. Manuf. 1–17 (2024). https://doi.org/10.1007/s10845-023-02294-y

  8. Garg, S., Tsipras, D., Liang, P., Valiant, G.: What can transformers learn in-context? A case study of simple function classes (2023)

    Google Scholar 

  9. Hu, Z., et al.: Deploying and evaluating LLMs to program service mobile robots (2023)

    Google Scholar 

  10. Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., Liu, B.: Continual pre-training of language models. In: The Eleventh International Conference on Learning Representations (2023)

    Google Scholar 

  11. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989)

    Google Scholar 

  12. Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., Almahairi, A.: Progressive prompts: continual learning for language models (2023)

    Google Scholar 

  13. Tan, C., Chen, Y., Shao, W., Chen, W.: Make a choice! Knowledge base question answering with in-context learning (2023)

    Google Scholar 

  14. Tan, Q., Ng, H.T., Bing, L.: Towards benchmarking and improving the temporal reasoning capability of large language models (2023)

    Google Scholar 

  15. Tan, Y., et al.: Evaluation of ChatGPT as a question answering system for answering complex questions (2023)

    Google Scholar 

  16. Wang, J., et al.: Large language models for robotics: opportunities, challenges, and perspectives (2024)

    Google Scholar 

  17. Weston, J., Bordes, A., Chopra, S., Mikolov, T.: Towards AI-complete question answering: a set of prerequisite toy tasks. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1502.05698

  18. Yuan, J., Tang, R., Jiang, X., Hu, X.: Large language models for healthcare data augmentation: an example on patient-trial matching (2023)

    Google Scholar 

Download references

Acknowledgements

This research was supported by Leonardo Labs and the EMERGE Project (Grant Agreement ID: 101070918). We extend our gratitude to both entities for their generous support, feedback, and guidance, significantly contributing to the success of this project.

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Coleman, E.N., Hurtado, J., Lomonaco, V. (2024). In-Context Interference In Chat-Based Large Language Models. In: Secchi, C., Marconi, L. (eds) European Robotics Forum 2024. ERF 2024. Springer Proceedings in Advanced Robotics, vol 32. Springer, Cham. https://doi.org/10.1007/978-3-031-76424-0_21

Download citation

Publish with us

Policies and ethics