skip to main content
case-study

Enhancing Non-Player Characters in Unity 3D using GPT-3.5

Published: 30 August 2024 Publication History

Abstract

This case study presents a comprehensive integration process of OpenAI's GPT-3.5 large language model into Unity 3D to enhance non-player characters (NPCs) in video games and interactive applications. The study aims to develop an architecture and open source software framework that enables NPCs to engage in dynamic real-time interactions with players and other characters. The background and motivation for the study are provided, highlighting the existing limitations of traditional NPC programming and the potential of advanced natural language models such as GPT-3.5 to overcome these limitations. The methodology section outlines the step-by-step process, covering framework design and preparation, core architecture development, humanoid avatar integration and animation, and important feature extensions. The progression of framework design and implementation is described, emphasizing key architectural concepts, design patterns, and essential classes and interfaces. The results of the case study are discussed, focusing on the valuable insights gained and the implications for future advancements. Lessons learned from the integration process are shared, along with suggestions for potential improvements and directions for future research. This case study provides a practical resource for game developers and researchers interested in leveraging advanced natural language processing capabilities to create more immersive and interactive NPC experiences in Unity 3D environments.

References

[1]
OpenAI. 2023. GPT-3.5. Retrieved May 27, 2024 from https://platform.openai.com/docs/models/gpt-3-5
[2]
Unity. 2023. Unity Engine. Retrieved May 27, 2024 from https://unity.com/products/unity-engine
[3]
Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. 2022. Craft an iron sword: Dynamically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay’22).
[4]
Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambhampati. 2023. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv e-Prints arXiv:2302.06705 (2023).
[5]
Chen Gao and Ali Emami. 2023. The Turing Quest: Can transformers make good NPCs? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). 93–103.
[6]
L. Casas and K. Mitchell. 2023. Intermediated reality with an AI 3D printed character. In SIGGRAPH’23: ACM SIGGRAPH 2023 Real-Time Live! Article 5, 2 pages.
[7]
Convai. 2023. Convai Home Page. Retrieved May 27, 2024 from https://www.convai.com/
[8]
Inworld AI. 2023. Inworld AI Home Page. Retrieved May 27, 2024 from https://www.inworld.ai/
[9]
Marc Whitten. 2023. Introducing Unity Muse and Unity Sentis, AI-powered creativity. Unity Blog. Retrieved May 27, 2024 from https://blog.unity.com/engine-platform/introducing-unity-muse-and-unity-sentis-ai
[10]
Microsoft. 2024. What Is Speech to Text? Retrieved May 27, 2024 from https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-to-text
[11]
Microsoft. 2024. What Is Text to Speech? Retrieved May 27, 2024 from https://learn.microsoft.com/en-us/azure/ai-services/speech-service/text-to-speech
[12]
OpenAI. 2023. Chat Completion API. Retrieved May 27, 2024 from https://platform.openai.com/docs/api-reference/chat/create
[13]
GitHub. 2023. OpenAI Unity API. Retrieved May 27, 2024 from https://github.com/hexthedev/OpenAi-Api-Unity
[14]
UMA Steering Group. 2023. UMA 2 - Unity Multipurpose Avatar. Retrieved May 27, 2024 from https://assetstore.unity.com/packages/3d/characters/uma-2-unity-multipurpose-avatar-35611
[15]
Ready Player Me. 2023. Ready Player Me Home Page. Retrieved May 27, 2024 from https://readyplayer.me/
[16]
Crazy Minnow Studio. 2023. SALSA LipSync. Retrieved May 27, 2024 from https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/
[17]
255 Pixel Studios. 2023. CITY package. AssetStore. Retrieved May 27, 2024 from https://assetstore.unity.com/packages/3d/environments/urban/city-package-107224
[18]
Unity Technologies. 2023. StarterAssets - Third Person Character Controller. Retrieved May 27, 2024 from https://assetstore.unity.com/packages/essentials/starter-assets-third-person-character-controller-urp-196526
[19]
Unity Technologies. 2023. Unity StarterAssets - First Person Character Controller. Retrieved May 27, 2024 from https://assetstore.unity.com/packages/essentials/starter-assets-first-person-character-controller-urp-196525
[20]
Unity Technologies. 2023. Unity Manual ScriptableObject. Retrieved May 27, 2024 from https://docs.unity3d.com/2021.3/Documentation/Manual/class-ScriptableObject.html
[21]
Unity Technologies. 2023. Unity Manual Navigation and Pathfinding. Retrieved May 27, 2024 from https://docs.unity3d.com/2021.3/Documentation/Manual/Navigation.html
[22]
Dracarys Interactive. 2023. AI Studio. Retrieved May 27, 2024 from https://github.com/Dracarys-Interactive/AIStudio
[23]
Dracarys Interactive. 2023. AI Studio. AssetStore. Retrieved May 27, 2024 from https://assetstore.unity.com/packages/tools/ai-ml-integration/ai-studio-259885

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Games: Research and Practice
Games: Research and Practice  Volume 2, Issue 3
September 2024
163 pages
EISSN:2832-5516
DOI:10.1145/3613704
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2024
Online AM: 24 May 2024
Accepted: 19 April 2024
Revised: 13 February 2024
Received: 20 July 2023
Published in GAMES Volume 2, Issue 3

Check for updates

Author Tags

  1. Non-player characters (NPCs)
  2. Unity 3D
  3. GPT-3.5
  4. large language model
  5. natural language processing
  6. video games
  7. interactive applications
  8. framework design
  9. architectural concepts
  10. real-time interactions
  11. immersive experiences

Qualifiers

  • Case-study

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 540
    Total Downloads
  • Downloads (Last 12 months)540
  • Downloads (Last 6 weeks)57
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media