Vero: An accessible method for studying human–AI teamwork

Highlights

• Presents an experimental method to study human–AI teamwork using video conferencing.

• The majority of participants believed the agent was an autonomous teammate.

• We extend Wizard-of-Oz methods to increase accessibility of human–AI experiments.

• Video-conferencing enables simultaneous study sessions with natural spoken language.

• Recommendations for deploying future studies on human-agent teaming are provided.

Abstract

Despite the recognized need to prepare for a future of human–AI collaboration, the technical skills necessary to develop and deploy AI systems are considerable, making such research difficult to perform without specialized knowledge. To make human–AI collaboration research more accessible, we developed a novel experimental method that combines a standard video conferencing platform, a set of animations, and Wizard of Oz methods to simulate a group interaction with an AI teammate. Through a case study, we demonstrate the flexibility and ease of deployment of this approach. We also provide evidence that the method creates a highly believable experience of interacting with an AI agent. By detailing this method, we hope that researchers regardless of background can replicate it to more easily answer questions that will inform the design and development of future human–AI collaboration technologies.