skip to main content
10.1145/2909824.3020256acmconferencesArticle/Chapter ViewAbstractPublication PageshriConference Proceedingsconference-collections
research-article

Marionette: Enabling On-Road Wizard-of-Oz Autonomous Driving Studies

Published: 06 March 2017 Publication History

Abstract

There is a growing need to study the interactions between drivers and their increasingly autonomous vehicles. This paper describes a method of using a low-cost, portable, and versatile driver interaction system in commercial passenger vehicles to enable on-road partial and fully autonomous driving interaction studies. By conducting on-road Wizard-of-Oz studies in naturalistic settings, we can explore a range of driving conditions and scenarios far beyond what can be conducted in laboratory simulator environments. The Marionette system uses off-the-shelf components to cre- ate bidirectional communication between the driving controls of a Wizard-of-Oz vehicle operator and a driving study participant. It signals to the study participant what the car is doing and enables researchers to study participant intervention in driving activity. Mar- ionette is designed to be easily replicated for researchers studying partially autonomous driving interaction. This paper describes the design and evaluation of this system.

Supplementary Material

suppl.mov (hrifp2885.mp4)
Supplemental video

References

[1]
James M Anderson, Kalra Nidhi, Karlyn D Stanley, Paul Sorensen, Constantine Samaras, and Oluwatobi A Oluwatola. 2014. Autonomous vehicle technology: A guide for policymakers. Rand Corporation.
[2]
Thomas Bak and Hans Jakobsen. 2004. Agricultural Robotic Platform with Four Wheel Steering for Weed Detection. Biosystems Engineering 87, 2 (2004), 125 -- 136. ISSN1537--5110
[3]
Sonia Baltodano, Srinath Sibi, Nikolas Martelaro, Nikhil Gowda, and Wendy Ju. 2015. RRADS: Real Road Autonomous Driving Simulation. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts (HRI'15 Extended Abstracts). ACM, New York, NY, USA, 283--283. ISBNx978--1--4503--3318--4
[4]
Ralf Bergholz, Klaus Timm, and Hubert Weisser. 2000. Autonomous vehicle arrangement and method for controlling an autonomous vehicle. (Nov. 21 2000). URL http://www.google.com/patents/US6151539 noteUS Patent 6,151,539.
[5]
Gerard J Blaauw. 1982. Driving experience and task demands in simulator and instrumented car: a validation study. Human Factors: The Journal of the Human Factors and Ergonomics Society 24, 4 (1982), 473--486.
[6]
Cynthia Breazeal, Cory D Kidd, Andrea Lockerd Thomaz, Guy Hoffman, and Matt Berlin. 2005. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 708--713.
[7]
Stephen M Casner, Edwin L Hutchins, and Don Norman. 2016. The challenges of partially automated driving. Commun. ACM 59, 5 (2016), 70--77.
[8]
Nils Dahlb\"ack, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz Studies: Why and How. In Proceedings of the 1st International Conference on Intelligent User Interfaces (IUI '93). ACM, New York, NY, USA, 193--200. ISBNx0--89791--556--9
[9]
Alexander Davies. 2015. I rode 500 miles in a self-driving car and saw the future. It's delightfully dull. (2015).
[10]
Frantisek Duchon, Peter Hubinskỳ, Jaroslav Hanzel, Andrej Babinec, and Michal Tölgyessy. 2012. Intelligent vehicles as the robotic applications. Procedia Engineering 48 (2012), 105--114.
[11]
Frank O Flemisch, Catherine A Adams, Sheila R Conway, Ken H Goodrich, Michael T Palmer, and Paul C Schutte. 2003. The H-Metaphor as a guideline for vehicle automation and interaction. (2003).
[12]
Barbara Freund, Stefan Gravenstein, Rebecca Ferris, and Elias Shaheen. 2002. Evaluating Driving Performance of Cognitively Impaired and Healthy Older Adults: A Pilot Study Comparing On-Road Testing and Driving Simulation. Journal of the American Geriatrics Society 50, 7 (2002), 1309--1310. ISSN1532--5415
[13]
Stuart T Godley, Thomas J Triggs, and Brian N Fildes. 2002. Driving simulator validation for speed research. Accident analysis & prevention 34, 5 (2002), 589--600.
[14]
Christian Gold, Daniel Damböck, Lutz Lorenz, and Klaus Bengler. 2013. "Take over!" How long does it take to get the driver back into the loop? Proceedings of the Human Factors and Ergonomics Society Annual Meeting 57, 1 (2013), 1938--1942.
[15]
Paul Green. 1985. The Wizard of Oz: a tool for rapid development of user interfaces. Final report. (1985). URL%http://deepblue.lib.umich.edu/handle/2027.42/174
[16]
Gabriel M Hoffmann, Claire J Tomlin, Michael Montemerlo, and Sebastian Thrun. 2007. Autonomous automobile trajectory tracking for off-road driving: Controller design, experimental validation and racing. In Proceedings of the 26th American control Conference. Citeseer, 2296--2301.
[17]
Mishel Johns, Brian Mok, David Michael Sirkin, Nikhil Manjunath Gowda, Catherine Allison Smith, Walter J. Talamonti Jr, and Wendy Ju. 2016. Exploring Shared Control in Automated Driving. In The Eleventh ACM/IEEE International Conference on Human Robot Interation (HRI '16). IEEE Press, Piscataway, NJ, USA, 91--98.
[18]
Nico Kaptein, Jan Theeuwes, and Richard Van Der Horst. 1996. Driving simulator validity: Some considerations. Transportation Research Record: Journal of the Transportation Research Board 1550 (1996), 30--36.
[19]
Krisada Kritayakirana and J. Christian Gerdes. 2012. Autonomous vehicle control at the limits of handling. International Journal of Vehicle Autonomous Systems 10, 4 (Jan. 2012), 271--296. ISSN1471-0226
[20]
Morteza Lahijanian and Marta Kwiatkowska. 2016. Social Trust: a Major Challenge for the Future of Autonomous Systems. (2016), To appear.
[21]
John D. Lee and Bobbie D. Seppelt. 2009. Springer Handbook of Automation. Springer Berlin Heidelberg, Berlin, Heidelberg, Chapter Human Factors in Automation Design, 417--436.
[22]
Todd Litman. 2014. Autonomous Vehicle Implementation Predictions. Victoria Transport Policy Institute 28 (2014). URL%http://sh.st/st/787f28ed3e745c14417e4aec27303038/http://www.vtpi.org/avip.pdf
[23]
Andrew Marinik, Richard Bishop, Vikki Fitchett, Justin F. Morgan, Tammy E. Trimble, Myra Blanco, and others. 2014. Human Factors Evaluation of Level 2 and Level 3 Automated Driving Concepts: Concepts of Operation. (2014). URL%https://vtechworks.lib.vt.edu/handle/10919/55082
[24]
Brian Mok, Mishel John, Nikhil Gowda, Srinath Sibi, and Wendy Ju. 2016. Take the Wheel: Effects of Available Modalities on Driver Intervention. In IEEE Intelligent Vehicles Symposium. IEEE.
[25]
Brian Mok, Mishel Johns, Key Jung Lee, David Miller, David Sirkin, Page Ive, and Wendy Ju. 2015. Emergency, Automation Off: Unstructured Transition Timing for Distracted Drivers of Automated Vehicles. In Intelligent Transportation Systems (ITSC), 2015 IEEE 18th International Conference on. IEEE, 2458--2464.
[26]
Michael Montemerlo, Jan Becker, Suhrid Bhat, Hendrik Dahlkamp, Dmitri Dolgov, Scott Ettinger, Dirk Haehnel, Tim Hilden, Gabe Hoffmann, Burkhard Huhnke, and others. 2008. Junior: The stanford entry in the urban challenge. Journal of field Robotics 25, 9 (2008), 569--597.
[27]
Hans B. Pacejka. 2012. Tire and Vehicle Dynamics. Butterworth-Heinemann. ISBNx978-0-08-097016--5
[28]
Cyrus Pinto. 2012. How autonomous vehicle policy in California and Nevada addresses technological and non-technological liabilities. Intersect: The Stanford Journal of Science, Technology and Society 5 (2012).
[29]
Matthew P. Reed and Paul A. Green. 1999. Comparison of driving performance on-road and in a low-cost simulator using a concurrent telephone dialling task. Ergonomics 42, 8 (1999), 1015--1037.
[30]
Laurel D Riek. 2012. Wizard of oz studies in hri: a systematic review and new reporting guidelines. Journal of Human-Robot Interaction 1, 1 (2012).
[31]
Dirk Rothenbücher, Jamy Li, David Sirkin, Brian Mok, and Wendy Ju. 2015. Ghost Driver: A Platform for Investigating Interactions Between Pedestrians and Driverless Vehicles. In Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '15). ACM, New York, NY, USA, 44--49. ISBNx978--1--4503--3858--5
[32]
Paul Saulnier, Ehud Sharlin, and Saul Greenberg. 2010. Exploring Interruption in HRI Using Wizard of Oz. In Proceedings of the 5th ACM/IEEE International Conference on Human-robot Interaction (HRI '10). IEEE Press, Piscataway, NJ, USA, 125--126. ISBNx978--1--4244--4893--7 URL%http://dl.acm.org/citation.cfm?id=1734454.1734504
[33]
Anna Schieben, Matthias Heesen, Julian Schindler, Johann Kelsch, and Frank Flemisch. 2009. The Theater-system Technique: Agile Designing and Testing of System Behavior and Interaction, Applied to Highly Automated Vehicles. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '09). ACM, New York, NY, USA, 43--46. ISBNx978--1--60558--571-0
[34]
G Schmidt, M Kiss, E Babbel, and A Galla. 2008. The Wizard on Wheels: Rapid Prototyping and User Testing of Future Driver Assistance Using Wizard of Oz Technique in a Vehicle. In Proceedings of the FISITA 2008 World Automotive Congress, Munich.
[35]
Hans-Christoph Steiner. 2009. Firmata: Towards Making Microcontrollers Act Like Extensions of the Computer. In NIME. 125--130.
[36]
Andrew R Swanson. 2013. Somebody Grab the Wheel: State Autonomous Vehicle Legislation and the Road to a National Regime. Marq. L. Rev. 97 (2013), 1085.
[37]
Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, MN Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, and others. 2008. Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics 25, 8 (2008), 425--466.
[38]
Ying Wang, Bruce Mehler, Bryan Reimer, Vincent Lammers, Lisa A. D'Ambrosio, and Joseph F. Coughlin. 2010. The validity of driving simulation for assessing differences between in-vehicle informational interfaces: A comparison with field testing. Ergonomics 53, 3 (2010), 404--420. notePMID: 20191415.
[39]
Astrid Weiss, Regina Bernhaupt, Daniel Schwaiger, Martin Altmaninger, Roland Buchner, and Manfred Tscheligi. 2009. User experience evaluation with a Wizard of Oz approach: Technical and methodological considerations. In 2009 9th IEEE-RAS International Conference on Humanoid Robots. IEEE, 303--308.

Cited By

View all
  • (2024)TimelyTale: A Multimodal Dataset Approach to Assessing Passengers' Explanation Demands in Highly Automated VehiclesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785448:3(1-60)Online publication date: 9-Sep-2024
  • (2024)GrokWalks: A Portable Virtual Reality Platform to Facilitate Studying Driver-Pedestrian InteractionsAdjunct Proceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3641308.3685013(284-288)Online publication date: 22-Sep-2024
  • (2024)Only Trust a Hidden Wizard: Investigating the Effects of Wizard Visibility in Automotive Wizard of Oz StudiesProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675738(74-82)Online publication date: 22-Sep-2024
  • Show More Cited By

Index Terms

  1. Marionette: Enabling On-Road Wizard-of-Oz Autonomous Driving Studies

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HRI '17: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction
    March 2017
    510 pages
    ISBN:9781450343367
    DOI:10.1145/2909824
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 March 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. autonomous vehicle interfaces
    2. marionette
    3. methodology
    4. on-road simulation
    5. steering wheel

    Qualifiers

    • Research-article

    Funding Sources

    • Stanford Center for Design Research Affiliates Program

    Conference

    HRI '17
    Sponsor:

    Acceptance Rates

    HRI '17 Paper Acceptance Rate 51 of 211 submissions, 24%;
    Overall Acceptance Rate 268 of 1,124 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)37
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)TimelyTale: A Multimodal Dataset Approach to Assessing Passengers' Explanation Demands in Highly Automated VehiclesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785448:3(1-60)Online publication date: 9-Sep-2024
    • (2024)GrokWalks: A Portable Virtual Reality Platform to Facilitate Studying Driver-Pedestrian InteractionsAdjunct Proceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3641308.3685013(284-288)Online publication date: 22-Sep-2024
    • (2024)Only Trust a Hidden Wizard: Investigating the Effects of Wizard Visibility in Automotive Wizard of Oz StudiesProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675738(74-82)Online publication date: 22-Sep-2024
    • (2024)The way of water: exploring the role of interaction elements in usability challenges with in-car VR experienceVirtual Reality10.1007/s10055-024-01001-328:3Online publication date: 7-Jun-2024
    • (2023)What and When to Explain?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108867:3(1-26)Online publication date: 27-Sep-2023
    • (2023)Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple WizardsProceedings of the ACM on Human-Computer Interaction10.1145/35795917:CSCW1(1-34)Online publication date: 16-Apr-2023
    • (2023)Prototyping Autonomous Vehicle Windshields with AR and Real-Time Object Detection Visualization: An On-Road Wizard-of-Oz StudyProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596051(2123-2137)Online publication date: 10-Jul-2023
    • (2023)3 Key Challenges in Designing Advanced Social Robotic Applications2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309623(1004-1011)Online publication date: 28-Aug-2023
    • (2023)Natural Language Interactions in Autonomous Vehicles: Intent Detection and Slot Filling from Passenger UtterancesComputational Linguistics and Intelligent Text Processing10.1007/978-3-031-24340-0_25(334-350)Online publication date: 26-Feb-2023
    • (2022)The Value of Context-Based Interface Prototyping for the Autonomous Vehicle Domain: A Method OverviewMultimodal Technologies and Interaction10.3390/mti70100047:1(4)Online publication date: 30-Dec-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media