Skip to main content
Springer Nature Link
Log in

Assisting the Remote Assistant: Augmenting Degraded Video Streams with Additional Sensor Data to Improve Situation Awareness in Complex Urban Traffic

  • Conference paper
  • First Online:
HCI International 2024 Posters (HCII 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2118))

Included in the following conference series:

  • 765 Accesses

Abstract

Remotely operating highly automated vehicles (HAVs, SAE 4 [1]) bears the potential to boost their large-scale deployment. A human operator supports the vehicle automation remotely in situations that exceed the automation’s capabilities. A high-quality video stream displaying the HAV’s environment is key for the remote operator to obtain and maintain situation awareness. One of the major technical obstacles is the frequent and serious limitation of connectivity between the remote operator and the HAV, particularly a drop of bandwidth. This results in a severely degraded video resolution. As a remedy, we propose the use of data transmitted from the HAV’s additional onboard sensors to augment a low-resolution video stream. A significantly lower bandwidth suffices to transmit sensor compared to video data, enabling their transmission even when the video resolution is degraded. This could help the remote operator gain situation awareness, particularly in remote assistance, a variant of remote operation in which the operator provides high-level advice to the vehicle [2]. To confirm the need for a sensor data augmented view of the traffic situation, an experimental online user study (N = 117) was conducted. The study presented short video clips of complex naturalistic urban road traffic to participants. The objective was to examine if overlaying the video stream with visualized sensor data improves a remote assistant’s situation awareness and whether the effect of overlaid sensor data depends on the video resolution. Results revealed a significant effect of video resolution on objective situation awareness. Additionally, an interaction effect between resolution and sensor-data overlay became evident on the perception level of situation awareness. Hence, sensor data augmentation of degraded video streams may support the remote assistant’s situation awareness by increasing the salience of relevant elements in a traffic situation. Future research will investigate sensor data augmentation in a standardized simulation environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Society of Automotive Engineers Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (SAE J 3016–202104) (2021). https://www.sae.org/standards/content/j3016_202104. Accessed 02 Jul 2021

  2. Automated Vehicle Safety Consortium AVSC Best Practice for ADS Remote Assistance Use Case (AVSC-I-04–2023) (2023)

    Google Scholar 

  3. Straßenverkehrsgesetz: Betrieb von Kraftfahrzeugen mit autonomer Fahrfunktion; Widerspruch und Anfechtungsklage: StVG § 1e (2021)

    Google Scholar 

  4. Kettwich, C., Schrank, A., Oehl, M.: Teleoperation of highly automated vehicles in public transport: user-centered design of a human-machine interface for remote-operation and its expert usability evaluation. MTI 5, 26 (2021). https://doi.org/10.3390/mti5050026

    Article  Google Scholar 

  5. Schrank, A., Walocha, F., Brandenburg. S., et al.: Human-centered design and evaluation of a workplace for the remote assistance of highly automated vehicles. Cogn Tech Work (2024). https://doi.org/10.1007/s10111-024-00753-x

  6. Linkov, V., Vanžura, M.: Situation awareness measurement in remotely controlled cars. Front. Psychol. 12, 592930 (2021). https://doi.org/10.3389/fpsyg.2021.592930

    Article  Google Scholar 

  7. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors J. Hum. Factors Ergon. Soc. 37, 32–64 (1995). https://doi.org/10.1518/001872095779049543

    Article  Google Scholar 

  8. Amador, O., Aramrattana, M., Vinel, A.: A survey on remote operation of road vehicles. IEEE Access 10, 130135–130154 (2022). https://doi.org/10.1109/ACCESS.2022.3229168

    Article  Google Scholar 

  9. Georg, J-M., Putz, E., Diermeyer, F.: Longtime effects of video quality, video canvases and displays on situation awareness during teleoperation of automated vehicles*. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 248–255 (2020)

    Google Scholar 

  10. DLR Transport FASCar – research vehicle for automated driving (2023). https://verkehrsforschung.dlr.de/en/fascar-research-vehicle-automated-driving. Accessed 26 May 2023

  11. Endsley, M.R.: SAGAT: A methodology for the measurement of situation awareness, Northrop C, Hawthorne, CA (1987)

    Google Scholar 

  12. Endsley, M.R., Bolté, B., Jones, D.G.: Designing for Situation Awareness: An Approach to User-Centered Design. Taylor & Francis, Boca Raton, London, New York (2003)

    Book  Google Scholar 

  13. Gregoriades, A., Pampaka, M.: Enhancing drivers’ situation awareness. In: Stanton, N.A., Landry, S., Di Bucchianico, G., et al. (eds.) Advances in Human Aspects of Transportation, vol. 484, pp. 301–312. Springer International Publishing, Cham (2017)

    Chapter  Google Scholar 

  14. Mutzenich, C., Durant, S., Helman, S., et al.: Updating our understanding of situation awareness in relation to remote operators of autonomous vehicles. Cogn. Res. Princ. Implic. 6, 9 (2021). https://doi.org/10.1186/s41235-021-00271-8

    Article  Google Scholar 

  15. Endsley, M.R., Selcon, S.J., Hardiman, T.D., et al.: A comparative analysis of Sagat and Sart for evaluations of situation awareness. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 42, pp. 82–86 (1998). https://doi.org/10.1177/154193129804200119

  16. Endsley, M.R.: The divergence of objective and subjective situation awareness: a meta-analysis. J. Cogn. Eng. Decis. Making 14, 34–53 (2020). https://doi.org/10.1177/1555343419874248

    Article  Google Scholar 

  17. Taylor, R.M.: Situation awareness rating technique (SART): the development of a tool for aircrew systems design (AGARD-CP-478). In: Situational Awareness in Aerospace Operations, pp. 3/1–3/17. NATO-AGARD, Neuilly Sur Seine, France (1990)

    Google Scholar 

Download references

Acknowledgments

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 101006664. The authors would like to thank all partners within Hi-Drive for their cooperation and valuable contribution. Furthermore, the authors would like to express their gratitude to Jörg Schäfer and Philipp Schmälzle for their aid in the augmentation of the videos and to Kai Sklorz for supporting conceptualization and data collection.

Author information

Authors and Affiliations

  1. German Aerospace Center (DLR), Institute of Transportation Systems, Lilienthalplatz 7, 38108, Braunschweig, Germany

    Andreas Schrank, Nils Wendorff & Michael Oehl

Authors
  1. Andreas Schrank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nils Wendorff
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Oehl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Schrank .

Editor information

Editors and Affiliations

  1. University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  3. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Stavroula Ntoa

  4. University of Central Florida, Orlando, FL, USA

    Gavriel Salvendy

Ethics declarations

The authors have no relevant competing interests to declare.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schrank, A., Wendorff, N., Oehl, M. (2024). Assisting the Remote Assistant: Augmenting Degraded Video Streams with Additional Sensor Data to Improve Situation Awareness in Complex Urban Traffic. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2024 Posters. HCII 2024. Communications in Computer and Information Science, vol 2118. Springer, Cham. https://doi.org/10.1007/978-3-031-61963-2_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-61963-2_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-61962-5

  • Online ISBN: 978-3-031-61963-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics