Abstract
Operating two-wheeled vehicles in four-wheel-dominant environments presents unique challenges and hazards to riders, requiring additional rider attention and resulting in increased inherent risk. Emerging display and simulation solutions offer the unique ability to help mitigate rider risk–augmented, mixed, and virtual reality (collectively extended reality; XR) can be used to rapidly prototype and test concepts, immersive virtual and mixed reality environments can be used to test systems in otherwise hard to replicate environments, and augmented and mixed reality can fuse the real world with digital information overlays and depth-based sensing capabilities to enhance rider situational awareness. This paper discusses the use of multimodal applications of XR and integration with commercial off the shelf components to create safe riding technology suites. Specifically, the paper describes informal and formal research conducted regarding the use of haptic, audio, and visual hazard alerting systems to support hands-on, heads-up, eyes-out motorcycle riding, as well as the use of an immersive mixed reality connected bicycle simulator for rapidly and representatively evaluating rider safety-augmenting technologies in a risk-free environment.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Real world virtual reality applications
- Applied augmented reality
- Motorcycle heads up displays
- Bicyclist alerting
- Riding simulation
- Multimodal alerting
1 Introduction
Operating two-wheeled vehicles in four-wheel-dominant environments presents unique challenges and hazards to riders, requiring additional rider attention and resulting in increased inherent risk. Accidents involving motorcycles or bicycles with traffic vehicles (e.g. passenger cars, SUVs, commercial trucks) are disproportionately fatal compared to other types of motorcycle or bicycle accidents. Additionally, a significant proportion of non-traffic motorcycle and bicycle accidents, compared to car accidents, are attributed to unique hazards riders face such as potholes, inclement weather, uneven terrain, sand/gravel, construction zones, and even sharp turns and steep grades. As such, heightened risk requires heightened situational awareness and safety measures for riders on the road. While solutions such as smartphone-based GPS, and handlebar-mounted alerting systems, and on-bike and/or -vehicle warning systems (e.g. Waze, SmartHalo, V2X) offer ways to warn bicyclists and motorcyclists of upcoming hazards to prevent crashes, these technologies have limited market reach (i.e., expensive, platform-specific, emerging technology) and usability flaws. For example, smartphone touchscreens are less effective for cyclists wearing protective equipment (i.e., gloves) and also require riders to momentarily glance down from the road. As such, novel approaches to significantly enhance situational awareness while simultaneously supporting heads-up, eyes-out, hands-on riding while maintain a low barrier to adoption are required to effectively alert riders to the dynamic dangers associated with riding.
In recent years, the advances in the development of multimodal systems and visual display technologies have placed XR (extended reality; collectively referring to augmented reality, mixed reality, and virtual reality) and heads up displays (HUDs) at the forefront of information visualization and interaction. These novel technologies offer effective solutions that address situational awareness and safety across real-world and virtual environments, from navigation and real time hazard alerting to testing in simulated environments.
Unlike conventional solutions, augmented reality heads up displays (HUDs) offer the ability to significantly enhance situational awareness while simultaneously supporting heads-up, eyes-out, hands-on riding. While HUDs maintain the ability to visually augment users’ field of view (FOV) with information at a relatively high level of specificity and detail, additional modalities, such as haptic and auditory displays, can also augment the riding experience while supporting attentive riding. Additionally, rather than assessing the detectability, usability, and viability of such alerting and communication modalities among the inherent dangers of live roadways, mixed reality (MR) and virtual reality (VR) offers the ability to replicate high fidelity, dynamic, and immersive riding environments that are also configurable and controllable, enabling us to test and validate augmented riding capabilities in realistic yet danger-free simulated environments.
This paper presents an overview of our work in the space of XR applied to transportation and enhancing the safety of two-wheeled riding (i.e. motorcycles and bicycles) with a specific focus on the practical application of AR for real time hazard alerting for motorcycle and bicycle riders and the use of VR for testing augmented alerting modalities in simulated riding environments.
2 Augmented Riding: On the Road
Although the automobile industry has invested in technology to improve driver safety (e.g., assisted braking, lane change hazard avoidance systems, obstacle detection capabilities, fully self-driving cars), motorcycle rider hazard avoidance technologies have remained largely unchanged over the past decades. Advances in object detection, terrain classification, and other computer vision technologies, as well as upcoming connected vehicle (CV) infrastructure present a new opportunity to enable unprecedented real-time hazard alerting for motorcyclists. However, no such technology has been adapted and made commercially available for motorcycles, despite the increasing prevalence of these types of systems on automobiles. One barrier to providing riders with routing information and real-time hazard alerts has been the inability to present this information en route in a format that does not distract riders. This barrier is quickly lowering, however, as portable display technology (e.g., smartphones, GPS devices, in-helmet audio headsets) and recent advances in consumer augmented reality (AR) displays (e.g., Microsoft HoloLens, Vuzix Blade) have paved the way for a number of motorcycle specific HUDs (e.g., NUVIZ, Crosshelmet, Everysight Raptor 2).
Compared to automobiles, warning systems present cyclists with unique challenges such as motorcycles or bicycles lacking large instrument clusters and center stacks to present warnings, and a heightened requirement to use both hands for operation. Furthermore, novel approaches to effectively provide warnings to riders must consider characteristics of the operational context (e.g., human factors, environmental factors, vehicle features) to ensure a successful solution presents hazard data to riders in a way that is not distracting but also facilitates rapid understanding to enable appropriate responses given the context of the environment.
2.1 Multimodal Applications for Rider Alerting
Outfitting riders with a HUD has the advantage of providing a display for the rider to receive information while not requiring space on the motorcycle itself. A HUD also has the benefit of allowing the technology to be ported to any motorcycle the rider may use. With a properly implemented hazard alert system, motorcyclists can benefit from many of the same technologies as cars with regard to routing and hazard alerts. Additionally, technologies that currently benefit drivers, including navigation on the center stack with applications such as Android Auto and Apple’s Car Play, can be extended to riders through a HUD. Future technologies will include alerts that utilize communications between vehicles, their operators, and the infrastructure. Through properly designed HUDs and alert systems, riders on their motorcycles can see many of the same benefits as drivers in their cars.
Under a larger effort focused on addressing these gaps through the utilization of emerging technologies for enhanced motorcycle rider safety, we designed, developed, and tested a system to support safe riding by alerting riders while en route to upcoming hazards, sourced and validated through crowdsourcing techniques, through visual, audio, and haptic AR reporting and alerting modalities (Fig. 1). We have deployed this integrated, cloud-based system to various COTS AR devices, including the NUVIZ and Everysight Raptor HUDs, the SubPac M2X, Woojer, and bHaptics vibro-tactile wearables, and Bose Frames and in-helmet audio devices.
To evaluate the effectiveness of these systems, we conducted both informal and formal evaluations, including a controlled live-riding human subjects research study to assess and validate the usability and acceptability of HUD- and audio-based hazard alerting on the NUVIZ HUD.
Informal Evaluation.
Through informal internal evaluations, including usability evaluations of interfaces, hazard alerting testing while riding in the back of a vehicle, stationary on-motorcycle haptic and audio alerting testing, and user acceptance interviews, we were able to iteratively refine, develop, and deploy visual- and audio-based alerting capabilities on the NUVIZ AR HUD to support formal human use evaluation.
For example, to incorporate audio and haptic feedback into our system, we applied the same ecological interface design (EID) techniques that we used to design the visual symbology. EID is usually applied to visual displays by using simple graphical display elements that explicitly map relevant information to emergent geometrical properties of the interface (Vincente and Rasmussen 1990; Bennett and Flach, 2011). However, an expanding body of research sought to extend the EID concepts to non-visual display channels, specifically audio and haptic displays (Sanderson and Watson 2005; Watson and Sanderson 2007; Wagman and Carello 2003). Therefore, we applied proven EID techniques and initial user feedback to design audio and haptic display symbologies to further increase rider hazard awareness. These symbologies were designed to work in isolation or in combination with each other to provide more robust alerting capabilities. For example, a rider with only a Bluetooth speaker in their helmet could hear audio alerts about an upcoming hazard type, while a rider with an AR HUD and a speaker can benefit from visual and audio alerts by seeing upcoming hazards on the display as well as hearing audio alerts about the most prominent hazard within a particular distance.
Specifically, haptic alerts have been proven extremely effective at capturing and directing attention while minimizing user annoyance and distraction. Frequency, duration, pulse rate, and intensity can all modify the user’s interpretation of and reaction to the alert. These metrics are especially important when designing a wide array of collision and hazard alerts. Relevant to motorcycle haptic alerting, we developed five alert categories that correspond to a distinct level of urgency and required response.
We informally tested the haptic outputs of these alerting categories in the intended use environment (i.e., on a motorcycle in an area with representative ambient noise) to assess test participants’ reaction to each alert. During this test, participants sat on an idling Harley Davidson 2013 Sportster Xl1200 motorcycle while each alert was played through the SubPac haptic vest. We counterbalanced alert order over the participant group. While the alert was played, participants were asked to raise their hand to signify the moment they first perceived the alert. After the alert finished playing, participants verbally ranked each alert on three Likert scales regarding perceptibility, urgency, and annoyance. Using the results, we updated our alerting symbology guidelines from our prior empirically-informed hypotheses, as outlined in Table 1.
This informal idle motorcycle on-body haptics alerting evaluation is just one example of the many informal tests and evaluations we ran before our formal live riding evaluation.
Formal Live Riding Evaluation Overview.
In our formal human subject’s research study, conducted in partnership with Virginia Tech Transportation Institute (VTTI), participants rode their personal motorcycles with visual hazard alerts presented through the NUVIZ HUD and audio alerts through a Sena 2.0 Bluetooth helmet-mounted headset, both wirelessly synced via Bluetooth. Two types of hazards were utilized (obstacles and surface hazards) and three visual hazard alert styles were paired with three audio alert styles. The visual alert styles included; full screen, single line, or map with audio components that accompanied the visual, shown in Fig. 2.
In total, three audio styles were evaluated; voice-based, tone-based, or no audio. The evaluation focused on detection, timing, comprehension, distraction and preference of the three different visual and audio alerting types during on-road riding as well as overall rider perception and interest in the technology. Overall, full screen alerts were the most effective visual alert style (map-based alerts second) and voice-based alerts were the most effective audio style. Study results showed that full screen alerts had the fewest number of collision misses and were rated as the most preferred, easiest to detect and most understandable visual alert style by participants. We also found that to maximize understandability, the full screen alerts should be paired with voice-based audio rather than tone as voice alerts scored higher than tone in comprehension. Our findings indicated that the second best and most preferred alert type was using a map-based visual alerts with feedback indicating that participants appreciate having navigation information available as they approach a hazard. When asked specifically about whether or not they found the hazard alerts to be useful, participants said that overall, they were useful. Participants see more utility in hazard alerts that utilize voice alerts rather than tone. When participants that experienced voice audio alerts were compared to participants that experienced tone-based alerts, the voice-based audio increased usefulness by 23%. Participants indicated that a helmet mounted HUD device will make riding safer, and in general they would be likely to purchase one in the future (willing to pay on average $261.36). If participants purchased a device like this, the most likely reported use case would be for riding in areas they have never visited before or for long trips. In terms of improvements, participants indicated that they would like to see a larger, brighter, and more centrally located and helmet-agnostic display that is not affected by sunlight/glare with larger and less text. Additionally, some participants expressed interest in haptic alerting (not included in VTTI evaluation) and time-to-hazard display (rather than distance). Finally, participants were asked to provide the top three hazards they think would be most useful for riders to be alerted about. The top two hazards mentioned were gravel in roadway and potholes/rough roads, and third most were traffic related, specifically crash ahead and traffic delays.
While our study examined the usability and acceptance of the system with motorcyclists, a key component to future iterations and testing emergent technology for two-wheeled riders is continuous testing and evaluation of improvements and iterations. Test courses are expensive to maintain and may not allow for full breadth of real-world conditions and limited engagements may fail to address potentially confounding human factors of technology use while en route. Novel technologies such as virtual, augmented, and mixed reality (XR) offer a low cost solution to conduct research and acquire feedback based on realistic operational scenarios. From a design standpoint, the use of AR when generating graphical display elements enabled us to acquire feedback based on operational scenarios (e.g. color display indoors vs. outdoors) during our iterative user-centered design process. This allowed us to then rapidly modify display components (e.g. display symbology) to be more intuitive and useful for end users as well as efficient in communicating hazard information to riders. Collectively, these technologies each present unique components and solutions to various aspects of research and development for emerging cyclist safety technologies. Mixed reality for example, can serve as a development-testing environment and can be used to inform future usability and fundamental research studies as it relates to effectiveness and performance. Graphical display elements as well as non-visual display elements (e.g. audio, haptic displays) can be evaluated through simulations to acquire informal user feedback based on realistic operational scenarios, while reducing the costs, risks, and time associated with human subjects live riding studies.
3 Mixed-Reality Riding Simulation
Mixed reality simulation environments offer unique capabilities to recreate immersive and dynamic real-world spatiotemporal situations that are otherwise expensive or unsafe. Mixed reality also supports multiple layers of information such as real-time geographical location, audio-visual alerts, and multimodal plugins, which allows for data and information driven end-to-end immersive experiences that meet the objectives of real-world operational environments. From a design and development standpoint, mixed reality allows for more dynamic rapid-prototyping, can serve as a development-testing environment, and can be used to inform future usability and fundamental research studies as they relate to on the road, multimodal heads-up, eyes-out riding system effectiveness and performance. However, for simulation environments to be effective, they require validation and the design of environments to be built with human and operational factors in mind. On the road data can be implemented and be used to inform the development of simulations with high levels of realism.
Under a related effort focused on connectivity for enhanced bicycle rider safety, we designed, developed, and implemented an immersive mixed reality connected bicycle simulator for rapidly and representatively evaluating rider safety-augmenting technologies in a risk-free environment. This MR bicycle simulator allows participants to enter into an immersive virtual reality (VR) urban environment and control a virtual bicycle by riding on a stationary VR-ready exercise bike (to enhance realism) and then interacting with various virtual and real-world tracked objects and variables.
3.1 Testing and Evaluation Safety
Adapting road safety technologies to cyclists, presents a need for safe testing and evaluation solutions. An ideal solution should provide an accurate model of hazards unique to motorcycles and bicycles in order to assess their potential to endanger riders to allow proper identification, characterization and assessment of hazards without risking rider safety.
There has been an increased focus on technology to increase the safety of cyclists (such as the Trek Cyclist Safety System), motivated by Federal, State, and Municipal level initiatives, such as Vision Zero (www.pedbikeinfo.org/topics/visionzero.cfm). Unfortunately, effective and ecologically valid human factors and performance testing of cycling technology is often prohibitively difficult and expensive to perform due to: (1) a lack of underlying infrastructure to support testing, (2) risks associated with conducting controlled riding studies that accurately portray the types of hazardous situations these novel technologies seek to mitigate, and (3) difficulty providing empirical evidence of the cost-benefit of this technology for the chosen environment to promote stakeholder engagement. This is particularly true for technology reliant on advanced infrastructure (e.g., DSRC V2X technology) that only have limited deployment in controlled settings in the United States. These challenges also apply to testing motorcyclist technology and if anything, have a heightened risk not only due to vehicles on the roadways but also the operation of the motorcycle itself.
3.2 NeXuS Bike Simulator
Mixed-reality simulations are a solution that allow for human subject evaluations and refinement of these technologies in ecologically valid environments, leveraging native control interfaces, and allowing developers to prototype a variety of alternate configurations to gather data on the factors most necessary for positive health out-comes. To overcome these challenges to the performance of ecologically valid cyclist safety testing, we developed the Native Cyclist Experience Using a Stationary (NeXuS) Bike Simulator system. A mixed reality, man-portable, and self-contained bicycle simulator that allows users to don a virtual reality (VR) head-mounted display (HMD) while riding a stationary exercise bicycle to immerse themselves in a simulated virtual environment (VE) designed to emulate real-world riding conditions (see Fig. 3).
In order to maximize the benefits of simulated evaluation testing, the NeXuS simulator consists of three core focus-areas: cycling realism, a scenario control module, and a dynamic agent-based traffic system. In order to ensure the actions taken by participants transfer between the real world and VE, participants control the simulator through use of a stationary bike that translates their force from pedaling to the simulator. Furthermore, to prevent participants from developing simulator sickness we use angular velocity and acceleration reported by the HMD to allow them to control their angular velocity through leaning and changes in weight distribution.
Live riding studies can be dangerous, and difficult to replicate and control. For example, live riding studies at multiple sites may have different conditions across sites that could affect the results of the study (e.g., friction, weather, lighting conditions, etc.). The scenario control module of the NeXuS simulator allows experimenters to modify and alter these environmental conditions within the simulation to provide a consistent repeatable experimental environment (see Fig. 4). The scenario control module also allows experimenters to generate a number of roadway hazards that would present hazards in a life-riding study, such as vehicles performing a right hook, and cutting off a cyclist, or vehicles running a red light as the cyclist approaches.
While dangerous cycling conditions can be evaluated under highly controlled environments, such as closed course tracks, they are difficult to evaluate in the dynamic traffic conditions a cyclist might encounter on their daily commute. The NeXuS bike simulator allows developers to simulate these conditions with its dynamic agent-based traffic system. For this traffic system, each vehicle in the simulator independently travels through the city going towards shifting randomized goals. Because the end state for each vehicle is fully randomized, traffic in the simulated environment is uniformly distributed ensuing participants encounter traffic wherever they are in the simulation. Each vehicle in the simulation independently incorporates agent-avoidance behavior, where they will change lanes to attempt to pass vehicles or cyclists if needed (determined by the scenario control module), and will make roadway decisions based on the state of city infrastructure systems (e.g., traffic lights).
3.3 Rapid Prototyping Using Simulation
The evaluation and refinement process for new technology can be prohibitively time consuming and expensive. It can require development of a prototype, development of a test track, and technological refinement. Simulator-based evaluations allow for this process to occur in a low-cost simulated environment that allows for incremental technological refinements of the technology.
Using the NeXuS bike simulator, we performed human subject evaluations of cyclist safety alerting technology powered by vehicle to vehicle (V2), and vehicle to infrastructure (V2I), connected vehicle (CV) technology. While it is expected that CV technology will be implemented in the United States in the near future, CV technology currently exists in limited-scope implementations that do not support the requirements of rigorous safety evaluations necessitated by new technology.
Using the NeXuS bike simulator we prototyped our CV-enabled cyclist alerting system as an extensible cyclist-capability module to provide new capabilities and technology to cyclists. We then designed and executed a within-subject study to evaluate and determine the optimal modality over which to alert cyclists to maximize safety and reduce distractibility. Riders rode around the simulated VR city on a pre-planned bicycle path and were presented with common cycling hazards (e.g., potholes, cars cutting cyclists off at intersections, cars overtaking from behind with minimal clearance) at controlled random intervals. Riders were then presented with various alert modalities (e.g., visual, audio, haptic handlebars or wrist wearable, or some combination thereof) and their hazard response and response time were objectively measured (compared against a control condition without the added alerting). Initial results from this evaluation were promising, indicating value in a larger follow-on study.
4 Conclusion
Extended reality platforms and simulations provide numerous benefits as end-to-end systems to design, develop, test, validate, and deploy practical XR systems as well as numerous emerging COTS devices–with particular relevance in the transportation domain. Transportation via vehicle, motorcycle, and bicycle places humans in largely the same environments with similar hazards and risk levels, yet the context of the equipment in use, system operation, user perceptibility, and viable communication displays varies greatly. For example, driving an automatic transmission car only requires one foot for gas and braking and at least one hand on the wheel for steering (albeit one hand on the wheel is more common for manual transmission driving), whereas riding a bicycle involves two feet pedaling and two hands on the handlebars for hand-braking, and riding a motorcycle typically involves one foot for shifting, one foot for braking, one hand for the throttle, and both hands for the hand brakes. Additionally, driving in a car is generally less affected by background noise than riding a bicycle, both of which are generally quieter contexts than riding a motorcycle due to the motorcycles’ noise pollution itself. As such, these unique configurations, variables, and contexts of user transportation on public roadways provide significant opportunity both for the use of novel display modalities for communicating information as well as the use of immersive simulations to evaluate these display solutions in hard-, expensive, and risky-to-replicate environments. Motorcycle and bicycle riding in particular demands hands-on, head-up, eyes-on riding while maintaining awareness of dynamic hazards among the periphery and incoming information such as turn-by-turn navigation. Augmented reality HUDs and HMDs and dashboard displays, integrated vibrotactile haptic feedback (e.g., handlebars, on-body/jacket, on-seat, on-steering wheel), and spatial audio provide robust alternatives to standard smartphone-based directions and alerting to effectively reach two-wheeled transportation modes from a safety, viability, and usability standpoint. Immersive simulations provide opportunities to test in varied scenarios that may otherwise not be feasible due to cost or saturation of the underlying technology. Simulations are also a cost-efficient solution for refinement of algorithms without concern for protocol or hardware standards driving the system. In addition to high fidelity environment replication and en route situation awareness-enhancement, XR technologies can be used as rapid prototyping environments to allow users to use technology before it is commercially available to offer feedback regarding usability, acceptability, and utility of the system. There are numerous applications for the use of multimodal display technologies at every point of the journey, from concept testing, to development, to rider deployment. As connected vehicle infrastructure continues to develop, along with the technological sophistication of motorcycles and bicycles, we are entering a new realm where vehicles can truly communicate with each other. Critical to the success of V2X however, is the effective communication of information on the road and among the vehicles, to the rider.
References
Vincente, K.J., Rasmussen, J.: The ecology of human-machine systems II: Mediating direct perception in complex work domains. Ecol. Psychol. 2, 207–249 (1990)
Bennett, K.B., Flach, J.M.: Display and Interface Design: Subtle Science, Exact Art. CRC Press, USA (2011)
Sanderson, P.M., Watson, M.O.: From information content to auditory display with ecological interface design: prospects and challenges. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 49, no. 3, pp. 259–263. Sage CA: Los Angeles, CA: SAGE Publications (2005)
Watson, M.O., Sanderson, P.M.: Designing for attention with sound: challenges and extensions to ecological interface design. Hum. factors 49(2), 331–346 (2007)
Wagman, J.B., Carello, C.: Haptically creating affordances: the user-tool interface. J. Exp. Psychol. Appl. 9(3), 175 (2003)
Acknowledgements
Research discussed in this paper was funded by the US Department of Transportation (DOT) Federal Highway Administration (FHWA).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kingsley, C., Thiry, E., Flowers, A., Jenkins, M. (2020). Augmented Riding: Multimodal Applications of AR, VR, and MR to Enhance Safety for Motorcyclists and Bicyclists. In: Stephanidis, C., Chen, J.Y.C., Fragomeni, G. (eds) HCI International 2020 – Late Breaking Papers: Virtual and Augmented Reality. HCII 2020. Lecture Notes in Computer Science(), vol 12428. Springer, Cham. https://doi.org/10.1007/978-3-030-59990-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-59990-4_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59989-8
Online ISBN: 978-3-030-59990-4
eBook Packages: Computer ScienceComputer Science (R0)