Skip to main content

Advertisement

SpringerLink
Log in

Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper presents the concept of Cognitive Human-Machine Interfaces and Interactions (CHMI2) for Unmanned Aircraft System (UAS) Ground Control Stations (GCS). CHMI2 represents a new approach to aviation human factors engineering that introduces adaptive functionalities in the design of operators’ command, control and display functions. A CHMI2 system assesses human cognitive states based on measurement of key psycho-physiological observables. The cognitive states are used to predict and enhance operator performance in the accomplishment of aviation tasks, with the objective of improving the efficiency and effectiveness of the overall human-machine teaming. The CHMI2 system presented in this paper employs a four-layer architecture comprising sensing, extraction, classification and adaptation functionalities. An overview of each layer is provided along with the layer’s metrics, algorithms and functions. Two relevant case studies are presented to illustrate the interactions between the different layers, and the conceptual design of the associated display formats is described. The results indicate that specific eye tracking variables provide discrimination between different modes of control. Furthermore, results indicate that the higher levels of automation supported by the CHMI2 are beneficial in Separation Assurance and Collision Avoidance (SA&CA) scenarios involving low-detectability obstacles and stringent time constraints to implement recovery manoeuvres. These preliminary results highlight that the introduction of CHMI2 functionalities in future UAS can significantly reduce reaction time and enhance operational effectiveness of unmanned aircraft response to collision and loss of separation events, as well as improve the overall safety and efficiency of operations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Jiang, T., Geller, J., Ni, D., Collura, J.: Unmanned aircraft system traffic management: Concept of operation and system architecture, International Journal of Transportation Science and Technology (2017)

  2. FAA: Integration of Civil Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS) Roadmap, 1st edn. Federal Aviation Administration, Washington (2013)

  3. Dalamagkidis, K., Valavanis, K.P., Piegl, L.A.: Thoughts and Recommendations on a UAS Integration Roadmap. In: On Integrating Unmanned Aircraft Systems into the National Airspace System, ed, pp. 161–191 (2012)

  4. Gardi, A., Ramasamy, S., Kistan, T., Sabatini, R.: UAS in the terminal area: challenges and opportunities. Wiley (2016)

  5. Viguria, A.: Autonomy architectures. In: Encyclopedia of aerospace engineering. Wiley (2016)

  6. Liu, J., Gardi, A., Ramasamy, S., Lim, Y., Sabatini, R.: Cognitive pilot-aircraft interface for single-pilot operations. Knowl.-Based Syst. 112, 37–53 (2016)

    Article  Google Scholar 

  7. ICAO: Doc 10019: Manual on Remotely Piloted Aircraft Systems (RPAS), 1st edn. International Civil Aviation Organization, Montréal (2015)

    Google Scholar 

  8. Neville, K., Blickensderfer, B., Archer, J., Kaste, K., Luxion, S.P.: A Cognitive Work Analysis to Identify Human-Machine Interface Design Challenges Unique to Uninhabited Aircraft Systems. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 418–422 (2012)

  9. Ramasamy, S., Sabatini, R., Gardi, A.: Cooperative and non-cooperative sense-and-avoid in the CNS + A context: A unified methodology. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 531–539 (2016)

  10. Ramasamy, S., Sabatini, R., Gardi, A., Liu, J.: LIDAR obstacle warning and avoidance system for unmanned aerial vehicle sense-and-avoid, Aerospace Science and Technology (2016)

  11. Lai, J., Ford, J.J., Mejias, L., Wainwright, A.L., O’Shea, P.J., Walker, R.A.: Field-of-view, detection range, and false alarm trade-offs in vision-based aircraft detection (2012)

  12. Lai, J., Mejias, L., Ford, J.J.: Airborne vision-based collision-detection system. J. Field Robot. 28, 137–157 (2011)

    Article  MATH  Google Scholar 

  13. Gevins, A., Leong, H., Du, R., Smith, M.E., Le, J., DuRousseau, D., et al.: Towards measurement of brain function in operational environments. Biol. Psychol. 40, 169–186 (1995)

    Article  Google Scholar 

  14. Glaholt, M.G.: Eye tracking in the cockpit: a review of the relationships between eye movements and the aviator’s cognitive state (2014)

  15. Rubio, S., Díaz, E, Martín, J., Puente, J.M.: Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53, 61–86 (2004)

    Article  Google Scholar 

  16. Kramer, A.F.: Physiological Metrics of Mental Workload: A Review of Recent Progress. In: Multiple-Task Performance, Damos, D. London: Taylor & Francis, pp. 279–328 (1991)

  17. Gevins, A., Smith, M.E.: Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 4, 113–131 (2003)

    Article  Google Scholar 

  18. Roscoe, A.H.: Assessing pilot workload. Why measure heart rate, HRV and respiration? Biol. Psychol. 34, 259–287 (1992)

    Article  Google Scholar 

  19. Henelius, A., Hirvonen, K., Holm, A., Korpela, J., Muller, K.: Mental workload classification using heart rate metrics. In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1836–1839 (2009)

  20. Marshall, S.P.: The index of cognitive activity: Measuring cognitive workload. In: 2002. Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants, pp. 7–5-7-9 (2002)

  21. Harris, Sr., R.L., Glover, B.J., Spady, Jr., A.A.: Analytical techniques of pilot scanning behavior and their application (1986)

  22. Jacob, R., Karn, K.S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind 2, 4 (2003)

    Google Scholar 

  23. Di Nocera, F., Camilli, M., Terenzi, M.: A random glance at the flight deck: Pilots’ scanning strategies and the real-time assessment of mental workload. J. Cogn. Eng. Decis. Making 1, 271–285 (2007)

    Article  Google Scholar 

  24. Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, pp. 71–78 (2000)

  25. Gilland, J.: Driving, eye-tracking and visual entropy: Exploration of age and task effects: ProQuest (2008)

  26. Kohlmorgen, J., Dornhege, G., Braun, M., Blankertz, B., Mller, K.-.R., Curio, G., et al.: Improving Human Performance in a Real Operating Environment Through Real-time Mental Workload Detection. In: Toward Brain-Computer Interfacing, Dornhege, G., Milln, J., Hinterberger, T., McFarland D. and Mller, K.-.R. Cambridge: MIT Press, pp. 409–422 (2007)

  27. Stevens, R., Galloway, T., Berka, C.: Integrating EEG models of cognitive load with machine learning models of scientific problem solving. Augmented Cogn. Past Present Fut. 2, 55–65 (2006)

    Google Scholar 

  28. Ke, Y., Qi, H., Zhang, L., Chen, S., Jiao, X., Zhou, P., et al.: Towards an effective cross-task mental workload recognition model using electroencephalography based on feature selection and support vector machine regression. Int. J. Psychophysiol. 98, 157–166 (2015)

    Article  Google Scholar 

  29. Yin, Z., Zhang, J., Wang, R.: Neurophysiological Feature-Based Detection of Mental Workload by Ensemble Support Vector Machines. In: Advances in Cognitive Neurodynamics (V), ed, pp. 469–475. Springer (2016)

  30. Wilson, G.F., Russell, C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors J. Human Factors Ergon. Soc. 45, 635–644 (2003)

    Article  Google Scholar 

  31. Noel, J.B., Bauer, K.W., Lanning, J.W.: Improving pilot mental workload classification through feature exploitation and combination: a feasibility study. Comput. Oper. Res. 32, 2713–2730 (2005)

    Article  MATH  Google Scholar 

  32. Yang, G., Lin, Y.: Using ECG signal to quantify mental workload based on wavelet transform and competitive neural network techniques (special issuebiosensors: data acquisition, processing and control). Biomed. Fuzzy Human Sci. Off. J. Biomed. Fuzzy Syst. Assoc. 14, 17–25 (2009)

    Google Scholar 

  33. Zarjam, P., Epps, J., Chen, F., Lovell, N.H.: Estimating cognitive workload using wavelet entropy-based features during an arithmetic task. Comput. Biol. Med. 43, 2186–2195 (2013)

    Article  Google Scholar 

  34. Honal, M., Schultz, T.: Determine task demand from brain activity. In: Biosignals (1), pp. 100–107 (2008)

  35. Wang, Z., Hope, R.M., Wang, Z., Ji, Q., Gray, W.D.: Cross-subject workload classification with a hierarchical Bayes model. NeuroImage 59, 64–69 (2012)

    Article  Google Scholar 

  36. Besson, P., Bourdin, C., Bringoux, L., Dousset, E., Maïano, C., Marqueste, T., et al.: Effectiveness of physiological and psychological features to estimate helicopter pilots’ workload: A Bayesian network approach. IEEE Trans. Intell. Transp. Syst. 14, 1872–1881 (2013)

    Article  Google Scholar 

  37. Guhe, M., Gray, W.D., Schoelles, M.J., Liao, W., Zhu, Z., Ji, Q.: Non-intrusive measurement of workload in real-time. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 1157–1161 (2005)

  38. Chaouachi, M., Jraidi, I., Frasson, C.: Modeling mental workload using EEG features for intelligent systems. In: International Conference on User Modeling, Adaptation, and Personalization, pp. 50–61 (2011)

  39. Sabatini, R., Moore, T., Hill, C.: A novel GNSS integrity augmentation system for civil and military aircraft. Int. J. Mech. Indust. Sci. Eng. 7, 1433–1449 (2013)

    Google Scholar 

  40. Mulder, M., Van Der Vaart, H.J.C.: An information-centered analysis of the tunnel-in-the-sky display, part: The interception of curved trajectories. Int. J. Aviat. Psychol. 16, 21–49 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering - Aerospace Engineering and Aviation, RMIT University, Melbourne, VIC, 3083, Australia

    Yixiang Lim, Subramanian Ramasamy, Alessandro Gardi, Trevor Kistan & Roberto Sabatini

  2. THALES Australia, World Trade Centre, Melbourne, VIC, 3005, Australia

    Trevor Kistan

Authors
  1. Yixiang Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Subramanian Ramasamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessandro Gardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Trevor Kistan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Roberto Sabatini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roberto Sabatini.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lim, Y., Ramasamy, S., Gardi, A. et al. Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft. J Intell Robot Syst 91, 755–774 (2018). https://doi.org/10.1007/s10846-017-0648-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-017-0648-9

Keywords

Search

Navigation