Skip to main content
SpringerLink
Log in

Optical Brain Imaging to Enhance UAV Operator Training, Evaluation, and Interface Development

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

As the use of unmanned aerial vehicles expands to near earth applications and force multiplying scenarios, current methods of operating UAVs and evaluating pilot performance need to expand as well. Many human factors studies on UAV operations rely on self reporting surveys to assess the situational awareness and cognitive workload of an operator during a particular task, which can make objective evaluations difficult. Functional Near-Infrared Spectroscopy (fNIR) is an emerging optical brain imaging technology that monitors brain activity in response to sensory, motor, or cognitive activation. fNIR systems developed during the last decade allow for a rapid, non-invasive method of measuring the brain activity of a subject while conducting tasks in realistic environments. This paper investigates deployment of fNIR for monitoring UAV operator’s cognitive workload and situational awareness during simulated missions. The experimental setup and procedures are presented with some early results supporting the use of fNIR for enhancing UAV operator training, evaluation and interface development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Schmidt, J., Parker, R.: Development of a UAV mishap factors database. In: Association of Unmanned Vehicle Systems, pp. 310–315. Washington, DC (1995)

  2. Rash, C.E., Leduc, P.A., Manning, S.D.: Human factors in U.S. military unmanned aerial vehicle accidents. Human Factors of Remotely Operated Vehicles 7, 117–131 (2006)

    Article  Google Scholar 

  3. Manning, S., et al.: The role of human causal factors in US army unmanned aerial vehicle accidents, USAARL Report No. 2004-11. U.A.A.R. Laboratory, Editor. Fort Rucker, AL (2004)

  4. Shappell, S., Wiegmann, D.: The Human Factors Analysis and Classification System—HFACS. Office of Aviation Medicine, Washington, DC (2000)

    Google Scholar 

  5. Hart, S.G., Staveland, L.E.: Development of the NASA-TLX (Task Load Index): results of the experimental and theorectical research. Human Mental Workload, pp. 139–183 (1988)

  6. Izzetoglu, M., et al.: Functional brain imaging using near infrared technology for cognitive activity assessment. In: IEEE Engineering in Medicine and Biology Magazine, Special issue on on the Role of Optical Imaging in Augmented Cognition, pp. 38–46 (2007)

  7. Weibel, R.E., Hansman, R.J.: Safety Considerations for Operation of Unmanned Aerial Vehicles in the National Airspace System. MIT International Center for Air Transportation, Cambridge (2005)

    Google Scholar 

  8. Oh, P.Y., Valavanis, K., Woods, R.: UAV workshop on civilian applications and commercial opportunities. In: IEEE International Conference on Robotics and Automation. Pasedena, CA (2008)

  9. Dixon, S.R., Wickens, C.D.: Control of Multiple-UAVs: a workload analysis. In: International Symposium on Aviation Psychology. Dayton, OH (2003)

  10. Kaber, D.B., Onal, E., Endley, M.R.: Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload. Hum. Factors Ergon. Manuf. 10(4), 409–430 (2000)

    Article  Google Scholar 

  11. Murphy, R.: Human–robot interaction in rescue robotics. IEEE Trans. Syst. Man Cybern. 34(2), 138–153 (2004)

    Article  MathSciNet  Google Scholar 

  12. Endlesy, M.R.: Design and evaluation for situation awareness enhancements. In: Proceedings of the Human Factors Society 32nd Annual Meeting. Santa Monica, CA (1988)

  13. Izzetoglu, K.: Neural correlates of cognitive workload and anesthetic depth: fNIR spectroscopy investigation in humans. In: Biomedical Engineering. Drexel University (2008)

  14. Dinges, D.F., Powell, J.W.: Microcomputer analyses of performace on a portable, simple visual RT task during sustained operations. Behav. Res. Meth. Instrum. Comput. 17, 652–655 (1985)

    Google Scholar 

  15. Oldfield, R.C.: The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9(1), 97–113 (1971)

    Article  Google Scholar 

  16. Izzetoglu, M., et al.: Functional brain imaging using near-infrared technology. IEEE Eng. Med. Biol. Mag. 26(4), 38–46 (2007)

    Article  Google Scholar 

  17. Izzetoglu, K., et al.: Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum.-Comput. Interact. 17(2), 211–231 (2004)

    Article  Google Scholar 

  18. Ayaz, H., et al.: Registering fNIR data to brain surface image using MRI templates. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 2671–2674 (2006)

    Article  Google Scholar 

  19. Green, W.E., Oh, P.Y.: An aerial robot prototype for situational awareness in closed quarters. In: IEEE International Conference on Intelligent Robots and Systems. Las Vegas, NV (2003)

  20. Fong, T., Thorpe, C.: Vehicle teleoperation interfaces. Auton. Robots 11(1), 9–18 (2001)

    Article  MATH  Google Scholar 

  21. Chen, J.Y.C., Haas, E.C., Barnes, M.J.: Human performance issues and user interface design for teleoperated robots. IEEE Trans. Syst. Man Cybern., Part C Appl. Rev. 37(6), 1231–1245 (2007)

    Article  Google Scholar 

  22. Van Erp, J.B.F.: Controlling unmanned vehicles: the human factors solution. In: RTO Meeting Proceedings 44 (RTO-MP-44) B8.1–B8.12 (2000)

  23. Glumm, M., Kilduff, P., Masley, A.: A Study on the Effects of Lens Focal Length on Remote Driver Performance. Tech. Rep. ARL-TR-25. Army Research Laboratory (1992)

  24. Kim, W.S., Hannaford, B., Fejczy, A.K.: Force-reflection and shared compliant control in operating telemanipulators with time delay. IEEE Trans. Robot. Autom. 8(2), 176–185 (1992)

    Article  Google Scholar 

  25. Hing, J.T., Oh, P.Y.: Development of an unmanned aerial vehicle piloting system with integrated motion cueing for training and pilot evaluation. J. Intell. Robot. Syst. 54, 3–19 (2009)

    Article  Google Scholar 

  26. Izzetoglu, K., et al.: Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum.-Comput. Interact. 17(2), 211–227 (2004)

    Article  Google Scholar 

  27. Ayaz, H., et al.: Assessment of Cognitive Neural Correlates for a Functional Near Infrared-Based Brain Computer Interface System. Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, pp. 699–708 (2009)

  28. Kelly, A., Garavan, H.: Human functional neuroimaging of brain changes associated with practice. Cerebral Cortex 15, 1089–1102 (2005)

    Article  Google Scholar 

  29. Milton, J., Small, S., Solodkin, A.: On the road to automatic: dynamic aspects in the development of expertise. Clin. Neurophysiol. 21, 134–143 (2004)

    Article  Google Scholar 

  30. Beauchamp, M.H., et al.: Dynamic functional changes associated with cognitive skill learning of an adapted version of the Tower of London task. NeuroImage 20, 1649–1660 (2003)

    Article  Google Scholar 

  31. Drury, J.L., et al.: Comparing situation awareness for two unmanned aerial vehicle human interface approaches. In: Proceedings of the SSRR 2006 Conference, Gaithersburg, MD (2006)

  32. Drury, J.L., Scott, S.D.: Awareness in unmanned aerial vehicle operations. The International C2 Journal 2(1), 1–10 (2008)

    Google Scholar 

  33. Cooper, J., Goodrich, M.A.: Towards combining UAV and sensor operator roles in UAVenabled visual search. Proceedings of ACM/IEEE International Conference on Human-Robot Interaction, pp. 351–358. Amsterdam, The Netherlands (2008)

  34. Ayaz, H., et al.: Cognitive workload assessment of air traffic controllers using optical brain imaging sensors. In: Marek, T., Karwowski, W., Rice, V. (eds.) Advances in Understanding Human Performance: Neuroergonomics, Human Factors Design, and Special Populations, pp. 21–31. CRC Press Taylor & Francis Group (2010)

Download references

Author information

Authors and Affiliations

  1. School of Biomedical Engineering, Science & Health Systems, Drexel University, 3141 Chestnut Street, Philadelphia, PA, USA

    Justin Menda, Hasan Ayaz, Patricia A. Shewokis, Kurtulus Izzetoglu & Banu Onaral

  2. Department of Mechanical Engineering and Mechanics, Drexel University, 3141 Chestnut Street, Philadelphia, PA, USA

    James T. Hing & Paul Oh

Authors
  1. Justin Menda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James T. Hing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hasan Ayaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia A. Shewokis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kurtulus Izzetoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Banu Onaral
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Paul Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justin Menda.

Additional information

The U.S. Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick, MD 21702-5014 is the awarding and administering acquisition office. This investigation was funded under a U.S. Army Medical Research Acquisition Activity; Cooperative Agreement W81XWH-08-2-0573. The content of the information herein does not necessarily reflect the position or the policy of the U.S. Government or the U.S. Army and no official endorsement should be inferred.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Menda, J., Hing, J.T., Ayaz, H. et al. Optical Brain Imaging to Enhance UAV Operator Training, Evaluation, and Interface Development . J Intell Robot Syst 61, 423–443 (2011). https://doi.org/10.1007/s10846-010-9507-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-010-9507-7

Keywords

Search

Navigation