Abstract
We present a novel method by which an aerial robot can learn sequences of task-relevant camera views within a multitasking environment. The robot learns these views by tracking the visual gaze of a human collaborator wearing an augmented reality headset. The spatial footprint of the human’s visual field is integrated in time and then fit to a Gaussian mixture model via expectation maximization. The modes of this model represent the visual-interest regions of the environment with each visual-interest region containing one human task. Using Q-learning, the robot is trained as to which visual-interest region it should photograph given the human’s most recent sequence of K tasks. This sequence of K tasks forms one state of a Markov Decision Process whose entry triggers an action—the robot’s selection of visual-interest region. The robot’s camera view is continuously streamed to the human’s augmented reality headset in order to artificially expand the human’s visual field-of-view. The intent is to increase the human’s multitasking performance and decrease their physical and mental effort. An experimental study is presented in which 24 humans were asked to complete toy construction tasks in parallel with spatially separated persistent monitoring tasks (e.g., buttons which would flash at random times to request input). Subjects participated in four 2-h sessions over multiple days. The efficacy of the autonomous view selection system is compared against control trials containing no assistance as well as supervised trials in which the subjects could directly command the robot to switch between views. The merits of this system were evaluated through both subjective measures, e.g., System Usability Scale and NASA Task Load Index, as well as objective measures, e.g., task completion time, reflex time, and head angular velocity. This algorithm is applicable to multitasking environments that require persistent monitoring of regions outside of a human’s (possibly restricted) field of view, e.g., spacecraft extravehicular activity.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Agan, M., Voisinet, L., & Devereaux, A. (1998). NASA’s wireless augmented reality prototype (WARP). In AIP conference proceedings (Vol. 420, pp. 236–242). AIP.
Anderson, D., & Buck, C. (1994). Free-flying camera design for space station EVA and telerobotic operations. In Space programs and technologies conference and exhibit.
Anderson, D., Buck, C., & Cohen, R. (1994). Applications of free-flying cameras for space-based operations. Technical report, SAE Technical Paper.
Atkins, E.M., Lennon, J.A., & Peasco, R.S. (2002). Vision-based following for cooperative astronaut-robot operations. In Proceedings, IEEE Aerospace conference.
Aziz, S. (2010). Lessons learned from the STS-120/ISS 10A robotics operations. Acta Astronautica, 66(1–2), 157–165.
Beard, R. W., & McLain, T. W. (2012). Small unmanned aircraft: Theory and practice. Princeton University Press.
Bentz, W., Dhanjal, S., & Panagou, D. (2019). Unsupervised learning of assistive camera views by an aerial co-robot in augmented reality multitasking environments. In Proceedings of the 2019 IEEE international conference on robotics and automation.
Bentz, W., Hoang, T., Bayasgalan, E., & Panagou, D. (2017). Complete 3-D dynamic coverage in energy-constrained multi-UAV sensor networks. Autonomous Robots, 42, 825–851.
Bentz, W. & Panagou, D. (2020). A Markov decision process approach to online learning of sequential camera views by an aerial co-robot in multitasking environments. In Submitted to the 2020 IEEE international conference on robotics and automation, under review. https://www.dropbox.com/s/0pf1n8q2vai4ddp/ICRA_2020.pdf?dl=0
Bishop, C. M. (2006). Pattern recognition and machine learning (pp. 435–439). Springer.
Bualat, M., Barlow, J., Fong, T., Provencher, C., Smith, T., & Zuniga, A. (2015). Astrobee: Developing a free-flying robot for the international space station. In AIAA SPACE 2015 conference and exposition. Pasadena, CA.
Carr, C. E., Schwartz, S. J., & Rosenberg, I. (2002). A wearable computer for support of astronaut extravehicular activity. In Sixth International Symposium on Wearable Computers, 2002. (ISWC 2002). Proceedings (pp. 23–30). IEEE.
Craik, F. I. M., & Bialystok, E. (2006). Planning and task management in older adults: Cooking breakfast. Memory & Cognition, 34(6), 1236–1249. https://doi.org/10.3758/BF03193268.
Damen, D., Haines, O., Leelasawassuk, T., Calway, A., & Mayol-Cuevas, W. (2014). Multi-user egocentric online system for unsupervised assistance on object usage. In European conference on computer vision (pp. 481–492). Springer.
Damen, D., Leelasawassuk, T., & Mayol-Cuevas, W. (2016). You-Do, I-learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance. Computer Vision and Image Understanding, 149, 98–112.
Doule, O. (2014). Ergonomy of head mounted displays inside analog spacesuit-mars analog extravehicular activities. In AIAA SPACE 2014 conference and exposition (p. 4406).
Doule, O., Miranda, D., & Hochstadt, J. (2017). Integrated display and environmental awareness system-system architecture definition. In AIAA SPACE and astronautics forum and exposition (p. 5269).
Erat, O., Isop, W. A., Kalkofen, D., & Schmalstieg, D. (2018). Drone-augmented human vision: Exocentric control for drones exploring hidden areas. IEEE Transactions on Visualization and Computer Graphics, 24(4), 1437–1446. https://doi.org/10.1109/TVCG.2018.2794058.
Fong, T., Nourbakhsh, I., Kunz, C., Fluckiger, L., Schreiner, J., Ambrose, R., Burridge, R., Simmons, R., Hiatt, L., Schultz, A., Trafton, J. G., Bugajska, M., & Scholtz, J. (2005). The peer-to-peer human-robot interaction project. In SPACE 2005. Long Beach, CA.
Fredrickson, S. E., Duran, S., Howard, N., & Wagenknecht, J. D. (2004). Application of the mini AERCam free flyer for orbital inspection. In Defense and security (pp. 26–35). International Society for Optics and Photonics.
Hall, S. (2004). Model following control strategies and human interface techniques for the treatment of time delay during teleoperation. Ph.D. Thesis, University of Maryland.
Hedayati, H., Walker, M., & Szafir, D. (2018). Improving collocated robot teleoperation with augmented reality. In Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction, HRI ’18 (pp. 78–86). ACM. ISBN: 978-1-4503-4953-6.
Hitchcock, A. & Sung, K. (2018). Multi-view augmented reality with a drone. In Proceedings of the 24th ACM symposium on virtual reality software and technology (p. 108). ACM.
Hoover, A. W. (2016). Lecture notes: Particle filter. http://cecas.clemson.edu/~ahoover/ece854/lecture-notes/lecture-pf.pdf. Lecture notes.
Howell, D. C. (2010). Statistical methods for psychology, chapter 5 (7th ed.). PWS-Kent Pub. Co.
Hunziker, H. (2006). Im Auge Des Lesers [The eye of the reader: Foveal and peripheral perception—From letter recognition to the joy of reading] (in German). Transmedia Stäubli Verlag.
Karami, A., Jeanpierre, L., & Mouaddib, A. (2010). Human–robot collaboration for a shared mission. In 2010 5th ACM/IEEE international conference on human-robot interaction (HRI) (pp. 155–156).
Lasota, P., Nikolaidis, S., & Shah, J. A. (2013). Developing an adaptive robotic assistant for close proximity human-robot collaboration in space. In AIAA Infotech@ Aerospace (I@ A) Conference (p. 4806).
LeVasseur, D. (2015). The history of SPHERES. https://www.nasa.gov/spheres/history.html
Lewis, F. L., & Liu, D. (2013). Reinforcement learning and approximate dynamic programming for feedback control (pp. 205–209). Cham: Wiley.
McGhan, C. L., Nasir, A., & Atkins, E. M. (2015). Human intent prediction using Markov decision processes. Journal of Aerospace Information Systems, 12(5), 393–397.
Nguyen, T. H. C., Nebel, J. C., Florez-Revuelta, F., et al. (2016). Recognition of activities of daily living with egocentric vision: A review. Sensors, 16(1), 72.
Nister, D. & Stewenius, H. (2006). Scalable recognition with a vocabulary tree. In 2006 IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 2161–2168). IEEE.
Nith, R., & Rekimoto, J. (2019). Falconer: A tethered aerial companion for enhancing personal space.
Papachristos, C., & Alexis, K. (2016). Augmented reality-enhanced structural inspection using aerial robots. In IEEE International Symposium on Intelligent Control (ISIC). Buenos Aires, Argentina.
Reardon, C., Lee, K., & Fink, J. (2018). Come see this! Augmented reality to enable human-robot cooperative search. In 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) (pp. 1–7).
Reardon, C., Lee, K., Rogers, J. G., & Fink, J. (2019). Augmented reality for human–robot teaming in field environments. In J. Y. Chen & G. Fragomeni (Eds.), )Augmented virtual & mixed reality. Applications and case studies (pp. 79–92). Springer.
Sandor, C., Cunningham, A., Dey, A., & Mattila, V. V. (2010). An augmented reality X-ray system based on visual saliency. In 2010 IEEE international symposium on mixed and augmented reality (pp. 27–36). IEEE.
Sauro, J. (2011). Measuring usability with the system usability scale (SUS). https://measuringu.com/sus/
Scharf, R. (2017). ISS and orion inspection capabilities and challenges. Technical Report, NASA Johnson Space Center. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20170000805.pdf
Schmidt, R. F. (1986). Fundamentals of sensory physiology (3rd ed., p. 159). Springer.
Stolen, M. F., Dillow, B., Jacobs, S. E., & Akin, D. L. (2008). Interface for EVA human–machine interaction. Technical report, SAE Technical Paper.
Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics (intelligent robotics and autonomous agents) (pp. 80–82). The MIT Press. ISBN: 0262201623.
Todorov, I., Del Missier, F., & Mäntylä, T. (2014). Age-related differences in multiple task monitoring. PLOS ONE, 9(9), 1–7. https://doi.org/10.1371/journal.pone.0107619.
Trafton, J. G., Cassimatis, N. L., Bugajska, M. D., Brock, D. P., Mintz, F. E., & Schultz, A. C. (2005). Enabling effective human–robot interaction using perspective-taking in robots. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 35(4), 460–470. https://doi.org/10.1109/TSMCA.2005.850592.
Walker, M., Hedayati, H., Lee, J., & Szafir, D. (2018). Communicating robot motion intent with augmented reality. In Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction, HRI ’18 (pp. 316–324). ACM. ISBN: 978-1-4503-4953-6.
Wen, M. C, & Kang, S. C. (2014). Augmented reality and unmanned aerial vehicle assist in construction management. In 2014 international conference on computing in civil and building engineering. https://doi.org/10.1061/9780784413616.195
Wu, C. J., & Hamada, M. (2009). Experiments: Planning, analysis, and optimization, chapter 2 (2nd ed.). Cham: Wiley.
Zhang, J., Zhuang, L., Wang, Y., Zhou, Y., Meng, Y., & Hua, G. (2013). An egocentric vision based assistive co-robot. In 2013 IEEE International Conference on Rehabilitation Robotics (ICORR) (pp. 1–7). IEEE.
Acknowledgements
The authors would like to acknowledge the support of a NASA Early Career Faculty Grant NNX16AT43G. Upon completion of his Ph.D., William Bentz was hired as a full time employee of NASA.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors do not believe that this constitutes a conflict of interest and have no further financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The authors would like to acknowledge the support of an Early Career Faculty grant from NASA’s Space Technology Research Grants Program.
Rights and permissions
About this article
Cite this article
Bentz, W., Qian, L. & Panagou, D. Expanding human visual field: online learning of assistive camera views by an aerial co-robot. Auton Robot 46, 949–970 (2022). https://doi.org/10.1007/s10514-022-10059-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10514-022-10059-4