Abstract
Two video analysis approaches (pose estimation and manual annotation) were applied to video recordings of two-person teams performing a mission planning task in a shared augmented reality (AR) environment. The analysis approaches calculated the distance relations between team members and annotated observed behaviors during the collaborative task. The 2D pose estimation algorithm lacked scene depth processing; therefore, we found some inconsistencies with the manual annotation. Although integration of the two analysis approaches was not possible, each approach by itself produced several insights on team behavior. The manual annotation analysis found four common team behaviors as well as behavior variations unique to particular teams and temporal situations. Comparing a behavior-based time on task percentage indicated behavior-type connections and some possible exclusions. The pose estimation analysis found the majority of the teams moved around the 3D scene at a similar distance apart on average with similar variation in fluctuation around a common distance range between team members. Outlying team behavior was detected by both analysis approaches and included: periods of very low distance relations, infrequent but very high distance relation spikes, significant task time spent adjusting the HoloLens device during wearing, and exceptionally long task time with gaps in pose estimation data processing.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Kase, S.E., Su, S.M., Perry, V.P., Durkee, A.C.: Observations of individual and team understanding in augmented reality mission planning. In: Proceedings of the 22nd International Command and Control Research and Technology Symposium (ICCRTS), Los Angeles, CA (2017)
Su, S., Perry, V., Guan, Q., Durkee, A., Neigel, A.R., Kase, S.: Sensor data fusion framework to improve holographic object registration accuracy for a shared augmented reality mission planning scenario. In: Chen, J.Y.C., Fragomeni, G. (eds.) Virtual, Augmented and Mixed Reality (VAMR) HCII 2018. LNCS, vol. 10909, pp. 202–214. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91581-4_15
Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments. ARI Technical Report 1014. U.S. Army Research Institute. Alexandria, VA (1994)
Kase, S., Su, S., Perry, V., Roy, H., Gamble, K.: An augmented reality shared mission planning scenario: observations on shared experience. In: Chen, J.Y.C., Fragomeni, G. (eds.) Virtual, Augmented and Mixed Reality (VAMR) HCII 2019. LNCS, vol. 11575, pp. 490–503. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21565-1_33
Kase, S., Perry, V., Roy, H., Cox, K., Su, S.: Comparative analysis of mission planning and execution times between the Microsoft HoloLens and the Surface Touch Table. In: Chen, J.Y.C., Fragomeni, G. (eds.) Virtual, Augmented and Mixed Reality (VAMR) HCII 2020. LNCS, vol. 12190, pp. 478–492. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49695-1_32
Roy, H., Cox, K.R., Fink, I., Perry, V., Su, S., Kase, S.E.: Tools for enabling teaming during mission planning and rehearsal. In: Proceedings of the SPIE Defense + Commercial Sensing Symposium 11426. Virtual, Augmented, and Mixed Reality (XR) Technology for Multi-Domain Operations, 114260A (2020)
Su, S., Perry, V., Bravo, L., Kase, S., Roy, H., Cox, H.: Virtual and augmented reality applications to support data analysis and assessment of science and engineering. Comput. Sci. Eng. 22(3), 27–29 (2020)
Su, S., Perry, V., Roy, H., Gamble, K., Kase, S.: 3D user interface for a multi-user augmented reality mission planning application. In: Cassenti, D.N. (ed.) Advances in Human Factors and Simulation, vol. 958, pp. 120–131. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-20148-7_12
Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., Gong, Y.: Locality-constrained linear coding for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp: 3360–3367, San Francisco, CA (2010)
Saldana, J.: The Coding Manual for Qualitative Researchers, pp. 1–42. SAGE Publications Ltd., Thousand Oaks (2016)
Bianco, S., Ciocca, G., Napoletano, P., Schettini, R.: An interactive tool for manual, semi-automatic and automatic video annotation. Comput. Vis. Image Underst. 131, 88–99 (2015)
Chamasemani, F.F., Affendey, L.S., Mustapha, N., Khalid, F.: Automatic video annotation framework using concept detectors. J. Appl. Sci. 15, 256–263 (2015)
Manikandan, N., Ganesan, K.: Deep learning based automatic video annotation tool for self-driving car. arXiv preprint arXiv: https://arxiv.org/abs/1904.12618 (2015)
VideoLan. VLC media player (2006). https://www.videolan.org/vlc/index.html
Cao, Z., Simon, T., Wei, S. -E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference Computer Vision Pattern Recognition, pp. 1302–1310 (2017)
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. Proc. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–186 (2021)
Hidalgo, G., Cao, Z., Simon, T., Wei, S.-E., Joo, H., Sheikh, Y.: OpenPose library. https://github.com/CMU-Perceptual-Computing-Lab/openpose
Haque, A., Peng, B., Luo, Z., Alahi, A., Yeung, S., Fei-Fei, L.: Towards viewpoint invariant 3D human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds) Proceedings of the European Conference on Computer Vision, vol. 9905, pp. 160–177 (2016). https://doi.org/10.1007/978-3-319-46448-0_10
Moon, G., Yong Chang, J., Mu Lee, K.: V2V-PoseNet: Voxel-to-voxel prediction network for accurate 3D hand and human pose estimation from a single depth map. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5079–5088 (2018)
Wang, K., Zhai, S., Cheng, H., Liang, X., Lin, L.: Human pose estimation from depth images via inference embedded multi-task learning. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 1227–1236 (2016)
Acknowledgements
This work was supported in part by the DOD High Performance Computing Modernization Program at DEVCOM Army Research Laboratory (ARL), Department of Defense Supercomputing Resource Center (DSRC).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kase, S., Perry, V., Roy, H., Cox, K., Su, S. (2021). Pose Estimation and Video Annotation Approaches for Understanding Individual and Team Interaction During Augmented Reality-Enabled Mission Planning. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality. HCII 2021. Lecture Notes in Computer Science(), vol 12770. Springer, Cham. https://doi.org/10.1007/978-3-030-77599-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-77599-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77598-8
Online ISBN: 978-3-030-77599-5
eBook Packages: Computer ScienceComputer Science (R0)