ABSTRACT
We consider a drone-based vision sensor network that captures collocated viewpoints of the scene underneath and sends them to a remote user for volumetric 360-degree navigable visual immersion on his virtual reality head-mounted display. The reconstruction quality of the immersive scene representation on the device and thus the quality of user experience will depend on the signal sampling rate and location of each drone. Moreover, there is a limit on the aggregate amount of data the network can sample and relay towards the user, stemming from transmission constraints. Finally, the user navigation actions will dynamically place different priorities on specific viewpoints of the captured scene. We make multiple contributions in this context. First, we formulate the viewpoint-priority-aware scene reconstruction error as a function of the assigned sampling rates and compute their optimal values that minimize the former, for given drone positions and system constraints. Second, we design an online view sampling policy that takes actions while exploring new drone locations to discover the best drone network configuration over the scene. We characterize its approximation versus convergence characteristics using novel spectral graph analysis and show considerable advances relative to the state-of-the-art. Finally, to enable the drone sensors to efficiently communicate their data back to the aggregation point, we formulate computationally efficient rate-distortion-power optimized transmission scheduling policies that meet the low-latency application requirements, while conserving the available energy. Our experimental results demonstrate the competitive advantages of our approach over multiple performance factors. This is a first-of-its-kind study of an emerging application of prospectively broad societal impact.
- J. G. Apostolopoulos, et al., "The road to immersive communication," Proceedings of the IEEE, vol. 100, no. 4, pp. 974--990, Apr. 2012.Google ScholarCross Ref
- T. Merel, "Augmented and virtual reality to hit $150 billion, disrupting mobile by 2020," Tech Chrunch Online Magazine, Apr. 2015.Google Scholar
- Agricultural Drones: MIT TR: 10 Breakthrough Technologies 2014.Google Scholar
- US Environ. Protection Agency, "U.S. Greenhouse Gas Inventory Report: 1990--2014," Apr. 2016.Google Scholar
- The White House. Climate Change Action Plan. http://www.whitehouse.gov/climate-change/Google Scholar
- J. Chakareski, "Uplink scheduling of visual sensors: When view popularity matters," IEEE Trans. Communications, Feb. 2015.Google ScholarCross Ref
- R. Vasudevan, et al., "Real-time stereo-vision system for 3D teleimmersive collaboration," in Proc. IEEE ICME, Jul. 2010.Google Scholar
- M. Hosseini and G. Kurillo, "Coordinated bandwidth adaptations for distributed 3D tele-immersive systems," in Proc. ACM MMVE Workshop, Mar. 2015. Google ScholarDigital Library
- G. Cheung, et al., "Interactive streaming of stored multiview video using redundant frame structures," IEEE TIP, Mar. 2011. Google ScholarDigital Library
- J. Chakareski, et al., "User-action-driven view-rate scalable multiview coding," IEEE TIP, Sep. 2013. Google ScholarDigital Library
- J. Chakareski, "Wireless streaming of interactive multi-view video via network compression and path diversity," IEEE TCOM, Apr. 2014.Google Scholar
- J. Chakareski, et al., "View-popularity-driven joint source and channel coding of view and rate scalable multi-view video," IEEE JSTSP, Apr. 2015.Google Scholar
- F. Qian, et al., "Optimizing 360 video delivery over cellular networks," in Proc. ACM Workshop All Things Cellular, Oct. 2016. Google ScholarDigital Library
- M. Hosseini and V. Swaminathan, "Adaptive 360 VR video streaming: Divide and conquer!" in Proc. IEEE ISM, Dec. 2016.Google ScholarCross Ref
- H. Q. Nguyen, et al., "Compression of human body sequences using graph wavelet filter banks," in Proc. IEEE ICASSP, May 2014.Google Scholar
- D. Thanou, et al., "Graph-based compression of dynamic 3D point cloud sequences," IEEE Trans. Image Processing, Apr. 2016.Google ScholarCross Ref
- A. Ng, et al., "Inverted autonomous helicopter flight via reinforcement learning," in Proc. Int'l Symp. Experimental Robotics, Jun. 2004.Google Scholar
- W. Burgard, et al., "Coordinated multi-robot exploration," IEEE Trans. Robotics, May 2005. Google ScholarDigital Library
- J. Delmerico, et al., "Active autonomous aerial exploration for ground robot path planning," IEEE Robotics and Automation Letters, Apr. 2017.Google ScholarCross Ref
- G. Hollinger and G. Sukhatme, "Sampling-based robotic information gathering algorithms," Int'l J. Robotics Research, Aug. 2014. Google ScholarDigital Library
- S. Gokturk, et al., "A time-of-flight depth sensor -- system description, issues and solutions," in Proc. IEEE CVPR Workshop, Jun. 2004. Google ScholarDigital Library
- M. Tanimoto, T. Fuji, and K. Suzuki, "Multi-view depth map of Rena and Akko & Kayo," ISO/IEC MPEG Document M14888, Oct. 2007.Google Scholar
- H.-Y. Shum, et al., Image-Based Rendering, Springer, Sep. 2006. Google ScholarDigital Library
- V. Velisavljević, G. Cheung, and J. Chakareski, "Bit allocation for multiview image compression using cubic synthesized view distortion model," in Proc. IEEE Int'l Hot3D Workshop, Jul. 2011. Google ScholarDigital Library
- S. Boyd and L Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Google ScholarDigital Library
- M. Chen, et al., "Utility maximization in peer-to-peer systems," in Proc. ACM SIGMETRICS, Jun. 2008. Google ScholarDigital Library
- D. Aldous and J. A. Fill, Reversible Markov Chains and Random Walks on Graphs. http://www.stat.berkeley.edu/ aldous/RWG/Google Scholar
- M. Puterman, Markov Decis. Processes, Wiley, 1994.Google Scholar
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, 1998. Google ScholarDigital Library
- H. Everett, "Generalized Lagrange multiplier method for solving problems of optimum allocation of resources," Operations Research, May-June 1963. Google ScholarDigital Library
- "VIRAT Video Dataset." http://www.viratdata.org/Google Scholar
- DJI Camera Drones for Aerial Surveillance. http://www.dji.com/Google Scholar
- W. Stevens, TCP/IP Illustr., Vol. 1: Protocols, 1994. Google ScholarDigital Library
- F. L. Lewis and D. Liu, Reinforcement learning and approximate dynamic programming for feedback control. Wiley, 2013.Google Scholar
Recommendations
First-person-view drone flying in mixed reality
SA '18: SIGGRAPH Asia 2018 PostersThis paper presents a study aimed at creating an aerial mixed reality environment by using a first-person-view drone. We develop a drone with a stereoscopic camera based on human binocular vision. Then, we present a novel mixed reality environment with ...
Drone-Augmented Human Vision: Exocentric Control for Drones Exploring Hidden Areas
Drones allow exploring dangerous or impassable areas safely from a distant point of view. However, flight control from an egocentric view in narrow or constrained environments can be challenging. Arguably, an exocentric view would afford a better ...
Multi-sensorial virtual reality and augmented human food interaction
MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food InteractionIn the field of virtual reality (VR) research, media technologies to create a realistic feeling of being present in a real/virtual world by duplicating multi-sensory information have been studied over a long period. Recently, technologies for multi-...
Comments