Abstract
In this paper, the interaction between applications of augmented reality and methods of control the unmanned aerial vehicles (UAVs) is studied. During the experiment, the application of augmented reality was used to control the UAV and the camera. Assessment of quality of service of the network and subjective score of quality of experience are carried out. Hurst exponent is used to determinate the correlation between quality of service of the communication network and quality of experience.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Augmented reality
- Flying ubiquitous sensor networks
- Unmanned aerial vehicles
- Quality of service
- Quality of experience
- Hurst exponent
1 Introduction
The management of unmanned aerial vehicles is one of the most important problem, which should be solved for the widespread implementation of UAV systems. The flying touch networks use the UAVs as devices to collect data from land sensors and transfer it to the server for processing. Such networks can be used, for example, in industrial production or agricultural industry for monitoring the processes or the UAVs can be used as flying routers [1,2,3,4].
There are three types of management of the UAV: manual control, automated management, automatic control. In case of remote manual control, the pilot is guided by the image transferred from the camera. Such control can be exercised by means of technologies of augmented reality and the tactile Internet. Augmented reality (AR) represents addition of the surrounding physical world with virtual data with the help of special devices of display, for example, of glasses [5]. The main task of the tactile Internet is transfer information about impact on a remote subject and the return transfer of response of this subject to influence with the minimal delays [6].
The managing man has an opportunity to influence the UAV through the application of augmented reality turns of the head and a ducking, watching change of the image from the camera in the device of display of augmented reality.
Solving a problem of management of the UAV by means of applications of augmented reality, it is necessary to solve several subtasks:
-
to provide transfer data about turn of the head from the device of augmented reality on the quadcopter;
-
to provide transmission of a video image from the camera UAV on the AR device;
-
to estimate quality of service on the network;
-
to estimate quality of experience of the user in augmented reality at management of the UAV.
2 Features of Data Transfer
The pilot can control movements of the quadcopter or the camera mounted on it by means of turns and a ducking. As the suspension bracket of the camera provides rotation only on a vertical axis, for support of the maximum review it is necessary to turn the pilotless device. In the AR device, which is on the head of the pilot, display of a video stream from the camera is provided. Thanks to it, the pilot can understand location of the UAV and transfer information about desirable turn of the camera. For this purpose, high-quality information display from camera and a timely response of the UAV and camera to movements of the managing man should be provided.
During transmission of a video image, we can face a problem of noises in data transfer channel. Transfer of video information is influenced by such parameters of quality of service of a network as losses and time delays [6]. It influences on quality of experience and can lead failures in control of the UAV, because in case of control the pilot can possess information about location of the device proceeding from the arriving video information. In this regard, for research was selected the model of a network (Fig. 1), in which the device of augmented reality exchanges data with the quadcopter with use technology of wireless data transfer of Wi-Fi of IEEE 802.11n. This technology allows to transfer the large volume of information with throughput to 150 Mbps [7], as it is required for high-quality transmission of video information.
Within this paper, influence of quality of service of network on quality of perception of the user in the AR system at management of the UAV will be investigated.
3 Setting of Experiment
Purpose: A research of influence of quality of experience in systems of augmented reality on control of the UAV with use of the laboratory stand.
For this research the laboratory stand in laboratory of the Internet of Things of St. Petersburg State University of Telecommunications was developed [8, 9]. The hardware complex consists of the UAV (quadcopter 3D Robotics IRIS Plus) with the camera onboard. To capture video, an Intel Edison module connected to a Microsoft LifeCam HD-3000 camera is used, which allows to record HD video in 720p. Videostreaming is implemented by means of the FFServer library and using the RTP and RTSP protocols. Via the device of wireless access, the quadcopter is connected to AR glasses through Wi-Fi gateway. Epson Moverio BT-200 augmented reality glasses represent the transparent binocular video augmented reality glasses. They are equipped with such sensors as a gyroscope and the accelerometer, have an opportunity to interact through Wi-Fi access point. The AR glasses are equipped with the Android OS, which allowed us to use the LibVLC library and quickly develop an application for transmitting video from the quadcopter and managing its camera. The structural scheme of the stand is provided on Fig. 2.
For collection of the experimental data and entering of additional noises into a network, the additional server with installed NetDisturb software was implemented between augmented reality glasses and the quadcopter. NetDisturb allows to make different noises and distortions in a network between the quadcopter and points, such as time delays, losses, capacity reduction [10]. NetDisturb was installed on a PC with two Gigabit Ethernet network cards. Two Wi-Fi access points were connected to this PC. The first AP was on channel 1 and the second AP was on channel 11, acting as router and repeater, correspondingly. Also software based on the pcap.net library for interception and the analysis of a traffic was developed to estimate influence of quality of service of a network on quality of experience of augmented reality in AR glasses in case of control of the quadcopter. The experiment was carried out in a room large enough to allow the UAV fly freely up to 7 m away.
The software, created for determination of influence of quality of service on quality of experience in systems of augmented reality, intercepts a traffic of data between the quadcopter and AR glasses. On base of received data, exactly time, when the packet was received, and packet size calculation of Hurst exponent is made. Hurst exponent is the measure, used in time series analysis [11,12,13]. Parameter will decrease, if the time delay between couple of identical changes of values of a time series increases. In this case the coefficient reflects a level of self-similarity of a traffic:
-
while 0 < H < 0, 5 – a time series is not self-similar, antipersistent, more probable for changing of the direction of a deviation, high definitions of a deviation follow for low and conversely;
-
while H = 0, 5 – random time series, the following value don’t depend on the previous values;
-
while H > 0, 5 – time series is self-similar, persistent.
-
The self-similar traffic is a traffic which looks equally at any temporal scales. In other words, the probability that the traffic on the following step will deviate average value in the same direction, as on the previous step, is so high as Hurst exponent is closer to one.
For clear determination of Hurst exponent, interception of packages will be carried out during one minute. For 1 min time series we have to calculate dispersion – a deviation of a random value from its average value among.
where (R/S)t — the normalized assessment of width of a range, c — a constant, n — the size of the researched unit of observations, H — Hurst exponent, i.e. a level of self-similarity of the researched process. For the given finite time series of X = X1, X2, …, Xn we will calculate average value of m:
Then we will calculate a time series of deviations from average Y:
Calculate a time series of deviations Z:
Calculate a range series R:
Calculate a standard deviations series S:
where u — average definition of values of a row from X1 to Xt.
Calculate a normalized range series (R/S):
To average a row (R/S)t on areas [X1, Xt] in case of different values t on an interval from 1 to n, one usually takes not all n points, but some enough set, which is evenly distributed on all interval from 1 to n.
The received averaged values and corresponding to them block lengths of averaging of t, need to be marked in logarithmic axes. The received points need to be approximated the linear function, constructed by means of the least-squares method. The slope (an inclination tangent of angle to the positive direction of an X-axis) the received line according to a formula (1) will be assessment of Hurst’s coefficient of an initial time series.
In this research Hurst exponent is used for determination of correlation between quality of service and quality of experience in AR systems in case of control of the UAV.
4 Description of Experiment
The examinee of augmented reality glasses, where information from the camera of the quadcopter is displayed, turns the head, thereby directing the camera, and subjectively estimates a response of the quadcopter on the done movements and change of the image in glasses. The experiment u be made several times, each time displays influence of different noises, such as

a time delay, losses and reduction of throughput. For correct subjective evaluation, the experiment without introduction of noises was taken for a standard, i.e. as close as possible to ideal conditions of data transfer.
5 Analysis of Results of Experiment
After experiment without introduction of noises, Hurst exponent was calculated. It was equal to 0, 61. It means that the traffic was self-similar and these results can be taken as a standard in case of further assessment of quality of experience, i.e. assessment of these results is 5. Further experiments with adding of a time delay were made. Results are provided in Table 1.
From the table is seen, that in case of time delays less 100 ms, quality of experience of video remains at the satisfactory level, but there are a lot of difficulties in case of control with this delay. Because of delay the managing man doesn’t know when it is necessary to stop movement, i.e. control is made at random, commands are transferred with delay, the response returns with a time delay. In this case Hurst exponent truly defines correlation between quality of service and quality of experience in the AR systems – the greatest value complies to the highest assessment of the testee. Thus, in case of time delays higher than 150 ms it is impossible to exercise control of the UAV by means of the application of augmented reality.
The next experiments were made with adding of random losses in link between the quadcopter and the AR device. Results of this experiment are provided in Table 2.
Proceeding from the received results, one may say, that losses significantly influence quality of experience. In addition to difficulties with control, there are also noises in recognition of the image on the device of augmented reality. Quality of experience of video promptly falls in case of increase in number of losses, it becomes difficult to sort the image, besides some timepoints “fall out” of video information, that exerts a negative impact on control of the quadcopter.
The last experiment was connected to throughput restriction, resulting in packet losses.
According to Table 3 it is possible to draw a conclusion that throughput restriction critically influences quality of experience of the user in systems of augmented reality. Also in case of capacity reduction, control of the quadcopter significantly becomes more difficult, because the managing man ceases to hold information about any movement of the aircraft. All data are transferred with big time delays, part of them is lost, in case of observation in the device of augmented reality begins the dizziness.
6 Conclusion
For control of unmanned aerial vehicles, it is possible to use applications of augmented reality. Because control takes place in real time, it is necessary to impose certain requirements to quality of service of a network for support of high-quality and comfortable control of the UAV. In this paper, influence of quality of service of a network on quality of experience of the user in the device of augmented reality by transmission of a video image from the aircraft camera was considered. By the received results we will draw a conclusion that on a network with the minimal time delays to 100 ms it is possible to control the quadcopter by means of the application of augmented reality without loss of quality of experience and control.
Further, it is possible to continue a research in the area of influence of quality of service of a network on a quadcopter response (the tactile Internet) in case of control by means of the DR applications, and also to consider the AR applications as a method of management of other different robotic systems.
References
Furukawa, T., Bourgault, F., Durrant-Whyte, H.F., Dissanayake, G.: Dynamic allocation and control of coordinated UAVs to engage multiple targets in a time-optimal manner. In: IEEE International Conference on Robotics and Automation ICRA 2004, vol. 3, pp. 2353–2358 (2004)
Park, H.S., Kim, K.-H.: AR-based vehicular safety information system for forward collision warning. In: Shumaker, R., Lackey, S. (eds.) VAMR 2014. LNCS, vol. 8526, pp. 435–442. Springer, Cham (2014). doi:10.1007/978-3-319-07464-1_40
Fu, W.T., Gasper, J., Kim, S.W.: 2013 IEEE International Symposium on in Mixed and Augmented Reality (ISMAR), pp. 59–66. IEEE (2013)
Filonenko, A., Vavilin, A., Kim, T., Jo, K.-H.: Augmented reality surveillance system for road traffic monitoring. In: Huang, D.-S., Jo, K.-H., Wang, L. (eds.) ICIC 2014. LNCS (LNAI), vol. 8589, pp. 310–317. Springer, Cham (2014). doi:10.1007/978-3-319-09339-0_32
Bilton, N.: Google begins testing its augmented reality glasses, 4 April 2012
Aijaz, A., Dohler, M., Aghvami, A.H., Friderikos, V., Frodigh, M.: Realizing the tactile Internet: haptic communications over next generation 5G cellular networks. IEEE Wireless Commun. 24, 82–89 (2016). doi:10.1109/MWC.2016.1500157RP. IEEE
IEEE 802.11-2012 IEEE Standard for Information technology–Telecommunications and information exchange between systems Local and metropolitan area networks–Specific requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications
Kirichek, R., Koucheryavy, A.: Internet of Things laboratory test bed. In: Zeng, Q.-A. (ed.) Wireless Communications, Networking and Applications. LNEE, vol. 348, pp. 485–494. Springer, New Delhi (2016). doi:10.1007/978-81-322-2580-5_44
Kirichek, R., Vladyko, A., Zakharov, M., Koucheryavy, A.: Model networks for Internet of Things and SDN. In: Proceedings of the 18th ICACT, pp. 76–79 (2016)
Koucheryavy, A., Vladyko, A., Kirichek, R.: State of the art and research challenges for public flying ubiquitous sensor networks. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) ruSMART 2015. LNCS, vol. 9247, pp. 299–308. Springer, Cham (2015). doi:10.1007/978-3-319-23126-6_27
Makolkina, M., Prokopiev, A., Paramonov, A., Koucheryavy, A.: The quality of experience subjective estimations and the hurst parameters values interdependence. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) NEW2AN 2014. LNCS, vol. 8638, pp. 311–318. Springer, Cham (2014). doi:10.1007/978-3-319-10353-2_27
Kirichek, R., Paramonov, A., Koucheryavy, A.: Flying ubiquitous sensor networks as a queueing system. In: Proceedings of the 17th ICACT, pp. 127–132 (2015)
Paramonov, A., Koucheryavy, A.: M2M traffic models and flow types in case of mass event detection. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) NEW2AN 2014. LNCS, vol. 8638, pp. 294–300. Springer, Cham (2014). doi:10.1007/978-3-319-10353-2_25
Acknowledgment
The publication was financially supported by the Ministry of Education and Science of the Russian Federation (the Agreement number 02.a03.21.0008), RFBR according to the research project No. 16-37-00209 mol_a “Development of the principles of integration the Real Sense technology and Internet of Things”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 IFIP International Federation for Information Processing
About this paper
Cite this paper
Makolkina, M., Kirichek, R., Teltevskaya, V., Surodeeva, E. (2017). Research of Interaction Between Applications of Augmented Reality and Control Methods of UAVs. In: Koucheryavy, Y., Mamatas, L., Matta, I., Ometov, A., Papadimitriou, P. (eds) Wired/Wireless Internet Communications. WWIC 2017. Lecture Notes in Computer Science(), vol 10372. Springer, Cham. https://doi.org/10.1007/978-3-319-61382-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-61382-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61381-9
Online ISBN: 978-3-319-61382-6
eBook Packages: Computer ScienceComputer Science (R0)