Abstract
Estates, especially those of public securityrelated companies and institutes, have to protect their privacy from adversary unmanned aerial vehicles (UAVs). In this paper, we propose a reinforcement learning-based control framework to prevent unauthorized UAVs from entering a target area in a dynamic game without being aware of the UAV attack model. This UAV control scheme enables a target estate to choose the optimal control policy, such as jamming the global positioning system signals, hacking, and laser shooting, to expel nearby UAVs. A deep reinforcement learning technique, called neural episodic control, is used to accelerate the learning speed to achieve the optimal UAV control policy, especially for estates with a large area, against complicated UAV attack policies. We analyze the computational complexity for the proposed UAV control scheme and provide its performance bound, including the risk level of the estate and its utility. Our simulation results show that the proposed scheme can reduce the risk level of the target estate and improve its utility against malicious UAVs compared with the selected benchmark scheme.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Q. Wang, Z. Chen, W. Mei, et al. Improving physical layer security using UAV-enabled mobile relaying [J]. IEEE Wireless Communication Letters, 2017, 6(3): 310–313.
M. Dong, K. Ota, M. Lin, et al. UAV-assisted data gathering in wireless sensor networks [J]. The Journal of Supercomputing, 2014, 70(3): 1142–1155.
D. He, S. Chan, M. Guizani. Communication security of unmanned aerial vehicles [J]. IEEE Wireless Communications, 2016, 24(4): 134–139.
K. Mansfield, T. Eveleigh, T. H. Holzer, et al. Unmanned aerial vehicle smart device ground control station cyber security threat model [C]//IEEE International Conference on Technologies for Homeland Security (HST), Waltham, MA, 2012: 722–728.
H. Sedjelmaci, S. M. Senouci, N. Ansari. Intrusion detection and ejection framework against lethal attacks in UAV-aided networks: A Bayesian game-theoretic methodology [J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(5): 1143–1153.
D. Sathyamoorthy. A review of security threats of unmanned aerial vehicles and mitigation steps [J]. Journal of Defence and Security, 2015: 6(1): 81–97.
Y. Xiao, V. K. Rayi, B. Sun, et al. A survey of key management schemes in wireless sensor networks [J]. Computer Communications, 2007, 30(11-12): 2314–2341.
T. Wang, J. Tan, W. Ding, et al. Inter-community detection scheme for social Internet of things: A compressive sensing over graphs approach [J]. IEEE Internet of Things Journal, 2018.
E. Vattapparamban, I. Guvenc, A. Yurekli, et al. Drones for smart cities: Issues in cybersecurity, privacy, and public safety [C]//International Wireless Communications and Mobile Computing Conference (IWCMC), Paphos, Cyprus, 2016: 216–221.
L. Xiao, Y. Li, C. Dai, et al. Reinforcement learning-based NOMA power allocation in the presence of smart jamming [J]. IEEE Transactions on Vehicular Technology, 2018, 67(4): 3377–3389.
S. Lv, L. Xiao, Q. Hu, et al. Anti-jamming power control game in unmanned aerial vehicle networksneural episodic control [C]//IEEE Global Communications Conference (GLOBECOM), Singapore, 2017.
L. Xiao, Y. Li, G. Han, et al. PHY-layer spoofing detection with reinforcement learning in wireless networks [J]. IEEE Transactions on Vehicular Technology, 2016, 65(12): 10037–10047.
A. J. Kerns, D. P. Shepard, J. A. Bhatti, et al. Unmanned aircraft capture and control via GPS spoofing [J]. Field Robotics, 2014, 31(4): 617–636.
F. Yang, J. Gao. Dimming control scheme with high power and spectrum efficiency for visible light communications [J]. IEEE Photonics Journal, 2017, 9(1): 7901612.
L. Xiao, Y. Li, G. Han, et al. A secure mobile crowdsensing game with deep reinforcement learning [J]. IEEE Transactions on Information Forensics and Security, 2018, 13(1): 35–47.
M. Min, L. Xiao, D. Xu, et al. Learning-based defense against malicious unmanned aerial vehicles [C]//IEEE 87th Vehicular Technology Conference, Porto, Portugal, 2018: 1–5.
A. Pritzel, B. Uria. Neural episodic control [J]. arXiv:1703.01988, 2017.
Z. Birnbaum, A. Dolgikh, V. Skormin, et al. Unmanned aerial vehicle security using recursive parameter estimation [J]. Journal of Intelligent & Robotic Systems, 2016, 84(1-4): 107–120.
B. Zhu, A. Zaini, L. Xie. Distributed guidance for interception by using multiple rotary-wing unmanned aerial vehicles [J]. IEEE Transactions on Industrial Electronics, 2017, 64(7): 5648–5656.
S. Seo, B. Lee, S. Im, et al. Effect of spoofing on unmanned aerial vehicle using counterfeited GPS signal [J]. Positioning, Navigation, and Timing, 2015, 4(2): 57–65.
X. Du, Y. Xiao, M. Guizani, et al. An effective key management scheme for heterogeneous sensor networks [J]. Ad Hoc Networks, 2007, 5(1): 24–34.
Y. Zeng, R. Zhang, T. J. Lim. Wireless communications with unmanned aerial vehicles: opportunities and challenges [J]. IEEE Communications Magazine, 2016, 54(5): 36–42.
B. Wang, Y. Wu, K. Liu, et al. An anti-jamming stochastic game for cognitive radio networks [J]. IEEE Journal on Selected Areas in Communications, 2011, 29(4): 877–889.
C. Zhang, W. Zhang. Spectrum sharing for drone networks [J]. IEEE Journal on Selected Areas in Communications, 2017, 35(1): 136–144.
L. Xiao, Y. Li, J. Liu, et al. Power control with reinforcement learning in cooperative cognitive radio networks against jamming [J]. The Journal of Supercomputing, 2015, 71(9): 3237–3257.
L. Xiao, D. Jiang, X. Wan, et al. Anti-jamming underwater transmission with mobility and learning [J]. IEEE Communications Letters, 2018, 22(3): 542–545.
L. Xiao, C. Xie, M. Min, et al. User-centric view of unmanned aerial vehicle transmission against smart attacks [J]. IEEE Transactions on Vehicular Technology, 2018, 67(4): 3420–3430.
L. Xiao, X. Lu, D. Xu, et al. UAV relay in VANETs against smart jamming with reinforcement learning [J]. IEEE Transactions on Vehicular Technology, 2018, 67(5): 4087–4097.
V. Mnih, K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning [J]. Nature, 2015, 518(7540): 529–533.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the National Natural Science Foundation of China (Nos. 61671396 and 91638204). The associate editor coordinating the review of this paper and approving it for publication was X. Cheng.
Geyi Sheng received her B.S. degree in communication engineering from Xiamen University, Xiamen, China, in 2017, where she is currently pursuing her M.S. degree in the Department of Communication Engineering. Her research interests include network security and wireless communications.
Minghui Min received her B.S. degree in automation from Qufu Normal University, Rizhao, China, in 2013, and her M.S. degree in control theory and control engineering from Shenyang Ligong University in joint training with Shenyang Institute of Automation, Chinese Academy of Sciences, Shengyang, China, in 2016. She is currently pursuing her Ph.D. degree in the Department of Communication Engineering, Xiamen University, Xiamen, China. Her research interests include network security and wireless communications.
Liang Xiao [corresponding author] (M’09, SM’13) is currently a Professor in the Department of Communication Engineering, Xiamen University, Fujian, China. She has served in several editorial roles, including an associate editor of IEEE Trans. Information Forensics & Security and IET Communications. Her research interests include wireless security, smart grids, and wireless communications. She won the best paper award for 2016 IEEE INFOCOM Big security WS. She received her B.S. degree in communication engineering from Nanjing University of Posts and Telecommunications, China, in 2000, her M.S. degree in electrical engineering from Tsinghua University, China, in 2003, and her Ph.D. degree in electrical engineering from Rutgers University, NJ, in 2009. She was a visiting professor in Princeton University, Virginia Tech, and the University of Maryland, College Park. She is a senior member of the IEEE.
Sicong Liu (S15-M17) received his B.S.E. and his Ph.D. degree, both in electronic engineering, from Tsinghua University, Beijing, China in 2012 and 2017 (with the highest honor). From 2010 to 2011, he was a visiting scholar in the City University of Hong Kong, China. From 2017 to 2018, he served as a senior research engineer in Huawei Technologies Co., Ltd. Currently, he is an assistant professor in the Department of Communications Engineering, School of Information Science and Technology, Xiamen University, China. Sicong Liu has published over 35 journal and conference research papers. He owns 7 Chinese invention patents. He is one of the core members that draft the Broadband Power Line Communications Standard in China. He has won the Best Doctoral Dissertation Award of Tsinghua University. He is a reviewer of many top journals and has served as the guest editor of the Future Internet Journal and a TPC chair/member of IEEE ICC, IEEE SmartGridComm, and several international conferences. His research interests lie in sparse signal processing, interference mitigation, wireless communications, network security, and machine learning.
Rights and permissions
About this article
Cite this article
Sheng, G., Min, M., Xiao, L. et al. Reinforcement Learning-Based Control for Unmanned Aerial Vehicles. J. Commun. Inf. Netw. 3, 39–48 (2018). https://doi.org/10.1007/s41650-018-0029-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41650-018-0029-y