Elsevier

Pattern Recognition

Volume 108, December 2020, 107531
Pattern Recognition

HCNN-PSI: A hybrid CNN with partial semantic information for space target recognition

https://doi.org/10.1016/j.patcog.2020.107531Get rights and content

Highlights

  • We use data augmentation techniques and image processing to represent the wide image of space target in the deep space background.

  • We propose a new localization method improved from minimum bounding rectangle (MBR) model.

  • We propose a hybrid network that can integrate multi-source information which can get a better recognition performance than that who only utilize single information.

Abstract

Space target recognition is the basic task of space situational awareness and has developed significantly in the last decade. This paper proposes a hybrid convolutional neural network with partial semantic information for space target recognition, which joints the global features and partial semantic information. Firstly, we propose a two-stage target detection network based on the characteristics of deep space targets. Secondly, we use the Mask R-CNN to segment the main components of the detected satellite. Thirdly, the recognized target and the segmented components are sent to the hybrid extractor to train the hybrid network. What we have done is to find the proper weights of the partial semantic information that plays different importance. The loss function of the hybrid network integrates the global-based and component-based loss with different weights. In comparison with several sets of comparative experiments, the proposed method has achieved a satisfactory result. Besides, we have simulated some real space target images by data processing and achieved a competitive performance in both the simulated dataset and the public dataset.

Introduction

As humans continue to expand their exploration of outer space, space exploration technology has received more and more attention. However, a large number of techniques related to space exploration are generally based on radar data or remote sensing data [1]. With fast development of optical devices, a large number of optical space images are now available daily through sensors on the ground or high-altitude satellites. How to make use of these data to promote space exploration technology is a question worth discussing.

Space situational awareness (SSA) system [2] mainly includes the detection, tracking, identification of space targets, as well as the assessment, verification, environmental monitoring and forecasting of space events. It is an important cornerstone for dealing with space threats and ensuring space security,while space target recognition is the basic task of SSA. Because outer space is so vast, optical sensors can capture relatively little information. Coupled with the huge noise from various kinds of reflected lights, how to extract the features that we need from the limited information is a tricky and difficult problem.

With the rapid development of artificial intelligence technology, there seems to be a general solution for image recognition of large samples. Even if the sample size is insufficient, the dataset can be augmented by some data augmentation methods [3], [4]. The increase in the number and investment of satellites has resulted in considerable space target imagery available, while the advanced technologies have not yet followed up. Traditional manual interpretation methods bring huge manpower burden and time consumption. At the same time, subjective errors due to visual fatigue could also seriously decrease the efficiency of image analysis. Furthermore, existing methods [5] apply artificial designed feature filters, which are difficult to adapt to all types of on-orbit images and bring out poor mobility.

There already exists several space target recognition methods [6] achieving favorable performance, which only gave the identification results without giving the relevant coordinate information. For high-resolution images with large targets, it is efficient to use one-stage way to meet identification requirements. But a space target image in real scene is characterized by a wide-area image and a very small target. So the optimal way is to use the two-stage method of locating and classifying to complete the space target recognition task in the wide field background.

Another problem worth thinking is that how to make good use of the priori knowledge of space objects for small target recognition [7]. As we know, many type of space targets are physically similar, with basic components including the star body, solar wings and antennas. In fact, each part of the semantic information plays a different role in determining the category of the target. The parts with the most diverse features tend to determine the category more. Therefore, how to assign different weights to the semantic information of different components to improve the category-determining function of the features with large differences is the key problem.

In this paper, we propose a hybrid CNN with partial semantic information (HCNN-PSI) for deep space target recognition. Target locating, component segmentation, and multi-source input recognition in wide-area images are progressively achieved through three main steps. Firstly, we detect the suspicious target from the deep space image and then send it to the rough identification network. In this part, we propose a two-stage target detection based on the characteristic of deep space image. In the second step, the selected space targets are sent to the segmentation network for component segmentation and the main information of each component is obtained. Thirdly, the deep space target and its main part are both sent to the fine identification network to identify the model of the satellite. In this section, we propose a network structure that can integrate global semantic information and local semantic information, whose function is to strengthen the weight of the decisive part.

In general, our contributions are summarized as follows. (1) We use data augmentation techniques and image processing to represent the wide image of space target in the deep space background. (2) This paper proposes a new localization method improved from minimum bounding rectangle (MBR) model, which can be effectively applied to two-stage target detection method with relatively simple background. (3) We propose a hybrid network that can integrate multi-source information which can get a better recognition performance than that who only utilize single information. Besides, heat map is used as a weight basis.

The remainder of this paper is organized as follows. Section 1 details space recognition in wide-area images, Section 2 investigates the related work about space target recognition. Section 3 gives the introduction about the proposed network. Section 4 presents the experimental results and makes a discussion. Finally, Section 4 concludes the paper.

Section snippets

Deep convolutional neural network

The arrival of the information age first brought about information explosion, and the massive data growth is amazing. Big data has also brought the rapid development of deep learning technology. In fact, the perceptron model [8] has long been proven to handle linear classification problems, but limited by the computer science at the time and the lack of relevant mathematical theory support, until now people have evoked a passion for deep learning technology.

Since AlexNet [9] won the ImageNet

Overall network framework

The architecture of the proposed Hybrid CNN with partial semantic information network structure is shown in Fig. 1. As we can see, the network is divided into three stages: deep space target detection module, component segmentation module and fine-grained recognition module. The first stage is responsible for positioning and identifying the satellite in the deep space image. The second stage is designed to divide the satellite into solar wings and satellite body. The third stage is used to

Experiments

To prove the effectiveness of the proposed method, we conduct extensive experiments to verify recognition accuracy and time consuming. All experiments are conducted on a computer with a GTX1080Ti GPU and an Intel Xeon E5-2609 CPU.

Conclusion

In reality, deep space targets image exhibits the characteristics of low pixel, simple background, and strong noise. Existing space target detection technologies are aimed at high resolution large images, which will bring out lower accuracy and more computation complexity.

In this paper, we propose a Hybrid CNN with partial semantic information for space target recognition. Firstly, We propose MBRT that improves the performance of locating, then apply CNN to extract deeper semantic information

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant nos. 61976166, 61772402, 61671339, U1605252, 61772387, 61922066, 61876142, and 61901112, in part by the National Key Research and Development Program of China under Grant nos. 2016QY01W0200 and 2018AAA0103202, in part by the National High-Level Talents Special Support Program of China under Grant CS31117200001, in part by the Innovation Capacity Support Plan of Shaanxi Province under Grant

Xi Yang received the B.Eng. degree in electronic information engineering and the Ph.D. degree in pattern recognition and intelligence system from Xidian University, Xi’an, China, in 2010 and 2015, respectively. From 2013 to 2014, she was a visiting Ph.D. student with the Department of Computer Science, University of Texas at San Antonio, San Antonio, TX, USA. In 2015, she joined the State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian

References (37)

  • L. Perez, J. Wang, The effectiveness of data augmentation in image classification using deep learning, arXiv:1712.04621...
  • T. Ye et al.

    Autonomous space target recognition and tracking approach using star sensors based on a Kalman filter

    Appl. Opt.

    (2015)
  • H. Zeng et al.

    Space target recognition based on deep learning

    IEEE International Conference on Information Fusion (Fusion)

    (2017)
  • F. Rosenblatt

    The perceptron: a probabilistic model for information storage and organization in the brain.

    Psychol. Rev.

    (1958)
  • A. Krizhevsky et al.

    Imagenet classification with deep convolutional neural networks

    Advances in Neural Information Processing Systems

    (2012)
  • K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556...
  • C. Szegedy et al.

    Going deeper with convolutions

    IEEE Conference on Computer Vision and Pattern Recognition

    (2015)
  • K. He et al.

    Deep residual learning for image recognition

    IEEE Conference on Computer Vision and Pattern Recognition

    (2016)
  • Cited by (0)

    Xi Yang received the B.Eng. degree in electronic information engineering and the Ph.D. degree in pattern recognition and intelligence system from Xidian University, Xi’an, China, in 2010 and 2015, respectively. From 2013 to 2014, she was a visiting Ph.D. student with the Department of Computer Science, University of Texas at San Antonio, San Antonio, TX, USA. In 2015, she joined the State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, where she is currently an associate professor in communications and information systems. Her current research interests include image/video processing, computer vision and multimedia information retrieval.

    Tan Wu received the B.Eng. degree in communication engineering from Xidian University, Xi’an, China, in 2017, where he is currently pursuing the Master degree in Information and Communication Engineering. His current research interests include deep learning and computer vision.

    Nannan Wang received the B.Sc. degree in information and computation science from the Xian University of Posts and Telecommunications in 2009 and the Ph.D. degree in information and telecommunications engineering from Xidian University in 2015. From September 2011 to September 2013, he was a Visiting Ph.D. Student with the University of Technology, Sydney, NSW, Australia. He is currently a Professor with the State Key Laboratory of Integrated Services Networks, Xidian University. He has published over 90 articles in refereed journals and proceedings, including IEEE T-PAMI, IJCV, NeurIPS etc. His current research interests include computer vision, pattern recognition, and machine learning.

    Yan Huang received the B.S. degree in electrical engineering, and the Ph.D. degree in signal and information processing, both from Xidian University, Xian, China, in 2013 and 2018, respectively. He was studying as a visiting Ph.D. student in Electrical and Computer Engineering department at University of Florida from Sep. 2016 to July 2017, and in Electrical and Systems Engineering department at the Washington University in St. Louis from July 2017 to Aug. 2018. He is currently an assistant professor at the State Key Laboratory of Millimeter Waves, Southeast University. His research interests include machine learning, synthetic aperture radar, image processing, remote sensing.

    Bin Song received the BS, MS, and Ph.D. in communication and information systems from Xidian University, Xian, China, in 1996, 1999, and 2002, respectively. In 2002, he joined the School of Telecommunications Engineering, Xidian University, where he is currently a professor in communications and information systems. He is also the associate director of State Key Laboratory of Integrated Services Networks. He has authored over 50 journal papers or conference papers and 30 patents. His research interests and areas of publication include the video compression and transmission technologies, video transcoding, error- and packet-loss-resilient video coding, distributed video coding, and video signal processing based on compressed sensing, big data, and multimedia communications.

    Xinbo Gao received the B.Eng., M.Sc. and Ph.D. degrees in electronic engineering, signal and information processing from Xidian University, Xi'an, China, in 1994, 1997, and 1999, respectively. From 1997 to 1998, he was a research fellow at the Department of Computer Science, Shizuoka University, Shizuoka, Japan. From 2000 to 2001, he was a post-doctoral research fellow at the Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong. Since 2001, he has been at the School of Electronic Engineering, Xidian University. He is currently a Cheung Kong Professor of Ministry of Education of P. R. China, a Professor of Pattern Recognition and Intelligent System of Xidian University and a Professor of Computer Science and Technology of Chongqing University of Posts and Telecommunications. His current research interests include Image processing, computer vision, multimedia analysis, machine learning and pattern recognition. He has published six books and around 300 technical articles in refereed journals and proceedings. Prof. Gao is on the Editorial Boards of several journals, including Signal Processing (Elsevier) and Neurocomputing (Elsevier). He served as the General Chair/Co-Chair, Program Committee Chair/Co-Chair, or PC Member for around 30 major international conferences. He is a Fellow of the Institute of Engineering and Technology and a Fellow of the Chinese Institute of Electronics.

    View full text