Elsevier

Neurocomputing

Volume 131, 5 May 2014, Pages 179-190
Neurocomputing

Hierarchical incorporation of shape and shape dynamics for flying bird detection

https://doi.org/10.1016/j.neucom.2013.10.026Get rights and content

Abstract

Flying bird detection (FBD) is critical in avoiding bird–aircraft collisions. Most existing approaches rely on motion detection to identify the flying bird, since it is a typical moving object. However, when there exist other moving objects, those methods often fail to distinguish flying birds from those objects due to the insufficiency of feature description. In this paper, we introduce a novel hierarchical feature model exploiting shape and shape dynamics to improve the ability of representing a flying bird, and then apply it to the FBD problem. As the shape of a flying bird is very distinctive in geometric structures and could provide discriminating spatial information, an improved shape context feature descriptor is proposed at the lower level to capture the spatial relations in bird shape. Then the shape descriptor is extended into the spatio-temporal domain and a shape dynamics description is built at the higher level, in which a 4-state Markovian model is adopted and is learned from training sequences. Moreover, to build a mapping from the lower level to the higher level of the hierarchy, a shape similarity index (SSI) based matching mechanism is designed. We apply these two-level features for detecting flying bird for improved safety of aircrafts flying at low-altitude. The experimental results show that the proposed method is effective and outperforms three other existing vision-based FBD approaches.

Introduction

Safety of aircrafts flying at low-altitude is important for a variety of tasks, such as air rescue, general aviation activities, airport terminal operations [1], etc. It is a challenging issue due to the various potentially hazardous objects. Especially, the moving hazardous objects (e.g. free-flying wildlife, model aircrafts, lanterns, balloons, kites, etc.) pose significant collision risks to low-flying aircrafts. Whereas these different hazards present different collision risks, the ability to detect and even to fine-classify them would be generally required for intelligent avoidance [2]. Statistics [3] disclose that bird–aircraft collisions constitute one of the highest risks to aircrafts, which makes flying bird detection (FBD) most necessary in our application context. Moreover, distinguishing birds from other dangerous objects and then providing more bird specific information for analysis would be also useful for further bird hazard management.

Unlike a generic object detection problem, the FBD task for low-flying aircrafts safety context has some of its own difficulties. It is a challenging task due to characteristics exhibited by the birds, for example, the available features such as size, color or texture are relatively weak for a bird, and its appearance and flight trajectory are also highly changeable while flying. The challenge is even greater when it should apply to low-altitude airspace environments, where the scene contains varied unpredictable yet potential bird hazards and requires faster processing. On the other hand, in practice, decreasing the distance from the aircraft to birds leads to a significantly increasing risk of bird–aircraft collisions. To give aircraft sufficient time to react and avoid strikes, it is necessary to identify the presence of birds flying in appropriate ranges. Therefore, this paper focuses on the rapid and reliable detection of bird flying near to the aircraft, specifically aiming to provide improved detectability of bird from other moving or look-alike objects in airspace.

Recent researches have increasingly resorted to various spectrum sensors (e.g. acoustic [4], [5], [6], microwave [6], [7], [8], [9], [10], visible [11], [12], [13], [14], [15], [16], [17], [18], [19], infrared [20], [21], [22], etc.) to enhance the detectability of flying birds. Especially, avian radar [8] is the current most popular technique due to its advantages of continuous, all-weather, 360 azimuth and wide-area surveillance. However, its intrinsic insufficiency limits the ability to sense birds flying close to the ground or in near range from radar source [11]. In addition, limitations with respect to resolution, cost-effectiveness and operating challenges also constrain its practical widely applications. Vision system is more affordable, maintainable, and adaptable for use, and hence is recommended to best supplement avian radar studies [11], [21].

Visual observation and analysis of birds and other small flying animals (e.g. bats, bees, etc.) using computer-vision methods have been an increasingly active research direction. To date, various vision-based techniques have been applied to detect flying birds. Motion-detection is a most commonly used method, since flying bird is a typical moving object. Examples include temporal differencing [11], [12], Gaussian mixture model (GMM) [12] or median based background subtraction [13], and optical flow analysis [14], [15]. However, these methods ignore the discriminating features of birds from other moving objects, prone to leading higher false positives in practical conditions. Hence, additional methods that further utilize bird-specific features (e.g. contour size [15], body length [16], [17], [18], shape feature [19], [23], Haar-like [24], [25], etc.) have been developed to achieve higher detection accuracy. While in many cases, where the scene contains other bird-like objects (e.g. low-flying airplanes, model aircrafts, kites or other unpredictable potentially hazards to aircrafts), they might be still inadequate for reliable detection. Therefore, the insufficiency of the existing feature descriptions for flying birds inspires us to introduce more unique information for improved representation ability in our context.

Meanwhile, research studies on the other small flying animals have also received much attention in recent years. For example, Margit et al. [26], [27], [28], [29] has worked on using thermal infrared imaging for bats study, in which the shape context feature of a flying bat have been utilized [28]. However, our work will advance the mere shape context description for FBD by using bird shape dynamics. Veeraraghavan et al. [30] has studied the dancing bees, for which each honeybee behavior is modeled as a Markov process based on the low-level motion states like hover, turning, waggling and motionless. Unlike bees, which often execute different dancing motions to communicate, the flying bird rarely switches between those motions (e.g. gliding, soaring, circling, level-flight, etc.) during a normal flight, and hence directly applying the proposed bee's model description might not be suitable for flying bird. But inspired by their work, a Markovian model can be used to describe the shape dynamics of flying birds, where we define low-level states corresponding to the various flying-pose shapes, like up-wing and down-wing.

In this paper, we propose a hierarchical structure that utilizes the spatial and spatio-temporal information contained in bird shapes while bird flying, and then apply it to the FBD problem.

To establish the FBD problem in our context, we begin with the following basic assumptions:

  • (1)

    Capturing videos using static cameras. This is because a stable platform provides a relatively easier detection conditions, and facilitates a long-term bird characteristics statistics study (e.g. flight, feeding, migration characteristics, etc.) for further ornithological survey, thus it is preferred in most surveillance applications.

  • (2)

    Limiting to one foreground object in a video segment. If there are multiple objects in the scene, we assume each has been isolated out using available increasingly advanced multiple target tracking techniques. It is usually feasible in many real applications. For example, in aerodrome environments, single bird is frequently observed due to bird control at airports.

  • (3)

    Constraining the bird motion. In the given video, birds should be in regular motion (e.g. flapping, bounding) but excludes gliding all the time. This is due to the fact that a bird usually purely glides at high-altitude, which rarely occurs in the low-altitude scenes in our context.

The input of the problem is a sequence of video frames that contain a moving object in low-altitude airspace. The output is to determine whether the object is the flying-bird or not for the current frame.

The contributions of this paper are three-fold:

  • (1)

    To achieve robust feature representation for flying birds, a hierarchical feature model that describes two-level features including shape and shape dynamics is introduced. Compared with other moving objects, the shape of a flying bird is very distinctive in geometric structures. It is composed of some key body parts such as the bill, the head, the wings, and the tail. The spatial relationship among these parts provides discriminative spatial information. Hence, at the lower level of the proposed model, a shape context based feature descriptor is introduced to capture the spatial relations in bird body shape. Meanwhile, when analyzing temporally, the bird shape varies with time and its change follows similar dynamics as mentioned in [30], [31]. Hence, we further extend the shape description into the spatio-temporal domain and then build a shape dynamics based representation at the higher level of our hierarchical framework. For modeling its time-variable characteristics while bird flying, the Markovian model is adopted and is learned from training sequences.

  • (2)

    To build a mapping from the lower level to the higher level embedded in our hierarchical feature model, a shape similarity index (SSI) based matching mechanism is designed. Based on the measured SSI, each shape in the lower level can be matched to a pre-defined key state. By replacing each shape with its matched state, the higher level dynamics contained in shape sequence can be interpreted as state change to achieve a better description for flying birds.

  • (3)

    To make rapid and reliable detection of a flying bird, a coarse-to-fine detection method is introduced, which exploits the hierarchical feature model. In the coarse level, to get higher speed, only the lower-level shape context feature is utilized to eliminate obvious non-bird objects for a single frame. In the fine level, to increase the accuracy, the higher-level shape dynamics contained in previous multi-frames is considered to help determining for the current frame.

The remainder of this paper is organized as follows. Section 2 introduces the proposed hierarchical feature model for representing the flying bird. Section 3 describes the proposed FBD method in details. Section 4 presents and analyzes the experimental results, and finally we give the conclusion in Section 5.

Section snippets

Hierarchical feature model for flying bird

In this section, a hierarchical feature model for representing the flying bird is introduced as shown in Fig. 1. The higher level of our hierarchy contains the shape dynamics, and the lower level is about the shape image. Moreover, to characterize the mapping from the lower level to the higher level, a SSI based matching mechanism is designed. Here, we will describe these two levels of features and their SSI-based mapping in detail, as following subsections.

Coarse-to-fine flying bird detection

Exploiting the hierarchical feature model presented in Section 2, a coarse-to-fine fashion for FBD problem is proposed.

As shown in Fig. 5, the proposed method can be divided into the learning phase and the detection phase. In the learning phase, the two-level features including shape and shape dynamics from training sequences are analyzed. In the detection phase, we first perform a pre-processing step to extract the moving foreground in the scene. The input video sequence is processed into a

Experiments

In this section, we describe a set of experiments performed to compare the detection performance of the proposed method as described in Section 3 and the other existing FBD approaches. For the sake of better comparison, the performance evaluation criteria adopted in our work is presented first, and then the parameters setting and comparison results are shown.

Conclusions and future work

In this paper, we present a method for detection of low-flying birds. The method is based on a novel hierarchical feature model, which describes two levels of flying bird features incorporating shape and shape dynamics. An improved Shape context feature is used to describe the shape of a flying bird. A template set of shape context features is constructed using 4 key poses of the flying bird, where one shape context feature represents one pose. The dynamics of bird shape is described by a

Acknowledgements

This work was supported in part by the National Basic Research Program of China (973 Program) under Grant 2011CB707000, the Foundation for Innovative Research Groups of National Natural Science Foundation (Grant No.61221061), and the National Key Technology R&D Program of China (Grant No.2012BAG04B01).

Jun Zhang received the B.S. degree in 1987, the M.S. degree in 1991, and the Ph.D. degree in 2001, from Beihang University, Beijing, PR China. He is currently a Professor and the Director of National Key Lab of CNS/ATM CAAC. His current research interests include intelligent transportation systems, integrated networks of air/space/ground for air traffic management, and air-ground collaborative airspace surveillance. He has been publishing more than 100 books, papers, and patents in these fields.

References (42)

  • P. Plonczkier et al.

    Radar monitoring of migrating pink-footed geese: behavioural responses to offshore wind farm development

    J. Appl. Ecol.

    (2012)
  • Q. Zhang, Avian Detection and Monitoring using Frequency-stepped Chirp Signal Radar, in: Piers...
  • Y. Murai, Y. Takeda, Y. Tasaka, Detection of Birds Flying in the Aerodrome by Image Analysis, in: Proceedings of the...
  • D.J Shah et al.

    Motion based bird sensing using frame differencing and Gaussian mixture

    UCR Undergraduate Res. J.

    (2008)
  • W.W. Verstraeten et al.

    Webcams for bird detection and monitoring: a demonstration study

    Sensors

    (2010)
  • Y. Li, Bird objects detection and tracking on the wild field circumstance, in: Proceedings of the WRI World Congress in...
  • X.-Y. Zhang, X.-J. Wu, X. Zhou, X.-G. Wang, Y.-Y. Zhang, Automatic detection and tracking of maneuverable birds in...
  • D.-Z. Song, N. Qin, Y.-L. Xu, Chang-Yong Kim, D. Luneau, K. Goldberg, Systems and algorithms for an autonomous...
  • D.-Z. Song, Y.-L. Xu, Monocular Vision-based Detection of a Flying Bird, Technical Report, Texas A&M University, USA,...
  • D.-Z. Song et al.

    Negative filter for detecting rare bird species from short video segments using a probable observation data set-based EKF method

    IEEE Trans. Image Process

    (2010)
  • M. Desholm, Thermal Animal Detection System (TADS): Development of a Method for Estimating Collision Frequency of...
  • Cited by (17)

    • Multileveled ternary pattern and iterative ReliefF based bird sound classification

      2021, Applied Acoustics
      Citation Excerpt :

      Generally, handcrafted features based machine learning and deep learning models have been used to classify bird sounds automatically [12–16]. Collection and analysis of bird sounds are used for different purposes such as environmental monitoring, bird species detection, target bird follow-up [17,18]. The error rate of computer-aided classification systems is low [19],.

    • A new skeleton based flying bird detection method for low-altitude air traffic management

      2018, Chinese Journal of Aeronautics
      Citation Excerpt :

      To facilitate the following concise descriptions, these five methods are shortened as BSPT, WBP, HISSD, HLAB and HCLS method. For BSPT, WBP and HISSD method, we directly adopt the default algorithm from their work10,11,28 for testing. In order to equally compare our method with the classification-based methods HLAB and HCLS, we use the same flying bird training dataset as detailed in Section 3.1, except for the fact that the dataset preprocessing mentioned in method25 is also conducted to extract region of interests of birds from images using the bounding box information available in the dataset.

    • A novel robust hybrid gravitational search algorithm for reusable launch vehicle approach and landing trajectory optimization

      2015, Neurocomputing
      Citation Excerpt :

      In order to meet the increasing demand for space utilization, reusable launch vehicle (RLV) is developed to provide the potential way to improve space vehicle economics. Since the approach and landing (A&L) is the last phase of the RLV’s returning to the earth [1], it is a critical flight phase that brings the unpowered RLV from the end of the terminal area energy management (TAEM) flight to the runway touchdown (TD), which makes the safety to be the primary and enforcing requirement [2,3]. The present literatures imposed most efforts on the development of new approaches that could deliver a RLV to its landing TD site safely and reliably [1,4,5].

    View all citing articles on Scopus

    Jun Zhang received the B.S. degree in 1987, the M.S. degree in 1991, and the Ph.D. degree in 2001, from Beihang University, Beijing, PR China. He is currently a Professor and the Director of National Key Lab of CNS/ATM CAAC. His current research interests include intelligent transportation systems, integrated networks of air/space/ground for air traffic management, and air-ground collaborative airspace surveillance. He has been publishing more than 100 books, papers, and patents in these fields.

    Qunyu Xu received the B.S. degree in Electronics and Information Engineering from Anhui University, Hefei, PR China, in 2007. Currently, she is pursuing the Ph.D. degree in School of Electronics and Information Engineering, Beihang University, Beijing, PR China. Her current research interests include image processing, computer vision, pattern recognition, machine learning, and their applications to detecting aviation hazardous objects for improved safety.

    Xianbin Cao is a professor in the school of electronic and information engineering, Beihang University, Beijing, PR China and is the director of the lab of intelligent transportation system. His current research interests include intelligent transportation systems, airspace transportation management and intelligent computation.

    Pingkun Yan is a full professor with the Center for OPTical IMagery Analysis and Learning (OPTIMAL), State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, Shaanxi, P. R. China. His research interests include computer vision, pattern recognition, machine learning, and their applications.

    Xuelong Li is a full professor with the Center for OPTical IMagery Analysis and Learning (OPTIMAL), State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, Shaanxi, P. R. China.

    View full text