Skip to main content
SpringerLink
Log in

Realtime multi-aircraft tracking in aerial scene with deep orientation network

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Tracking the aircrafts from an aerial view is very challenging due to large appearance, perspective angle, and orientation variations. The deep-patch orientation network (DON) method was proposed for the multi-ground target tracking system, which is general and can learn the target’s orientation based on the structure information in the training samples. Such approach leverages the performance of tracking-by-detection framework into two aspects: one is to improve the detectability of the targets by using the patch-based model for the target localization in the detection component and the other is to enhance motion characteristics of the individual tracks by incorporating the orientation information as an association metric in the tracking component. Based on the DON structure, you only look once (YOLO) and faster region convolutional neural network (FrRCNN) detection frameworks with simple online and realtime tracking (SORT) tracker are utilized as a case study. The Comparative experiments demonstrate that the overall detection accuracy is improved at the same processing speed of both detection frameworks. Furthermore, the number of Identity switches (IDsw) has reduced about 67% without affecting the computational complexity of the tracking component. Consequently, the presented method is efficient for realtime ground target-tracking scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP J. Image Video Process. 2008(1), 246309 (2008)

    Google Scholar 

  2. Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: Image processing (ICIP), 2016 IEEE international conference on, IEEE, pp. 3464–3468 (2016)

  3. Bian, X., Chen, C., Tian, L., Du, Q.: Fusing local and global features for high-resolution scene classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 10(6), 2889–2901 (2017)

    Article  Google Scholar 

  4. Biresaw, T.A., Nawaz, T., Ferryman, J., Dell, A.I.: Vitbat: Video tracking and behavior annotation tool. In: Advanced video and signal based surveillance (AVSS), 2016 13th IEEE international conference on, IEEE, pp. 295–301 (2016)

  5. Bora, M., Jyoti, D., Gupta, D., Kumar, A.: Effect of different distance measures on the performance of k-means algorithm: an experimental study in matlab (2014). arXiv preprint arXiv:1405.7471

  6. Cheng, G., Yang, C., Yao, X., Guo, L., Han, J.: When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative CNNS. In: IEEE transactions on geoscience and remote sensing (2018)

  7. Cheng, M.M., Zhang, Z., Lin, W.Y., Torr, P.: Bing: Binarized normed gradients for objectness estimation at 300 fps, Computer Vision and Pattern Recognition. IEEE, pp. 3286–3293 (2014)

  8. Cheng, G., Zhou, P., Han, J.: Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images. IEEE Trans. Geosci. Remote Sens. 54(12), 7405–7415 (2016)

    Article  Google Scholar 

  9. Cheng, G., Li, Z., Yao, X., Guo, L., Wei, Z.: Remote sensing image scene classification using bag of convolutional features. IEEE Geosci. Remote Sens. Lett. 14(10), 1735–1739 (2017)

    Article  Google Scholar 

  10. Choi, W.: Near-online multi-target tracking with aggregated local flow descriptor. In: Proceedings of the IEEE international conference on computer vision, pp. 3029–3037 (2015)

  11. Dat (2014) http://www.ucassdl.cn/resource.asp. Accessed Jun 2014

  12. Dat (2015) https://github.com/wuhuiIOS/AircraftsDataset. Accessed 14 Jan 2015

  13. Dat (2018) https://github.com/bczhangbczhang/Airport-Dataset. Accessed 27 Feb 2018

  14. Dicle, C., Camps, O.I., Sznaier, M.: The way they move: tracking multiple targets with similar appearance. In: Proceedings of the IEEE international conference on computer vision, pp. 2304–2311 (2013)

  15. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  16. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587 (2014)

  17. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp. 1440–1448 (2015)

  18. Han, J., Zhang, D., Cheng, G., Liu, N., Xu, D.: Advanced deep-learning techniques for salient and category-specific object detection: a survey. IEEE Signal Process. Mag. 35(1), 84–100 (2018)

    Article  Google Scholar 

  19. Hou, R., Chen, C., Shah, M.: Tube convolutional neural network (t-cnn) for action detection in videos. In: IEEE international conference on computer vision (2017)

  20. Kalman, R.E., et al.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)

    Article  Google Scholar 

  21. Kim, C., Li, F., Ciptadi, A., Rehg, J.M.: Multiple hypothesis tracking revisited. In: Proceedings of the IEEE international conference on computer vision, pp. 4696–4704 (2015)

  22. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)

  23. Kuhn, H.W.: The Hungarian method for the assignment problem. 50 years of integer programming 1958–2008, pp. 29–47 (2010)

    Google Scholar 

  24. Li, Y., Huang, C., Nevatia, R.: Learning to associate: Hybridboosted multi-target tracker for crowded scene. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. In: IEEE conference on, IEEE, pp. 2953–2960 (2009)

  25. Li, W., Xiang, S., Wang, H., Pan, C.: Robust airplane detection in satellite images. In: IEEE international conference on image processing, ICIP 2011, pp. 2821–2824. Belgium, September, Brussels (2011)

  26. Liu, G., Sun, X., Fu, K., Wang, H.: Aircraft recognition in high-resolution satellite images using coarse-to-fine shape prior. IEEE Geosci. Remote Sens. Lett. 10(3), 573–577 (2013)

    Article  Google Scholar 

  27. Maher, A., Gu, J., Zhang, B.: Deep-patch orientation network for aircraft detection in aerial images. In: Chinese conference on image and graphics technologies, Springer, pp. 178–188 (2017)

  28. Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: Mot16: A benchmark for multi-object tracking (2016). arXiv preprint arXiv:1603.00831

  29. Perera, A.A., Srinivas, C., Hoogs, A., Brooksby, G., Hu, W.: Multi-object tracking through simultaneous long occlusions and split-merge conditions. In: Computer vision and pattern recognition, 2006 IEEE computer society conference on, IEEE, vol 1, pp. 666–673 (2006)

  30. Razakarivony, S., Jurie, F.: Discriminative autoencoders for small targets detection. In: IAPR international conference on pattern recognition, pp. 3528–3533 (2014)

  31. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection (2015). arXiv preprint arXiv:1506.02640

  32. Redmon, J., Farhadi, A.: Yolo9000: Better, faster, stronger (2016). arXiv preprint arXiv:1612.08242

  33. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp. 91–99 (2016)

  34. Rezatofighi, S.H., Milan, A., Zhang, Z., Shi, Q., Dick, A., Reid, I.: Joint probabilistic data association revisited. In: Proceedings of the IEEE international conference on computer vision, pp. 3047–3055 (2015)

  35. Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J., Mostefa, D., Soundararajan, P.: The clear 2006 evaluation. In: International evaluation workshop on classification of events, activities and relationships, pp. 1–44. Springer (2006)

  36. Sun, H., Sun, X., Wang, H., Li, Y., Li, X.: Automatic target detection in high-resolution remote sensing images using spatial sparse coding bag-of-words model. IEEE Geosci. Remote Sens. Lett. 9(1), 109–113 (2011)

    Article  Google Scholar 

  37. Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric (2017). arXiv preprint arXiv:1703.07402

  38. Wu, H., Zhang, H., Zhang, J., Xu, F.: Fast aircraft detection in satellite images based on convolutional neural networks. In: IEEE international conference on image processing, pp. 4210–4214 (2015)

  39. Xiang, Y., Alahi, A., Savarese, S. Learning to track: Online multi-object tracking by decision making. In: Proceedings of the IEEE international conference on computer vision, pp. 4705–4713 (2015)

  40. Yao, X., Han, J., Cheng, G., Qian, X., Guo, L.: Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans. Geosci. Remote Sens. 54(6), 3660–3671 (2016)

    Article  Google Scholar 

  41. Yao, X., Han, J., Zhang, D., Nie, F.: Revisiting co-saliency detection: a novel approach based on two-stage multi-view spectral rotation co-clustering. IEEE Trans. Image Process. 26(7), 3196–3209 (2017)

    Article  MathSciNet  Google Scholar 

  42. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European conference on computer vision, Springer, pp. 818–833 (2014)

  43. Zhu, H., Chen, X., Dai, W., Fu, K., Ye, Q., Jiao, J.: Orientation robust object detection in aerial images using deep convolutional neural network. In: Image processing (ICIP), 2015 IEEE international conference on, IEEE, pp. 3735–3739 (2015)

  44. Zhang, L., Lin, L., Liang, X., He, K.: Is faster r-cnn doing well for pedestrian detection? In: European Conference on computer vision, Springer, pp. 443–457 (2016b)

  45. Zhang, B., Perina, A., Li, Z., Murino, V., Liu, J., Ji, R.: Bounding multiple gaussians uncertainty with application to object tracking. Int. J. Comput. Vis. 118(3), 364–379 (2016a)

    Article  MathSciNet  Google Scholar 

  46. Zhang, B., Gu, J., Chen, C., Han, J., Su, X., Cao, X., Liu, J.: One-two-one networks for compression artifacts reduction in remote sensing. ISPRS J. Photogramm. Remote Sens. (2018a). https://doi.org/10.1016/j.isprsjprs.2018.01.003

    Article  Google Scholar 

  47. Zhang, B., Luan, S., Chen, C., Han, J., Wang, W., Perina, A., Shao, L.: Latent constrained correlation filter. IEEE Trans. Image Process. 27(3), 1038–1048 (2018b)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The work was supported by the Natural Science Foundation of China under Contract 61672079 and 61473086, and Shenzhen Peacock Plan KQTD2016112515134654. This work is supported by the Open Projects Program of National Laboratory of Pattern Recognition. This work was supported by the National Basic Research Program of China (2015CB352501). Baochang Zhang is also with Shenzhen Academy of Aerospace Technology, Shenzhen, China. Baochang Zhang is the corresponding author.

Author information

Authors and Affiliations

  1. School of Automation Science and Electrical Engineer, Beihang University, Beijing, China

    Ali Maher & Baochang Zhang

  2. Military Technical College, Cairo, Egypt

    Hussein Taha

Authors
  1. Ali Maher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hussein Taha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baochang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baochang Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maher, A., Taha, H. & Zhang, B. Realtime multi-aircraft tracking in aerial scene with deep orientation network. J Real-Time Image Proc 15, 495–507 (2018). https://doi.org/10.1007/s11554-018-0780-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-018-0780-1

Keywords

Search

Navigation