Skip to main content
SpringerLink
Log in

Design and Optimization of Evaluation Metrics in Object Detection and Tracking for Low-Altitude Aerial Video

  • Conference paper
  • First Online:
Big Data and Security (ICBDS 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1210))

Included in the following conference series:

  • 1113 Accesses

Abstract

The combination of Unmanned Aerial Vehicle (UAV) technology and computer vision has become popular in a wide range of applications, such as surveillance and reconnaissance, while popular evaluation measures are sometimes not applicable for specific tasks. In order to evaluate visual object detection and tracking algorithms of low-altitude aerial video properly, we first summarize the evaluation basis of computer vision tasks, including ground truth, prediction-to-ground truth assignment strategy and distance measures between prediction and ground truth. Then, we analyze the advantages and disadvantages of visual object detection and tracking performance measures, including average precision (AP), F-measure, and accuracy. Finally, for the low-altitude (nearly 100 m) surveillance mission of small unmanned aerial vehicles, we discuss the threshold optimization method of popular measures and the design strategy of application measures. Our work provides a reference in the aspect of performance measures design for researchers of UAV vision.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cehovin, L., Leonardis, A., Kristan, M.: Visual object tracking performance measures revisited. IEEE Trans. Image Process. 25(3), 1261–1274 (2016)

    MathSciNet  MATH  Google Scholar 

  2. Everingham, M., Gool, L.V., Williams, C.K.I., et al.: The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  3. Lin, T.Y., Maire, M., Belongie, S., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele B., Tuytelaars, T. (eds.) Computer Vision–ECCV 2014, LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

  4. Kristan, M., et al.: The sixth visual object tracking VOT2018 challenge results. In: Leal-Taixé, L., Roth, S. (eds.) Computer Vision–ECCV 2018 Workshops, LNCS, vol. 11129, pp. 3–53. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11009-3_1

  5. Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1834–1848 (2015)

    Article  Google Scholar 

  6. Vidit, J., Erik, L.M.: FDDB: a benchmark for face detection in unconstrained settings. Technical Report UM-CS-2010-009, Department of Computer Science, University of Massachusetts, Amherst (2010)

    Google Scholar 

  7. Wojek, C., Dollar, P., Schiele, B., et al.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2012)

    Article  Google Scholar 

  8. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE, Providence (2012)

    Google Scholar 

  9. Baykara, H.C., Bıyık, E., Gül, G., et al.: Real-time detection, tracking and classification of multiple moving objects in UAV videos. In: 29th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 945–950. IEEE, Boston (2017)

    Google Scholar 

  10. Du, D.W., Qi, Y.K., Yu, H.Y., et al.: The unmanned aerial vehicle benchmark: object detection and tracking. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision–ECCV 2018. LNCS, vol. 11214, pp. 375–391. Springer, Cham (2018)

    Google Scholar 

  11. Mueller, M., Smith, N., Ghanem, B.: A benchmark and simulator for UAV tracking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision–ECCV 2016. LNCS, vol. 9905, pp. 445–461. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_27

  12. Zhang, P.Y., Zhong, Y.X., Li, X.Q.: SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications. CoRR abs/1907.11093 (2019)

    Google Scholar 

  13. Zhu, P.F., Wen, L.Y., Du, D.W., et al.: VisDrone-VDT2018: the vision meets drone video detection and tracking challenge results. In: Leal-Taixé, L., Roth, S. (eds.) Computer Vision–ECCV 2018 Workshops. LNCS, vol. 11133, pp. 496–518. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-11021-5_29

  14. Measuring Object Detection models-mAP-What is Mean Average Precision? https://towardsdatascience.com/what-is-map-understanding-the-statistic-of-choice-for-comparing-object-detection-models-1ea4f67a9dbd. Accessed 07 Oct 2019

  15. Smeulders, A.W.M., Chu, D.M., Cucchiara, R., et al.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2014)

    Article  Google Scholar 

  16. Zhang, K.H., Zhang, L., Yang, M.H.: Real-time compressive tracking. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer Vision–ECCV 2012. LNCS, vol. 7574, pp. 864–877. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_62

  17. Godec, M., Roth, P.M., Bischof, H.: Hough-based tracking of non-rigid objects. Comput. Vis. Image Underst. 117(10), 1245–1256 (2013)

    Article  Google Scholar 

  18. Kosub, S.: A note on the triangle inequality for the Jaccard distance. Pattern Recogn. Lett. 120, 36–38 (2019)

    Article  Google Scholar 

  19. Leal-Taixé, L., Milan, A., Reid, I.D., et al.: MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. CoRR, abs/1504.01942 (2015)

    Google Scholar 

  20. Kristan, M., et al.: The visual object tracking VOT2016 challenge results. In: Hua, G., Jegou, H. (eds.) Computer Vision – ECCV 2016 Workshops, LNCS, vol. 9914, pp. 777–823. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_54

  21. Zou, Z.X., Shi, Z.W., Guo, Y.H., et al.: Object Detection in 20 Years: A Survey. CoRR abs/1905.05055 (2019)

    Google Scholar 

  22. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893. IEEE, San Diego (2005)

    Google Scholar 

  23. Dollar, P., Wojek, C., Schiele, B., et al.: Pedestrian detection: a benchmark. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 304–311. IEEE, Miami (2009)

    Google Scholar 

  24. Everingham, M., Eslami, S.M.A., Gool, L.V., et al.: The Pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)

    Article  Google Scholar 

  25. Kisantal, M., Wojna, Z., Murawski, J., et al.: Augmentation for small object detection. CoRR abs/1902.07296 (2019)

    Google Scholar 

  26. Kristan, M., Pflugfelder R., Leonardis A., et al.: The visual object tracking VOT2013 challenge results. In: 2013 IEEE International Conference on Computer Vision Workshops, pp. 98–111. IEEE, Sydney (2013)

    Google Scholar 

  27. Wu, Y., Lim J., Yang, M.H.: Online object tracking: a benchmark. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2411–2418. IEEE, Portland (2013)

    Google Scholar 

  28. Foolwood Homepage. https://github.com/foolwood/benchmark_results. Accessed 07 Oct 2019

  29. Kristan, M., Matas, J., Leonardis, A., et al.: A novel performance evaluation methodology for single-target trackers. IEEE Trans. Pattern Anal. Mach. Intell. 38(11), 2137–2155 (2016)

    Article  Google Scholar 

  30. Kristan, M., Pflugfelder, R., Leonardis, A., et al.: The visual object tracking VOT2014 challenge results. In: Agapito, L., Bronstein, M., Rother, C. (eds.) Computer Vision - ECCV 2014 Workshops. LNCS, vol. 8926, pp. 191–217. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-16181-5_14

  31. Kristan, M., Pflugfelder, R., Matas, J., et al.: The visual object tracking VOT2015 challenge results. In: 2015 IEEE International Conference on Computer Vision Workshop, pp. 564–586. IEEE, Santiago (2016)

    Google Scholar 

  32. Xu, X.W., Zhang, X.Y., Yu, B., et al.: DAC-SDC Low Power Object Detection Challenge for UAV Applications. CoRR abs/1809.00110 (2018)

    Google Scholar 

  33. Zhu, P.F., Wen, L.Y., Bian, X., et al.: Vision Meets Drones: A Challenge. CoRR abs/1804.07437 (2018)

    Google Scholar 

  34. Babenko, B., Yang, M.H., Belongie, S.J.: Robust object tracking with online multiple instance learning. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1619–1632 (2011)

    Article  Google Scholar 

  35. Molchanov, P., Tyree, S., Karras, T., et al.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)

Download references

Author information

Authors and Affiliations

  1. Nanjing University of Posts and Telecommunications, Nanjing, 210023, Jiangsu, China

    Li Wang, Xin Shu, Wei Zhang & Yunfang Chen

  2. Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing University of Posts and Telecommunications, Nanjing, 210023, Jiangsu, China

    Wei Zhang

Authors
  1. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunfang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yunfang Chen .

Editor information

Editors and Affiliations

  1. Nanjing Institute of Technology, Nanjing, China

    Yuan Tian

  2. Nanjing University of Information Science and Technology, Nanjing, China

    Tinghuai Ma

  3. King Saud University, Riyadh, Saudi Arabia

    Muhammad Khurram Khan

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, L., Shu, X., Zhang, W., Chen, Y. (2020). Design and Optimization of Evaluation Metrics in Object Detection and Tracking for Low-Altitude Aerial Video. In: Tian, Y., Ma, T., Khan, M. (eds) Big Data and Security. ICBDS 2019. Communications in Computer and Information Science, vol 1210. Springer, Singapore. https://doi.org/10.1007/978-981-15-7530-3_18

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-7530-3_18

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-7529-7

  • Online ISBN: 978-981-15-7530-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation