Skip to main content
SpringerLink
Log in

MIDFA: Memory-Based Instance Division and Feature Aggregation Network for Video Object Detection

  • Conference paper
  • First Online:
Advances in Knowledge Discovery and Data Mining (PAKDD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13937))

Included in the following conference series:

Abstract

Previous video object detection methods focus on aggregating the features of other frames into the current frame to alleviate the image degradation, but they rarely focus on multi-class scenes. Aggregating features of different classes will generate confusing information and affect network performance. This problem can be solved by using a classifier to divide the features. However, classifier method has three problems: (a) Heterogeneous high-similarity objects and homogeneous low-similarity objects affect the accuracy of the classifier. (b) Two objects whose positions overlap also affect the classifier. (c) Previous classifier method did not exploit sufficient global information. Therefore, we propose a new method that divides the features of different instances to deal with the problems of (a) and (b). Then we designed two new memories (one is Init Memory and the other is MDR) to solve problem (c). These three parts constitute the MIDFA network. Experiments show that our method achieves 83.76% mAP on the ImageNet VID dataset based on ResNet-101, and 84.6% mAP on ResNeXt-101. In addition, we also conduct experiments on a custom-designed multi-class VID dataset, and adding Instance Division and MDR can increase the mAP of the network by 0.6% compared to using only Init Memory.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bertasius, G., Torresani, L., Shi, J.: Object detection in video with spatiotemporal sampling networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 342–357. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01258-8_21

    Chapter  Google Scholar 

  2. Chen, K., et al.: Optimizing video object detection via a scale-time lattice. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7814–7823 (2018)

    Google Scholar 

  3. Chen, Y., Cao, Y., Hu, H., Wang, L.: Memory enhanced global-local aggregation for video object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10337–10346 (2020)

    Google Scholar 

  4. Deng, H., et al.: Object guided external memory network for video object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6678–6687 (2019)

    Google Scholar 

  5. Deng, J., Pan, Y., Yao, T., Zhou, W., Li, H., Mei, T.: Relation distillation networks for video object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7023–7032 (2019)

    Google Scholar 

  6. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)

    Google Scholar 

  7. Gong, T., et al.: Temporal ROI align for video object recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1442–1450 (2021)

    Google Scholar 

  8. Han, L., Wang, P., Yin, Z., Wang, F., Li, H.: Class-aware feature aggregation network for video object detection. IEEE Trans. Circuits Syst. Video Technol. 32, 8165–8178 (2021)

    Google Scholar 

  9. Han, L., Wang, P., Yin, Z., Wang, F., Li, H.: Context and structure mining network for video object detection. Int. J. Comput. Vision 129(10), 2927–2946 (2021)

    Article  Google Scholar 

  10. Han, W., et al.: Seq-NMS for video object detection. arXiv preprint arXiv:1602.08465 (2016)

  11. He, F., Gao, N., Li, Q., Du, S., Zhao, X., Huang, K.: Temporal context enhanced feature aggregation for video object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 10941–10948 (2020)

    Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  13. Jiang, Z., et al.: Learning where to focus for efficient video object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 18–34. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_2

    Chapter  Google Scholar 

  14. Lin, L.: Dual semantic fusion network for video object detection. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1855–1863 (2020)

    Google Scholar 

  15. Luo, H., Xie, W., Wang, X., Zeng, W.: Detect or track: towards cost-effective video object detection/tracking. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8803–8810 (2019)

    Google Scholar 

  16. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  17. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  18. Shvets, M., Liu, W., Berg, A.C.: Leveraging long-range temporal relationships between proposals for video object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9756–9764 (2019)

    Google Scholar 

  19. Sun, G., Hua, Y., Hu, G., Robertson, N.: MAMBA: multi-level aggregation via memory bank for video object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 2620–2627 (2021)

    Google Scholar 

  20. Wu, H., Chen, Y., Wang, N., Zhang, Z.: Sequence level semantics aggregation for video object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9217–9225 (2019)

    Google Scholar 

  21. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

  22. Zhu, X., Dai, J., Yuan, L., Wei, Y.: Towards high performance video object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7210–7218 (2018)

    Google Scholar 

  23. Zhu, X., Wang, Y., Dai, J., Yuan, L., Wei, Y.: Flow-guided feature aggregation for video object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 408–417 (2017)

    Google Scholar 

  24. Han, M., Wang, Y., Chang, X., Qiao, Yu.: Mining inter-video proposal relations for video object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 431–446. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_26

    Chapter  Google Scholar 

Download references

Acknowledgement

This work is supported by Shanghai “Science and Technology Innovation Action Plan” Venus Project (Sailing Special Project) under Grant 23YF1412900.

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shanghai, China

    Qiaochuan Chen, Min Zhou & Hang Yu

  2. Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education, Shanghai, China

    Qiaochuan Chen

Authors
  1. Qiaochuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hang Yu .

Editor information

Editors and Affiliations

  1. Kyoto University, Kyoto, Japan

    Hisashi Kashima

  2. IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA

    Tsuyoshi Ide

  3. National Chiao Tung University, Hsinchu, Taiwan

    Wen-Chih Peng

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, Q., Zhou, M., Yu, H. (2023). MIDFA: Memory-Based Instance Division and Feature Aggregation Network for Video Object Detection. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13937. Springer, Cham. https://doi.org/10.1007/978-3-031-33380-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-33380-4_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-33379-8

  • Online ISBN: 978-3-031-33380-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation