Skip to main content
SpringerLink
Log in

M3T: Multi-class Multi-instance Multi-view Object Tracking for Embodied AI Tasks

  • Conference paper
  • First Online:
Image and Vision Computing (IVCNZ 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13836))

Included in the following conference series:

  • 898 Accesses

Abstract

In this paper, we propose an extended multiple object tracking (MOT) task definition for embodied AI visual exploration research task - multi-class, multi-instance and multi-view object tracking (M3T). The aim of the proposed M3T task is to identify the unique number of objects in the environment, observed on the agent’s way, and visible from far or close view, from different angles or visible only partially. Classic MOT algorithms are not applicable for the M3T task, as they typically target moving single-class multiple object instances in one video and track objects, visible from only one angle or camera viewpoint. Thus, we present the M3T-Round algorithm designed for a simple scenario, where an agent takes 12 image frames, while rotating 360° from the initial position in a scene. We, first, detect each object in all image frames and then track objects (without any training), using cosine similarity metric for association of object tracks. The detector part of our M3T-Round algorithm is compatible with the baseline YOLOv4 algorithm [1] in terms of detection accuracy: a 5.26 point improvement in AP75. The tracker part of our M3T-Round algorithm shows a 4.6 point improvement in HOTA over GMOTv2 algorithm [2], a recent, high-performance tracking method. Moreover, we have collected a new challenging tracking dataset from AI2-Thor [3] simulator for training and evaluation of the proposed M3T-Round algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: YOLOv4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)

  2. MOT15 results. https://motchallenge.net/results/MOT15/. Accessed 20 Sept 2022

  3. Kolve, E., et al.: AI2-THOR: an interactive 3D environment for visual AI. arXiv preprint arXiv:1712.05474 (2017)

  4. Batra, D., et al.: Rearrangement: a challenge for embodied AI. arXiv preprint arXiv:2011.01975 (2020)

  5. Hall, D., et al.: The robotic vision scene understanding challenge. arXiv preprint arXiv:2009.05246 (2020)

  6. Ciaparrone, G., et al.: Deep learning in video multi-object tracking: a survey. Neurocomputing 381, 61–88 (2020)

    Article  Google Scholar 

  7. Leal-Taixé, L., et al.: MOT challenge 2015: towards a benchmark for multi-target tracking. arXiv preprint arXiv:1504.01942 (2015)

  8. Redmon, J., Ali, F.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  9. Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)

    Google Scholar 

  10. He, K., et al.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  11. Ren, S., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  12. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  13. Zhu, X., et al.: Deformable DETR: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)

  14. Kotar, K., Mottaghi, R.: Interactron: embodied adaptive object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14860–14869 (2022)

    Google Scholar 

  15. Yang, J., et al.: Embodied visual recognition. arXiv preprint arXiv:1904.04404 (2019)

  16. Li, A., Liu, L., Wang, K., Liu, S., Yan, S.: Clothing attributes assisted person reidentification. IEEE Trans. Circ. Syst. Video Technol. 25(5), 869–878 (2015)

    Article  Google Scholar 

  17. Farenzena, M., Bazzani, L., Perina, A., Murino, V., Cristani, M.: Person re-identification by symmetry-driven accumulation of local features. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition, pp. 2360–2367. IEEE (2010)

    Google Scholar 

  18. Zhao, J., et al.: Heterogeneous relational complement for vehicle re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 205–214 (2021)

    Google Scholar 

  19. Yu, J., et al.: Camera-tracklet-aware contrastive learning for unsupervised vehicle re-identification. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 905–911. IEEE (2022)

    Google Scholar 

  20. Bansal, V., Foresti, G.L., Martinel, N.: Where did i see it? Object instance re-identification with attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 298–306 (2021)

    Google Scholar 

  21. Bewley, A., et al.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468. IEEE (2016)

    Google Scholar 

  22. Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3645–3649. IEEE (2017)

    Google Scholar 

  23. Zhu, X., et al.: ViTT: vision transformer tracker. Sensors 21(16), 5608 (2021)

    Article  Google Scholar 

  24. Sun, P., et al.: TransTrack: multiple object tracking with transformer. arXiv preprint arXiv:2012.15460 (2020)

  25. Meinhardt, T., et al.: TrackFormer: multi-object tracking with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8844–8854 (2022)

    Google Scholar 

  26. Zeng, F., et al.: MOTR: end-to-end multiple-object tracking with transformer. arXiv preprint arXiv:2105.03247 (2021)

  27. Savva, M., et al.: Habitat: a platform for embodied AI research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9339–9347 (2019)

    Google Scholar 

  28. Luiten, J., et al.: HOTA: A higher order metric for evaluating multi-object tracking. Int. J. Comput. Vis. 129(2), 548–578 (2021). https://doi.org/10.1007/s11263-020-01375-2

    Article  Google Scholar 

  29. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear MOT metrics. EURASIP J. Image Video Process. 1, 1–10 (2008). https://doi.org/10.1155/2008/246309

    Article  Google Scholar 

  30. Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 17–35. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_2

    Chapter  Google Scholar 

  31. Weihs, L., et al.: Visual room rearrangement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5922–5931 (2021)

    Google Scholar 

  32. Deitke, M., et al.: ProcTHOR: large-scale embodied AI using procedural generation. arXiv preprint arXiv:2206.06994 (2022)

  33. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Piotr Dollár, C., Zitnick, L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Edith Cowan University, Perth, WA, 6027, Australia

    Mariia Khan, Jumana Abu-Khalaf & David Suter

  2. Leibniz University Hannover, 30167, Hannover, Germany

    Bodo Rosenhahn

Authors
  1. Mariia Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jumana Abu-Khalaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Suter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bodo Rosenhahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariia Khan .

Editor information

Editors and Affiliations

  1. Auckland University of Technology, Auckland, New Zealand

    Wei Qi Yan

  2. Auckland University of Technology, Auckland, New Zealand

    Minh Nguyen

  3. Auckland University of Technology, Auckland, New Zealand

    Martin Stommel

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Khan, M., Abu-Khalaf, J., Suter, D., Rosenhahn, B. (2023). M3T: Multi-class Multi-instance Multi-view Object Tracking for Embodied AI Tasks. In: Yan, W.Q., Nguyen, M., Stommel, M. (eds) Image and Vision Computing. IVCNZ 2022. Lecture Notes in Computer Science, vol 13836. Springer, Cham. https://doi.org/10.1007/978-3-031-25825-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25825-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25824-4

  • Online ISBN: 978-3-031-25825-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation