Skip to main content
Springer Nature Link
Log in

Multi-branch Deep Neural Model for Natural Language-Based Vehicle Retrieval

  • Conference paper
  • First Online:
Computer Vision and Machine Intelligence

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 586))

  • 778 Accesses

Abstract

Natural language interfaces (NLIs) have seen tremendous popularity in recent times. The utility of natural language descriptions for identifying vehicles in city-scale smart traffic systems is an emerging problem that has received significant research interest. NL-based vehicle identification/retrieval can significantly improve existing systems’ usability and user-friendliness. In this paper, the problem of NL-based vehicle retrieval is explored, which focuses on the retrieval/identification of a unique vehicle from a single-view video given the vehicle’s natural language description. Natural language descriptions are leveraged to identify a specific target vehicle based on its visual features and environmental features such as trajectory and neighbours. We propose a multi-branch model that learns the target vehicle’s visual features, environmental features, and direction and uses the concatenated feature vector to calculate a similarity score by comparing it with the feature vector of the given natural language description, thus identifying the vehicle of interest. The Cityflow-NL dataset was used for the purpose of training/validation, and the performance was measured using MRR (Mean Reciprocal Rank). The proposed model achieved a standardised MRR score of 0.15, which is on par with state-of-the-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bai, S., Zheng, Z., Wang, X., Lin, J., Zhang, Z., Zhou, C., Yang, H., Yang, Y.: Connecting language and vision for natural language-based vehicle retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4034–4043 (2021)

    Google Scholar 

  2. Clark, K., Luong, M., Le, Q., Manning, C.: Electra: Pre-training Text Encoders as Discriminators Rather than Generators (2020). ArXiv Preprint ArXiv:2003.10555

  3. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018). arXiv preprint arXiv:1810.04805

  4. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An Image Is Worth 16\(\,\times \,\)16 Words: Transformers for Image Recognition at Scale (2020). ArXiv Preprint ArXiv:2010.11929

  5. Feng, Q., Ablavsky, V., Sclaroff, S.: CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions (2021). ArXiv Preprint ArXiv:2101.04741

  6. Feng, Q., Ablavsky, V., Sclaroff, S.: CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions (2021). arXiv:2101.04741,2021

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  8. Khorramshahi, P., Rambhatla, S., Chellappa, R.: Towards accurate visual and natural language-based vehicle retrieval systems. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4183–4192 (2021)

    Google Scholar 

  9. Lee, S., Woo, T., Lee, S.: SBNet: Segmentation-based network for natural language-based vehicle search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4054-4060 (2021)

    Google Scholar 

  10. Leviathan, Y., Matias, Y.: An AI system for accomplishing real-world tasks over the phone. Google Duplex (2018)

    Google Scholar 

  11. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A Robustly Optimised Bert Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019)

  12. Naphade, M., Anastasiu, D., Sharma, A., Jagrlamudi, V., Jeon, H., Liu, K., Chang, M., Lyu, S., Gao, Z.: The NVIDIA AI city challenge. In: Prof, SmartWorld (2017)

    Google Scholar 

  13. Naphade, M., Chang, M., Sharma, A., Anastasiu, D., Jagarlamudi, V., Chakraborty, P., Huang, T., Wang, S., Liu, M., Chellappa, R., Hwang, J., Lyu, S.: The 2018 NVIDIA AI city challenge. In: Proceedings of CVPR Workshops, pp. 53–60 (2018)

    Google Scholar 

  14. Naphade, M., Tang, Z., Chang, M., Anastasiu, D., Sharma, A., Chellappa, R., Wang, S., Chakraborty, P., Huang, T., Hwang, J., Lyu, S.: The 2019 AI city challenge. In: The IEEE Conference On Computer Vision And Pattern Recognition (CVPR) Workshops, pp. 452–460 (2019)

    Google Scholar 

  15. Naphade, M., Wang, S., Anastasiu, D., Tang, Z., Chang, M., Yang, X., Yao, Y., Zheng, L., Chakraborty, P., Lopez, C., Sharma, A., Feng, Q., Ablavsky, V., Sclaroff, S.: The 5th AI city challenge. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2021)

    Google Scholar 

  16. Naphade, M., Wang, S., Anastasiu, D., Tang, Z., Chang, M., Yang, X., Zheng, L., Sharma, A., Chellappa, R., Chakraborty, P.: The 4th AI city challenge. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2665–2674 (2020)

    Google Scholar 

  17. Nguyen, T., Pham, Q., Doan, L., Trinh, H., Nguyen, V., Phan, V.: Contrastive learning for natural language-based vehicle retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4245–4252 (2021)

    Google Scholar 

  18. Pan, X., Luo, P., Shi, J., Tang, X. Two at once: enhancing learning and generalisation capacities via ibn-net. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 464–479 (2018)

    Google Scholar 

  19. Park, E., Kim, H., Jeong, S., Kang, B., Kwon, Y.: Keyword-based vehicle retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4220–4227 (2021)

    Google Scholar 

  20. Pennington, J., Socher, R., Manning, Glove, C.: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  21. Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: visual reasoning with a general conditioning layer. In: Proceedings of the AAAI Conference on Artificial Intelligence 32 (2018)

    Google Scholar 

  22. Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.: Learning Transferable Visual Models from Natural Language Supervision (2021). ArXiv Preprint ArXiv:2103.00020

  23. Santoro, A., Raposo, D., Barrett, D., Malinowski, M., Pascanu, R., Battaglia, P., Lillicrap, T.: A Simple Neural Network Module for Relational Reasoning (2017). ArXiv Preprint ArXiv:1706.01427

  24. Scribano, C., Sapienza, D., Franchini, G., Verucchi, M., Bertogna, M.: All you can embed: natural language based vehicle retrieval with spatio-temporal transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4253–4262 (2021)

    Google Scholar 

  25. Sebastian, C., Imbriaco, R., Meletis, P., Dubbelman, G., Bondarev, E., et al.: TIED: a cycle consistent encoder-decoder model for text-to-image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4138–4146 (2021)

    Google Scholar 

  26. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-scale Image Recognition (2014). ArXiv Preprint ArXiv:1409.1556

  27. Sun, Z., Liu, X., Bi, X., Nie, X., Yin, Y.: DUN: Dual-path temporal matching network for natural language-based vehicle retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4061–4067 (2021)

    Google Scholar 

  28. Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114 (2019)

    Google Scholar 

  29. Tang, Z., Naphade, M., Liu, M., Yang, X., Birchfield, S., Wang, S., Kumar, R., Anastasiu, D., Hwang, J.: CityFlow: a city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8797–8806 (2019)

    Google Scholar 

  30. Wang, H., Hou, J., Chen, N.: A survey of vehicle re-identification based on deep learning. IEEE Access. 7, 172443–172469 (2019)

    Article  Google Scholar 

  31. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, National Institute of Technology Karnataka, Surathkal, Mangalore, 575025, India

    N. Shankaranarayan & S. Sowmya Kamath

Authors
  1. N. Shankaranarayan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Sowmya Kamath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to N. Shankaranarayan .

Editor information

Editors and Affiliations

  1. Computer Vision Laboratory, University of Sassari, Alghero, Sassari, Italy

    Massimo Tistarelli

  2. Computer Vision and Biometrics Lab, Department of Information Technology, Indian Institute of Information Technology Allahabad, Prayagraj, India

    Shiv Ram Dubey

  3. Computer Vision and Biometrics Lab, Department of Information Technology, Indian Institute of Information Technology, Allahabad, India

    Satish Kumar Singh

  4. University of Münster, Münster, Germany

    Xiaoyi Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shankaranarayan, N., Sowmya Kamath, S. (2023). Multi-branch Deep Neural Model for Natural Language-Based Vehicle Retrieval. In: Tistarelli, M., Dubey, S.R., Singh, S.K., Jiang, X. (eds) Computer Vision and Machine Intelligence. Lecture Notes in Networks and Systems, vol 586. Springer, Singapore. https://doi.org/10.1007/978-981-19-7867-8_48

Download citation

Publish with us

Policies and ethics