Abstract
Cross-modal retrieval aims at searching semantically similar examples in one modality by using a query from another modality. Its typical applications including image-based text retrieval (IBTR) and text-based image retrieval (TBIR). Due to the rapid growth of multimodal data and the success of deep learning, cross-modal retrieval has received increasing attention and achieved significant progress in recent years. Dual-path CNN is a novel framework in this domain, which yields competitive performance by utilizing instance loss and inter-modal loss. However, it is still less discriminative in modeling the intra-modal relationship, which is also important in bridging a more discriminative cross-modal embedding network. To this end, we propose to incorporate an additional intra-modal loss into the framework to remedy this problem by preserving the intra-modal structure. Further, we develop a novel batch flexible sampling approach to train the entire network effectively and efficiently. Our approach, named Discriminative Dual-Path CNN (DDPC), achieves the state-of-the-art results on the MS-COCO dataset, improving IBTR by 4.9% and TBIR by 5.9% based on Recall@1 on the 5K test set.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ge, J., Liu, X., Hong, R., et al.: Deep graph Laplacian hashing for image retrieval. In: Pacific Rim Conference on Multimedia, pp. 3–13 (2017)
Hong, R., Li, L., Cai, J., et al.: Coherent semantic-visual indexing for large-scale image retrieval in the cloud. IEEE Trans. Image Process. 26(9), 4128–4138 (2017)
Wang, B., Xu, Y., Han, Y., et al.: Movie question answering: remembering the textual cues for layered visual contents. arXiv preprint arXiv:1804.09412 (2018)
Li, C., Deng, C., Li, N., et al.: Self-supervised adversarial hashing networks for cross-modal retrieval. arXiv preprint arXiv:1804.01223 (2018)
Peng, Y., Qi, J., Huang, X., et al.: CCL: cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans. Multimed. 20(2), 405–420 (2018)
Hong, R., Zhang, L., Zhang, C., et al.: Flickr circles: aesthetic tendency discovery by multi-view regularized topic modeling. IEEE Trans. Multimed. 18(8), 1555–1567 (2016)
Hong, R., Zhang, L., Tao, D.: Unified photo enhancement by discovering aesthetic communities from flickr. IEEE Trans. Image Process. 25(3), 1124–1135 (2016)
Hong, R., Hu, Z., Wang, R., et al.: Multi-view object retrieval via multi-scale topic models. IEEE Trans. Image Process. 25(12), 5814–5827 (2016)
Liu, J., Zha, Z., Tian, Q., et al.: Multi-scale triplet CNN for person re-identification. In: ACM International Conference on Multimedia, pp. 192–196 (2016)
Liu, J., Zha, Z., Chen, X., et al.: Dense 3D-convolutional neural network for person re-identification in videos. ACM Trans. Multimed. Comput. Commun. Appl. 14(4) (2018)
Ji, Z., Zhen, W., Pang, Y.: Deep pedestrian attribute recognition based on LSTM. In: IEEE International Conference on Image Processing (2017)
Liu, J., Zha, Z., Zhang, H., et al.: Context-aware visual policy network for sequence-level image captioning. In: ACM International Conference on Multimedia (2018)
Gong, Y., Ke, Q., Isard, M., et al.: A multi-view embedding space for modeling internet images, tags, and their semantics. Int. J. Comput. Vis. 106(2), 210–233 (2014)
Klein, B., Lev, G., Sadeh, G., et al.: Fisher vectors derived from hybrid Gaussian–Laplacian mixture models for image annotation. In: IEEE Computer Vision and Pattern Recognition, pp. 4437–4446 (2015)
Mao, J., Xu, W., Yang, Y., et al.: Deep captioning with multimodal recurrent neural networks (m-RNN). In: International Conference on Learning Representations (2015)
Kiros, R., Salakhutdinov, R., Zemel, R.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 (2014)
Niu, Z., Zhou, M., Wang, L., et al.: Hierarchical multimodal LSTM for dense visual-semantic embedding. In: IEEE International Conference on Computer Vision, pp. 1899–1907 (2017)
Zheng, Z., Zheng, L., Garrett, M., et al.: Dual-path convolutional image-text embedding. In: IEEE International Conference on Computer Vision (2018)
Weinberger, K., Saul, L.: Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10, 207–244 (2009)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Mikolov, T., Chen, K., Corrado, G., et al.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for MATLAB. In: ACM International Conference on Multimedia, pp. 689–692 (2015)
Lev, G., Sadeh, G., Klein, B., Wolf, L.: RNN Fisher vectors for action recognition and image annotation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 833–850. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_50
Huang, Y., Wang, W., Wang, L.: Instance-aware image and sentence matching with selective multimodal LSTM. In: IEEE Computer Vision and Pattern Recognition, pp. 2310–2318 (2017)
Karpathy, A., Li, F.F.: Deep visual-semantic alignments for generating image descriptions. In: IEEE Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)
Karpathy, A., Joulin, A., Li, F.F.: Deep fragment embeddings for bidirectional image sentence mapping. In: Advances in Neural Information Processing Systems, pp. 1889–1897 (2014)
Shaw, B., Jebara, T.: Structure preserving embedding. In: ACM International Conference on Machine Learning, pp. 937–944 (2009)
Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: IEEE Computer Vision and Pattern Recognition, pp. 5005–5013 (2016)
Lin, X., Parikh, D.: Leveraging visual question answering for image-caption ranking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 261–277. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_17
Acknowledgements
This work was supported by the National Basic Research Program of China under Grant 2014CB340400, and the National Natural Science Foundation of China under Grants 61771329, 61472273, and 61632018.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, H., Ji, Z., Pang, Y. (2018). Cross-Modal Retrieval with Discriminative Dual-Path CNN. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11166. Springer, Cham. https://doi.org/10.1007/978-3-030-00764-5_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-00764-5_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00763-8
Online ISBN: 978-3-030-00764-5
eBook Packages: Computer ScienceComputer Science (R0)