Abstract
Sketch-based image retrieval is a fundamental computer vision problem. Instead of using hand-designed features to represent sketches and images, recent researches apply deep learning approaches combined with fine-grained matching to retrieve images with fine-grained details. Although these researches allow user to use hand-free sketches drawn for retrieving similar objects, the color matching is ignored which induces a low retrieval precision. To address this problem, we propose a single color sketch-based image retrieval (SCSBIR) approach using HSV color feature considering both shape matching and color matching in this paper. The SCSBIR problem is investigated using deep learning networks, in which deep features are used to represent color sketches and images. A novel ranking method considering both shape matching and color matching is also proposed. In addition, we build a SCSBIR dataset with color sketches and images, and train and test our method by using this dataset. The test results show that our method has a better retrieval performance. The research in this paper can not only promote its application in the commercial field, but also provide reference for the future research in this field.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Visual Comput. Graphics 17(11), 1624–1636 (2011)
Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (2013)
Yu, Q., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M., Loy, C.C.: Sketch me that shoe. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 799–807 (2016)
Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. (TOG) 35(4), 119 (2016)
Li, Y., Hospedales, T.M., Song, Y.Z., Gong, S.: Fine-grained sketch-based image retrieval by matching deformable part models. In: Proceedings of the British Machine Vision Conference (2014)
Song, J., Yu, Q., Song, Y.Z., Xiang, T., Hospedales, T.M.: Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5551–5560 (2017). https://doi.org/10.1109/ICCV.2017.592
Wang, F., Kang, L., Li, Y.: Sketch-based 3D shape retrieval using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1875–1883 (2015)
Qi, A., Song, Y., Xiang, T.: Semantic Embedding for Sketch-Based 3D Shape Retrieval. BMVC (2018)
Daras, P., Axenopoulos, A.: A 3D shape retrieval framework supporting multimodal queries. Int. J. Comput. Vision 89(2–3), 229–247 (2010)
Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., Alexa, M.: Sketch-based shape retrieval. ACM Trans. Graph. 31(4), 31–1 (2012)
Bui, T., Collomosse, J.: Scalable sketch-based image retrieval using color gradient features. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1012–1019 (2015). https://doi.org/10.1109/ICCVW.2015.133
Cheng, Z., Li, X., Loy, C.C.: Pedestrian color naming via convolutional neural network. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10112, pp. 35–51. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54184-6_3
Reddy, N., Reddy, G., Narayana, M.: Color sketch based image retrieval. Int. J. Adv. Res. Electrical Electron. Instrum. Eng. 03, 12179–12185 (2014). https://doi.org/10.15662/ijareeie.2014.0309054
Ortega, M., Rui, Y., Chakrabarti, K., Porkaew, K., Mehrotra, S., Huang, T.S.: Supporting ranked boolean similarity queries in MARS. IEEE Trans. Knowl. Data Eng. 10(6), 905–925 (1998)
Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 391–405. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_26
Taigman, Y., Yang, M., Ranzato, M.A., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, pp. 1701–1708. IEEE (2014). https://doi.org/10.1109/CVPR.2014.220
Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, pp. 815–823. IEEE (2015). https://doi.org/10.1109/CVPR.2015.7298682
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, pp. 770–778. IEEE (2016). https://doi.org/10.1109/CVPR.2016.90
Yu, Q., Yang, Y., Song, Y.Z., Xiang, T., Hospedales, T.: Sketch-a-net that beats humans. arXiv preprint arXiv:1501.07873 (2015)
Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. 31(4), 44–1 (2012)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, pp. 248–255. IEEE (2009). https://doi.org/10.1109/CVPR.2009.5206848
Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Comput. Graph. 34(5), 482–498 (2010)
Hu, R., Barnard, M., Collomosse, J.: Gradient field descriptor for sketch based retrieval and localization. In: Proceedings of 2010 IEEE International Conference on Image Processing, Hong Kong, China, pp. 1025–1028. IEEE (2010). https://doi.org/10.1109/ICIP.2010.5649331
Hu, R., Wang, T., Collomosse, J.: A bag-of-regions approach to sketch-based image retrieval. In: Proceedings of 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium, pp. 3661–3664. IEEE (2011). https://doi.org/10.1109/ICIP.2011.6116513
Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. In: CVPR 2011 Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, pp. 761–768. IEEE (2011). https://doi.org/10.1109/CVPR.2011.5995460
Cao, Y., Wang, H., Wang, C., Li, Z., Zhang, L., Zhang, L.: Mindfinder: interactive sketch-based image search on millions of images. In: ACM Multimedia, pp. 1605–1608. ACM, New York (2010)
Xia, Y., Wang, S., Li, Y., You, L., Yang, X., Zhang, J.J.: Fine-grained color sketch-based image retrieval. In: Gavrilova, M., Chang, J., Thalmann, N.M., Hitzer, E., Ishikawa, H. (eds.) CGI 2019. LNCS, vol. 11542, pp. 424–430. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22514-8_40
Acknowledgements
This research is supported by the PDE-GIR project which has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 778035. Yanran Li has received research grands from the South West Creative Technology Network.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer-Verlag GmbH Germany, part of Springer Nature
About this chapter
Cite this chapter
Xia, Y., Wang, S., Li, Y., You, L., Yang, X., Zhang, J.J. (2020). Single Color Sketch-Based Image Retrieval in HSV Color Space. In: Gavrilova, M., Tan, C., Chang, J., Thalmann, N. (eds) Transactions on Computational Science XXXVII. Lecture Notes in Computer Science(), vol 12230. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-61983-4_5
Download citation
DOI: https://doi.org/10.1007/978-3-662-61983-4_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-61982-7
Online ISBN: 978-3-662-61983-4
eBook Packages: Computer ScienceComputer Science (R0)