Skip to main content
SpringerLink
Log in

Partially shaded sketch-based image search in real mobile device environments via sketch-oriented compact neural codes

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

With the advent of touch screens in mobile devices, sketch-based image search is becoming the most intuitive method to query multimedia contents. Traditionally, sketch-based queries were formulated with hand-drawn shapes without any shades or colors. The absence of such critical information from sketches increased the ambiguity between natural images and their sketches. Although it was previously considered too cumbersome for users to add colors to hand-drawn sketches in image retrieval systems, the modern day touch input devices make it convenient to add shades or colors to query sketches. In this work, we propose deep neural codes extracted from partially colored sketches by an efficient convolutional neural network (CNN) fine-tuned on sketch-oriented augmented dataset. The training dataset is constructed with hand-drawn sketches, natural color images, de-colorized, and de-texturized images, coarse and fine edge maps, and flipped and rotated images. Fine-tuning CNN with augmented dataset enabled it to capture features effectively for representing partially colored sketches. We also studied the effects of shading and partial coloring on retrieval performance and show that the proposed method provides superior performance in sketch-based large-scale image retrieval on mobile devices as compared to other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ahmad, J., Muhammad, K., Lloret, J., Baik, S.W.: Efficient conversion of deep features to compact binary codes using fourier decomposition for multimedia big Data IEEE Trans. Ind. Inf. PP, 1–1 (2018)

    Google Scholar 

  2. Wang, S., Zhang, J., Han, T.X., Miao, Z.: Sketch-based image retrieval through hypothesis-driven object boundary selection with hlr descriptor. IEEE Trans. Multimed 17, 1045–1057 (2015)

    Article  Google Scholar 

  3. Ahmad, J., Sajjad, M., Rho, S., Baik, S.W.: Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems. Multimed. Tools Appl. 75, 12669–12692 (2016)

    Article  Google Scholar 

  4. Ahmad, J., Sajjad, M., Mehmood, I., Rho, S., Baik, S.W.: Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems J. Real-Time Image Proc. 13, 431–447 (2017)

    Article  Google Scholar 

  5. Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117, 790–806 (2013)

    Article  Google Scholar 

  6. Kim, S., Guy, S.J., Hillesland, K., Zafar, B., Gutub, A.A.-A., Manocha, D.: Velocity-based modeling of physical interactions in dense crowds. Vis. Comput. 31, 541–555 (2015)

    Article  Google Scholar 

  7. Tseng, K.-Y., Lin, Y.-L., Chen, Y.-H., Hsu, W.H.: Sketch-based image retrieval on mobile devices using compact hash bits. In Proceedings of the 20th ACM International Conference on Multimedia, pp. 913–916 (2012)

  8. Al-Otaibi, N.A., Gutub, A.A.: 2-Leyer security system for hiding sensitive text data on personal computers. Lect. Notes Inf. Theory 2(2) (2014)

  9. Abdelgawad, H., Shalaby, A., Abdulhai, B., Gutub, A.A.A.: Microscopic modeling of large-scale pedestrian–vehicle conflicts in the city of Madinah, Saudi Arabia. J. Adv. Transp. 48, 507–525 (2014)

    Article  Google Scholar 

  10. Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. 31, 44:1–44:10 (2012)

    Google Scholar 

  11. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255 (2009)

  12. Ahmad, J., Muhammad, K., Baik, S.W.: Medical image retrieval with compact binary codes generated in frequency domain using highly reactive convolutional features. J. Med. Syst. 42, 24 (2017)

    Article  Google Scholar 

  13. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1, 1097–1105 (2012)

    Google Scholar 

  14. Ahmad, J., Muhammad, K., Bakshi, S., Baik, S.W.: Object-oriented convolutional features for fine-grained image retrieval in large surveillance datasets. Fut. Gen. Comput. Syst. 81, 314–330 (2018)

    Article  Google Scholar 

  15. Ahmad, J., Sajjad, M., Mehmood, I., Baik, S.W.: SiNC: saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PLoS One 12, e0181707 (2017)

    Article  Google Scholar 

  16. Qi, Y., Song, Y.-Z., Zhang, H., Liu, J.: Sketch-based image retrieval via Siamese convolutional neural network. In Image Processing (ICIP), 2016 IEEE International Conference on, pp. 2460–2464 (2016)

  17. Krizhevsky, A., Hinton, G.E.: Using very deep autoencoders for content-based image retrieval. In: Proceedings of the 19th European Symposium on Artificial Neural Networks, Bruges, Belgium (2011)

  18. Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Computer Vision–European Conference on Computer Vision (ECCV), Springer, pp. 584–599 (2014)

  19. Gutub, A., Alharthi, N.: Improving Hajj and Umrah Services Utilizing Exploratory Data Visualization Techniques, presented at the Hajj Forum. Umm Al-Qura University–King Abdulaziz Historical Hall, Makkah (2016)

    Google Scholar 

  20. Gutub, A.: Exploratory data visualization for smart systems. Smart cities 2015-3rd annual digital grids and smart cities workshop, Burj Rafal Hotel Kempinski, Riyadh (2015)

  21. Gutub, A.: Social media and its impact on e-Governance. ME smart cities 2015-4th middle east smart cities summit, 8–9 Dec, Dubai (2015)

  22. Chen, T., Cheng, M.-M., Tan, P., Shamir, A., Hu, S.-M.: Sketch2photo: Internet image montage. ACM Trans. Graph. (TOG) 28, 124 (2009)

    Google Scholar 

  23. Fu, L., Kara, L.B.: Recognizing network-like hand-drawn sketches: a convolutional neural network approach. In ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 671–681 (2009)

  24. Sarvadevabhatla, R.K., Babu, R.V.: Freehand sketch recognition using deep features. arXiv:1502.00254 (2015)

  25. Al-Otaibi, N.A., Gutub, A.A.: Flexible stego-system for hiding text in images of personal computers based on user security priority. In: Proceedings of: 2014 International Conference on Advanced Engineering Technologies (AET-2014), pp. 250–256 (2014)

  26. Seddati, O., Dupont, S., Mahmoudi, S.: Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on, pp. 1–6 (2015)

  27. Liu, L., Shen, F., Shen, Y., Liu, X., Shao, L.: Deep sketch hashing: fast free-hand sketch-based image retrieval. arXiv:1703.05605 (2017)

  28. Wang, X., Duan, X., Bai, X.: Deep sketch feature for cross-domain image retrieval. Neurocomputing 207, 387–397 (2016)

    Article  Google Scholar 

  29. Ahmad, J., Mehmood, I., Baik, S.W.: Efficient object-based surveillance image search using spatial pooling of convolutional features. J. Vis. Commun. Image Rep. 45, 62–76 (2017)

    Article  Google Scholar 

  30. Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 1 MB model size. arXiv:1602.07360 (2016)

  31. Ahmad, J., Muhammad, K., Baik, S.W.: Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. PLoS One 12, e0183838 (2017)

    Article  Google Scholar 

  32. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 629–639 (1990)

    Article  Google Scholar 

  33. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  34. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  35. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 (2013)

  36. Gschwend, D.: Zynqnet: an fpga-accelerated embedded convolutional neural network. MS thesis, Swiss Federal Institute of Technology Zurich (ETH-Zurich) (2016)

  37. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications arXiv:1704.04861 (2017)

  38. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR 2, abs/1510.00149 (2015)

    Google Scholar 

  39. MathWorks (2015) MATLAB. Available: http://www.mathworks.com/products/parallel-computing/

  40. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., et al.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia, pp. 675–678 (2014)

  41. Nvidia DIGITS: Available: https://developer.nvidia.com/digits. (2016)

  42. Caltech-256 Object Category Dataset: Available: http://resolver.caltech.edu/CaltechAUTHORS:CNS-TR-2007-001

  43. Çalışır, F., Baştan, M., Ulusoy, Ö., Güdükbay, U.: Mobile multi-view object image search. Multimed. Tool Appl. 76, 12433–12456 (2017)

    Article  Google Scholar 

  44. Yang, Y., Hospedales, T.M.: Deep neural networks for sketch recognition. arXiv:1501.07873 (2015)

  45. Lin, K., Yang, H.-F., Hsiao, J.-H., Chen, C.-S., Deep learning of binary hash codes for fast image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 27–35 (2015)

  46. Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hashing. VLDB, pp. 518–529 (1999)

  47. Datar, M., Immorlica, N., Indyk, P., Mirrokni, V.S.: Locality-sensitive hashing scheme based on p-stable distributions. In: Proceedings of the Twentieth Annual Symposium on Computational Geometry, pp. 253–262 (2004)

  48. Weiss, Y., Torralba, A., Fergus, R.: Spectral hashing. Advances in neural information processing systems, pp. 1753–1760 (2009)

  49. Jin, Z., Li, C., Lin, Y., Cai, D.: Density sensitive hashing. IEEE Trans. Cybern. 44, 1362–1371 (2014)

    Article  Google Scholar 

  50. Heo, J.-P., Lee, Y., He, J., Chang, S.-F., Yoon, S.-E.: Spherical hashing. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 2957–2964 (2012)

Download references

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF) Grant Funded by the Korea Government (MSIP) (No. 2016R1A2B4011712).

Author information

Authors and Affiliations

  1. Department of Computer Science, Islamia College, Peshawar, Pakistan

    Jamil Ahmad

  2. Digital Contents Research Institute, Sejong University, Seoul, South Korea

    Khan Muhammad & Sung Wook Baik

  3. Department of Mathematics, Islamia College Peshawar, Peshawar, Pakistan

    Syed Inayat Ali Shah

  4. School of Computing Science and Engineering, VIT University, Vellore, India

    Arun Kumar Sangaiah

Authors
  1. Jamil Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khan Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Syed Inayat Ali Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arun Kumar Sangaiah
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sung Wook Baik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sung Wook Baik.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmad, J., Muhammad, K., Shah, S.I.A. et al. Partially shaded sketch-based image search in real mobile device environments via sketch-oriented compact neural codes. J Real-Time Image Proc 16, 227–240 (2019). https://doi.org/10.1007/s11554-018-0784-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-018-0784-x

Keywords

Search

Navigation