Skip to main content
SpringerLink
Log in

Iterative samples labeling for sketch recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketch recognition is an important issue in human-computer interaction, especially in sketch-based interface. To provide a scalable and flexible tool for user-centered sketch recognition, this paper proposes an iterative sketch collection annotation method for classifier-training by interleaving online metric learning, semi-supervised clustering and user intervention. It can discover the categories of the collections iteratively by combing online metric learning with semi-supervised clustering, and put the user intervention into the loop of each iteration. The features of our methods lie in three aspects. Firstly, the unlabeled collections are annotated with less effort in a group by group form. Secondly, users can annotate the collections flexibly and freely to define the sketch recognition personally for different applications. Finally, the scalable collection can be annotated efficiently by combining the dynamically processing and online learning. The extensive experimental results prove the effectiveness of our proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Bellet A, Habrard A and Sebban M (2013) A survey on metric learning for feature vectors and structured data. Tech Rep arXiv :1306.6709

  2. Cai D, He X (2012) Manifold adaptive experimental design for text categorization. IEEE Trans Knowl Data Eng 24(4):707–719

    Article  Google Scholar 

  3. Cai X, Nie F, Huang H and Kamangar F (2011) Heterogeneous image feature integration via multi-modal spectral clustering. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1977–1984

  4. Chechik G, Sharma V, Shalit U, Bengio S (2010) Large scale online learning of image similarity through ranking. J Mach Learn Res 11:1109–1135

    MathSciNet  MATH  Google Scholar 

  5. Crammer K, Dekel O, Keshet J et al (2006) Online passive-aggressive algorithms. J Mach Learn Res 7:551–585

    MathSciNet  MATH  Google Scholar 

  6. Eitz M, Hays J, Alexa M (2012) How do human sketch objects? ACM Trans Graph 31(4):1–10

    Google Scholar 

  7. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  8. Fu Z, Ip H, Lu H and Lu Z (2011) Multi-modal constraint propagation for heterogeneous image clustering. In: Proceedings of ACM Multimedia (ACM MM), pp. 143–152

  9. Fu Z, Lu Z, Ip HH-S et al (2015) Local similarity learning for pairwise constraint propagation. Multimed Tools Appl 74(11):3739–3758

    Article  Google Scholar 

  10. Galleguillos C, McFee B, Belongie S and Lanckriet G (2010) Multiclass object localization by combining local contextual interactions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113–120

  11. Galleguillos C, McFee B, Belongie S and Lanckriet G (2011) From region similarity to category discovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2665–2672

  12. Galleguillos C, McFee B, Lanckriet GRG (2014) Iterative category discovery via multiple kernel metric learning. Int J Comput Vis 108(1–2):115–132

    Article  MathSciNet  MATH  Google Scholar 

  13. Griffin G, Holub A and Perona P (2007) Caltech-256 object category dataset. Tech Rep CNSTR-2007-001

  14. Hu R, Collomosse J (2013) A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806

    Article  Google Scholar 

  15. Huang H, Chuang Y and Chen C (2012) Affinity aggregation for spectral clustering. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 773–780

  16. Huang Y, Wu Z, Wang L, Tan T (2014) Feature coding in image classification: a comprehensive study. IEEE Trans Pattern Anal Mach Intell 36(3):493–506

    Article  Google Scholar 

  17. Kapoor A, Grauman K, Urtasun R, Darrell T (2010) Gaussian processes for object categorization. Int J Comput Vis 88(2):169–188

    Article  Google Scholar 

  18. Kriegel H-P, Schubert M and Zimek A (2008) Angle-based outlier detection in high-dimensional data. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 444–452

  19. Lee Y and Grauman K (2010) Object-graphs for context-aware category discovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8

  20. Lee Y and Grauman K (2011) Learning the easy things first: Self-paced visual category discovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1721–1728

  21. Li X and Guo Y (2013) Adaptive active learning for image classification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 859–866

  22. Li Y, Hospedales TM, Song Y, Gong S (2015) Free-hand sketch recognition by multi-kernel feature learning. Comput Vis Image Underst 137(1):1–11

    Google Scholar 

  23. Li Y, Song YZ and Gong SG (2013) Sketch recognition by ensemble matching of structured features. In: Proceedings of British Machine Vision Conference (BMVC), pp. 35.1-35.11.

  24. Liu W, Mu C, Ji RR et al (2015) Low-rank similarity metric learning in high dimensions. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI), pp. 2792–2799

  25. Liu K, Sun Z, MS et al (2015) Iterative collection annotation for sketch recognition. In: Proceedings of Pacific-Rim Conference on Multimedia (PCM), pp. 55–65

  26. Long B, Yu P and Zhang Z (2008) A general model for multiple view unsupervised learning. In: Proceedings of SIAM International Conference on Data Mining (ICDM), pp. 822–833

  27. Lu ZW and Ip HHS (2010) Constrained spectral clustering via exhaustive and efficient constraint propagation. In: Proceedings of the 11th European Conference on Computer Vision (ECCV), pp. 1–14

  28. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42(3):145–175

    Article  MATH  Google Scholar 

  29. Schneider RG, Tuytelaars T (2014) Sketch classification and classification-driven analysis using fisher vectors. ACM Trans Graph 33(6):174, 1–9

    Article  Google Scholar 

  30. Shechtman E, Irani M (2007) Matching local self-similarities across images and videos. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8

  31. Sun S (2013) A survey of multi-view machine learning. Neural Comput & Applic 23(7–8):2031–2038

    Article  Google Scholar 

  32. Sun Z, Wang C, Zhang L and Zhang L (2012) Query-adaptive shape topic mining for hand-drawn sketch recognition. In: Proceedings of ACM International Conference on Multimedia (ACM MM), pp. 519-528

  33. Sun Z, Zhang L, Tang E (2005) An incremental learning method based on SVM for online sketchy shape recognition. LNCS 3610:655–659

    Google Scholar 

  34. Tuytelaars T, Lampert CH, Blaschko M, Buntine W (2010) Unsupervised object discovery: a comparison. Int J Comput Vis 88(2):284–302

    Article  Google Scholar 

  35. Vedaldi A, Fulkerson B (2008) VLFeat: an open and portable library of computer vision algorithms, <http://www.vlfeat.org/>

  36. Wang H, CW, and Yuan J (2014) Multi-feature spectral clustering with minimax optimization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4106–4113

  37. Wigness M, Draper BA and Beveride JR (2014) Selectively guiding visual concept discovery. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision, pp. 247–254

  38. Xia H, Hoi SCH, Jin R (2014) Online multiple kernel similarity learning for visual search. IEEE Trans Pattern Anal Mach Intell 36(3):536–549

    Article  Google Scholar 

  39. Yang Y, Ma Z, Nie F et al (2015) Multi-class active learning by uncertainty sampling with diversity maximization. Int J Comput Vis 113(2):113–127

    Article  MathSciNet  Google Scholar 

  40. Yu Q, Yang Y, Song YZ and Xiang T (2015) Sketch-a-net that beats humans. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 7.1–7.12

  41. Zhang L, Wang L, Lin W et al (2014) Geometric optimum experimental design for collaborative image retrieval. IEEE Trans Circ Syst Video Technol 24(2):346–359

    Article  Google Scholar 

  42. Zhou D, Bousquet O, Lal T, Weston J and Scholkopf B (2004) Learning with local and global consistency. In: Proceedings of the 18th Annual Conference on Neural Information Processing Systems (NIPS), pp. 321–328

  43. Zhou D and Burges CJC (2007) Spectral clustering and transductive learning with multiple views. In: Proceedings of International Conference on Machine Learning (ICML), pp. 1159–1166

Download references

Acknowledgments

This work is supported by the National High Technology Research and Development Program of China (Project No. 2007AA01Z334), National Natural Science Foundation of China (Project No. 61321491 and 61272219), Innovation Fund of State Key Laboratory for Novel Software Technology (Project No. ZZKT2013A12).

Author information

Authors and Affiliations

  1. State Key Lab for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Kai Liu, Zhengxing Sun, Mofei Song & Bo Li

Authors
  1. Kai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengxing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mofei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhengxing Sun.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, K., Sun, Z., Song, M. et al. Iterative samples labeling for sketch recognition. Multimed Tools Appl 76, 12819–12852 (2017). https://doi.org/10.1007/s11042-016-3700-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3700-z

Keywords

Search

Navigation