Skip to main content
SpringerLink
Log in

Automatic visual pattern mining from categorical image dataset

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

We study in this paper the problem of visual pattern mining, which is to identify visually distinctive and semantically meaningful regions in images for solving various visual recognition tasks. Toward this goal, we propose a novel deep neural network architecture called PatternNet for discovering visual patterns that are both discriminative and representative. The proposed PatternNet leverages the filters in the last convolution layer of a convolutional neural network to find locally consistent visual patches, and by combining these filters we can effectively discover unique visual patterns. In addition, PatternNet can discover visual patterns efficiently without performing expensive image patch sampling, and this advantage provides an order of magnitude speedup compared to most other approaches. We evaluate the proposed PatternNet subjectively by showing randomly selected visual patterns which are discovered by our method and quantitatively by performing image classification with the identified visual patterns and comparing our performance with the current state-of-the-art. We also directly evaluate the quality of the discovered visual patterns by leveraging the identified patterns as proposed objects in an image and compare with other relevant methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. E.g., random images downloaded from Flickr, or random images selected from all the other categories.

References

  1. Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: ACM SIGMOD record, vol 22, pp 207–216. ACM

  2. Alexe B, Deselaers T, Ferrari V (2012) Measuring the objectness of image windows. IEEE Trans Pattern Anal Mach Intell 34(11):2189–2202

    Article  Google Scholar 

  3. Berg T, Belhumeur PN (2013) Poof: part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 955–962. IEEE

  4. Carreira J, Sminchisescu C (2010) Constrained parametric min-cuts for automatic object segmentation. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 3241–3248. IEEE

  5. Chai Y, Lempitsky V, Zisserman A (2013) Symbiotic segmentation and part localization for fine-grained categorization. In: 2013 IEEE international conference on computer vision (ICCV), pp 321–328. IEEE

  6. Chen G, Yang J, Jin H, Shechtman E, Brandt J, Han TX (2015) Selective pooling vector for fine-grained recognition. In: 2015 IEEE winter conference on applications of computer vision (WACV), pp 860–867. IEEE

  7. Doersch C, Gupta A, Efros AA (2013) Mid-level visual element discovery as discriminative mode seeking. In: Advances in neural information processing systems, pp 494–502

  8. Endres I, Hoiem D (2010) Category independent object proposals. In: Computer vision–ECCV 2010, pp 575–588. Springer

  9. Gavves E, Fernando B, Snoek CG, Smeulders AW, Tuytelaars T (2013) Fine-grained categorization by alignments. In: Proceedings of the IEEE international conference on computer vision, pp 1713–1720

  10. Gavves E, Fernando B, Snoek CG, Smeulders AW, Tuytelaars T (2014) Local alignments for fine-grained categorization. Int J Comput Vis 111(2):191–212

    Article  Google Scholar 

  11. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), pp 580–587. IEEE

  12. Hariharan B, Arbeláez P, Girshick R, Malik J (2015) Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 447–456

  13. Harzallah H, Jurie F, Schmid C (2009) Combining efficient object localization and image classification. In: 2009 IEEE 12th international conference on computer vision, pp 237–244. IEEE

  14. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: 2017 IEEE international conference on computer vision (ICCV), pp 2980–2988. IEEE

  15. Juneja M, Vedaldi A, Jawahar C, Zisserman A (2013) Blocks that shout: distinctive parts for scene classification. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 923–930. IEEE

  16. Krause J, Jin H, Yang J, Fei-Fei L (2015) Fine-grained recognition without part annotations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5546–5555

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  18. Li LJ, Su H, Fei-Fei L, Xing EP (2010) Object bank: a high-level image representation for scene classification & semantic feature sparsification. In: Advances in neural information processing systems, pp 1378–1386

  19. Li Q, Wu J, Tu Z (2013) Harvesting mid-level visual concepts from large-scale internet images. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 851–858. IEEE

  20. Li Y, Liu L, Shen C, van den Hengel A (2015) Mid-level deep pattern mining. In: CVPR, pp 971–980

  21. Lowe DG (1999) Object recognition from local scale-invariant features. In: The proceedings of the seventh IEEE international conference on computer vision, 1999, vol 2, pp 1150–1157. IEEE

  22. Parizi SN, Vedaldi A, Zisserman A, Felzenszwalb P (2014) Automatic discovery and optimization of parts for image classification. arXiv preprint arXiv:1412.6598

  23. Pu J, Jiang YG, Wang J, Xue X (2014) Which looks like which: exploring inter-class relationships in fine-grained visual categorization. In: Computer vision–ECCV 2014, pp 425–440. Springer

  24. Quattoni A, Torralba A (2009) Recognizing indoor scenes. In: IEEE conference on computer vision and pattern recognition, CVPR 2009, pp 413–420. IEEE

  25. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  26. Sermanet P, Frome A, Real E (2014) Attention for fine-grained categorization. arXiv preprint arXiv:1412.7054

  27. Shou Z, Gao H, Zhang L, Miyazawa K, Chang SF (2018) Autoloc: weakly supervised temporal action localization in untrimmed videos. In: ECCV, pp 162–179

  28. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  29. Singh S, Gupta A, Efros A (2012) Unsupervised discovery of mid-level discriminative patches. In: Computer vision-ECCV 2012, pp 73–86

  30. Sun J, Ponce J (2013) Learning discriminative part detectors for image classification and cosegmentation. In: 2013 IEEE international conference on computer vision (ICCV), pp 3400–3407. IEEE

  31. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  32. Uijlings JR, van de Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171

    Article  Google Scholar 

  33. Vedaldi A, Gulshan V, Varma M, Zisserman A (2009) Multiple kernels for object detection. In: 2009 IEEE 12th international conference on computer vision, pp 606–613. IEEE

  34. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Computer vision–ECCV 2014, pp 818–833. Springer

  35. Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Computer vision–ECCV 2014, pp 834–849. Springer

  36. Zhang W, Li H, Ngo CW, Chang SF (2014) Scalable visual instance mining with threads of features. In: Proceedings of the ACM international conference on multimedia, pp 297–306. ACM

Download references

Author information

Authors and Affiliations

  1. Microsoft Research, Redmond, WA, 98052, USA

    Hongzhi Li & Lei Zhang

  2. Columbia University, New York, NY, 10027, USA

    Joseph G. Ellis & Shih-Fu Chang

Authors
  1. Hongzhi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph G. Ellis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shih-Fu Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongzhi Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Ellis, J.G., Zhang, L. et al. Automatic visual pattern mining from categorical image dataset. Int J Multimed Info Retr 8, 35–45 (2019). https://doi.org/10.1007/s13735-018-0163-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13735-018-0163-1

Search

Navigation