Skip to main content
SpringerLink
Log in

Image categorization using a semantic hierarchy model with sparse set of salient regions

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Image categorization in massive image database is an important problem. This paper proposes an approach for image categorization, using sparse set of salient semantic information and hierarchy semantic label tree (HSLT) model. First, to provide more critical image semantics, the proposed sparse set of salient regions only at the focuses of visual attention instead of the entire scene was formed by our proposed saliency detection model with incorporating low and high level feature and Shotton’s semantic texton forests (STFs) method. Second, we also propose a new HSLT model in terms of the sparse regional semantic information to automatically build a semantic image hierarchy, which explicitly encodes a general to specific image relationship. And last, we archived image dataset using image hierarchical semantic, which is help to improve the performance of image organizing and browsing. Extension experimental results showed that the use of semantic hierarchies as a hierarchical organizing framework provides a better image annotation and organization, improves the accuracy and reduces human’s effort.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Griffiths T, Jordan M, Tenenbaum J, Blei DM. Hierarchical topic models and the nested chinese restaurant process. Advances in Neural Information Processing Systems, 2004, 16: 106–114

    Google Scholar 

  2. Bannour H, Hudelot C. Towards ontologies for image interpretation and annotation. In: Proceedings of the 9th International Workshop on Content-Based Multimedia Indexing (CBMI). 2011, 211–216

    Chapter  Google Scholar 

  3. Tousch A M, Herbin S, Audibert J Y. Semantic hierarchies for image annotation: a survey. Pattern Recognition, 2012, 45(1): 333–345

    Article  Google Scholar 

  4. Marszalek M, Schmid C. Semantic hierarchies for visual object recognition. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2007, 1–7

    Chapter  Google Scholar 

  5. Wei X Y, Ngo C W. Ontology-enriched semantic space for video search. In: Proceedings of the 15th International Conference on Multimedia. 2007, 981–990

    Chapter  Google Scholar 

  6. Deng J, Dong W, Socher R, Li L J, Li K, Fei-Fei L. Imagenet: a largescale hierarchical image database. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009, 248–255

    Chapter  Google Scholar 

  7. Snow R, Jurafsky D, Ng A Y. Semantic taxonomy induction from heterogenous evidence. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. 2006, 801–808

    Google Scholar 

  8. Miller G A. Wordnet: a lexical database for English. Communications of the ACM, 1995, 38(11): 39–41

    Article  Google Scholar 

  9. Jin Y, Khan L, Wang L, Awad M. Image annotations by combining multiple evidence & wordnet. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. 2005, 706–715

    Chapter  Google Scholar 

  10. Joshi D, Datta R, Zhuang Z, Weiss W, Friedenberg M, Li J, Wang J Z. Paragrab: a comprehensive architecture for web image management and multimodal querying. In: Proceedings of the 32nd International Conference on Very Large Data Bases. 2006, 1163–1166

    Google Scholar 

  11. Datta R, Ge W, Li J, Wang J Z. Toward bridging the annotationretrieval gap in image search. IEEE MultiMedia, 2007, 14(3): 24–35

    Article  Google Scholar 

  12. Torralba A, Fergus R, Freeman W T. 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(11): 1958–1970

    Article  Google Scholar 

  13. Sivic J, Russell B C, Zisserman A, Freeman WT, Efros A A. Unsupervised discovery of visual object class hierarchies. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1–8

    Chapter  Google Scholar 

  14. Bart E, Porteous I, Perona P, Welling M. Unsupervised learning of visual taxonomies. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1–8

    Chapter  Google Scholar 

  15. Yao B Z, Yang X, Lin L, Lee M W, Zhu S C. I2t: image parsing to text description. Proceedings of the IEEE, 2010, 98(8): 1485–1508

    Article  Google Scholar 

  16. Griffin G, Perona P. Learning and using taxonomies for fast visual categorization. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1–8

    Chapter  Google Scholar 

  17. Marszaek M, Schmid C. Constructing category hierarchies for visual recognition. In: Proceedings of the 10th European Conference on Computer Vision. 2008, 479–491

    Google Scholar 

  18. Ahuja N, Todorovic S. Learning the taxonomy and models of categories present in arbitrary images. In: Proceedings of 11th IEEE International Conference on Computer Vision (ICCV). 2007, 1–8

    Google Scholar 

  19. Li L J, Wang C, Lim Y, Blei D M, Fei-Fei L. Building and using a semantivisual image hierarchy. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2010, 3336–3343

    Chapter  Google Scholar 

  20. Fan J, Gao Y, Luo H. Hierarchical classification for automatic image annotation. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2007, 111–118

    Chapter  Google Scholar 

  21. Fan J, Gao Y, Luo H. Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. IEEE Transactions on Image Processing, 2008, 17(3): 407–426

    Article  MathSciNet  Google Scholar 

  22. Fan J, Gao Y, Luo H, Jain R. Mining multilevel image semantics via hierarchical classification. IEEE Transactions on Multimedia, 2008, 10(2): 167–187

    Article  Google Scholar 

  23. Wu L, Hua X S, Yu N, Ma W Y, Li S. Flickr distance: a relationship measure for visual concepts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 863–875

    Article  Google Scholar 

  24. Bannour H, Hudelot C. Building semantic hierarchies faithful to image semantics. In: Proceedings of the 18th International Conference on Advances in Multimedia Modeling. 2012, 4–15

    Chapter  Google Scholar 

  25. Moosmann F, Nowak E, Jurie F. Randomized clustering forests for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(9): 1632–1646

    Article  Google Scholar 

  26. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

    Article  Google Scholar 

  27. Wu L, Hu Y, Li M, Yu N, Hua X S. Scale-invariant visual language modeling for object categorization. IEEE Transactions on Multimedia, 2009, 11(2): 286–294

    Article  Google Scholar 

  28. Bannour H, Hudelot C. Hierarchical image annotation using semantic hierarchies. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 2012, 2431–2434

    Google Scholar 

  29. Deng J, Berg A C, Li K, Fei-Fei L. What does classifying more than 10 000 image categories tell us? In: Proceedings of the 11th European Conference on Computer Vision. 2010, 71–84

    Google Scholar 

  30. Theeuwes J. Top-down and bottom-up control of visual selection. Acta Psychologica, 2010, 135(2): 77–99

    Article  Google Scholar 

  31. Hou X, Zhang L. Saliency detection: a spectral residual approach. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2007, 1–8

    Chapter  Google Scholar 

  32. Harel J, Koch C, Perona P. Graph-based visual saliency. Advances in Neural Information Processing Systems, 2006, 545–552

    Google Scholar 

  33. Achanta R, Hemami S, Estrada F, Susstrunk S. Frequency-tuned salient region detection. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009, 1597–1604

    Chapter  Google Scholar 

  34. Goferman S, Zelnik-Manor L, Tal A. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10): 1915–1926

    Article  Google Scholar 

  35. Zheng Y, Liu C p, Liu G, Wang Z h. Saliency detection based on inhibition of blur regions. Microelectronics & Computer, 2012, 29(3): 84–88

    Google Scholar 

  36. Yang Z, Chunping L, Zhaohui W, Yi J, Shengrong G. A saliency detection model based on multi-feature fusion. In: Proceedings of the 7th International Conference on Computational Intelligence and Security (CIS). 2011, 1062–1066

    Google Scholar 

  37. Sivic J, Russell B C, Efros A A, Zisserman A, Freeman W T. Discovering objects and their location in images. In: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV). 2005, 370–377

    Google Scholar 

  38. Li Z, Wang Y, Chen J, Xu J, Larid J. Image topic discovery with saliency detection. Journal of Machine Learning Research, 2003, 3: 993–1022

    Google Scholar 

  39. Wu L, Hoi S C. Enhancing bag-of-words models with semantics-preserving metric learning. IEEE Multimedia Magazine, 2011, 18(1): 24–37

    Article  Google Scholar 

  40. Wu L, Hoi S C, Yu N. Semantics-preserving bag-of-words models and applications. IEEE Transactions on Image Processing, 2010, 19(7): 1908–1920

    Article  MathSciNet  Google Scholar 

  41. Shotton J, Johnson M, Cipolla R. Semantic texton forests for image categorization and segmentation. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1–8

    Chapter  Google Scholar 

  42. Friston K, Kiebel S. Cortical circuits for perceptual inference. Neural Networks, 2009, 22(8): 1093–1104

    Article  MathSciNet  Google Scholar 

  43. Shotton J, Winn J, Rother C, Criminisi A. The MSRC 21-class object recognition database, 2006

    Google Scholar 

  44. Everingham M, Van Gool L, Williams C K, Winn J, Zisserman A. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 2010, 88(2): 303–338

    Article  Google Scholar 

  45. Shotton J, Winn J, Rother C, Criminisi A. Textonboost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Computer Vision, 2009, 81(1): 2–23

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Soochow University, Suzhou, 215006, China

    Chunping Liu & Shengrong Gong

  2. The Second Hospital of Nanjing, Nanjing, 210003, China

    Yang Zheng

Authors
  1. Chunping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengrong Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunping Liu.

Additional information

Chunping Liu received her PhD degree in pattern recognition and artificial intelligence from Nanjing University of Science & Technology in 2002. She was a visiting scholar in computer vision lab of University of Central Florida from 2010 to 2011. She is now an associate professor of computer science, pattern recognition and image processing at the School of Computer Science & Technology in Soochow University. Her research interests include computer vision, image analysis and recognition, in particular in the domains of visual saliency detection, object detection and recognition, and scene understanding. She has published more than 60 refereed journal articles and conference proceedings on image analysis, computer vision, and pattern recognition.

Yang Zheng received her MS degree at computer application technology from the School of Computer Science and Technology, Soochow University in 2012. She is now an engineer of the information center in the second hospital of Nanjing. Her interests are image processing and analysis.

Shengrong Gong received his MS degree from Harbin Institute of Technology in 1993 and PhD degree from Beihang University in 2001. He is a professor and doctoral supervisors of the School of Computer Science and Technology, Soochow University. Currently he is a senior member of Chinese computer society, editors of communication journal, virtual reality professional of Chinese Society of image and graphics. He acted as chairman for 2010–2011 YOCSEF of the Academic Committee of Suzhou sub-forum. He got twice award of the Scientific and Technological Progress, and has published more than 100 academic articles. His research interests are image and video process, pattern recognition and computer vision.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, C., Zheng, Y. & Gong, S. Image categorization using a semantic hierarchy model with sparse set of salient regions. Front. Comput. Sci. 7, 838–851 (2013). https://doi.org/10.1007/s11704-013-2410-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-013-2410-1

Keywords

Search

Navigation