Elsevier

Neurocomputing

Volume 173, Part 3, 15 January 2016, Pages 2041-2048
Neurocomputing

Brief Papers
A PMJ-inspired cognitive framework for natural scene categorization in line drawings

https://doi.org/10.1016/j.neucom.2015.09.046Get rights and content

Abstract

Humans׳ remarkable capacity on rapid natural scene categorization has been widely studied in neuroscience. Recently, a functional MRI (fMRI) study showed that in human brain, decoding of natural scenes from line drawings was very similar to those from color photographs. In this paper, based on recently proposed computational cognition model of Perception, Memory and Judgement (PMJ model), we investigate the computational model of line drawings and propose a PMJ-inspired cognitive framework for natural scene categorization in line drawings. The Ohio State University (OSU) dataset was used, which included 475 color photographs in six categories, i.e., beaches, city streets, forests, highways, mountains and offices, as well as 475 corresponding line drawings produced by trained artists. Experimental results show that our proposed cognitive framework achieves 48.4% recognition rate in leave-one-out cross-validation, which is much higher than fMRI-data-driven decoding accuracy in the visual-processing hierarchy (29% in V1, 27% in V2+VP, 26% in V4, 29% in PPA and 23% in RSC).

Introduction

It is well known that humans had remarkable capacity at perceiving and categorizing natural scenes, and the neural mechanisms of rapid natural scene categorization in human brain had been widely investigated (e.g., [1]). It was reported [2] that in human brain, scene category information can be encoded in patterns of functional MRI (fMRI) activity in the parahippocampal place area (PPA), the retrosplenial cortex (RSC), the lateral occipital complex (LOC), and the primary visual cortex (V1). Visual areas V2, VP and V4 are also interested since they build representations based on V1 information.

Recently, a study on natural scene categorization by reducing the scenes to mere lines was presented in [3]. In this study, color photographs in six categories (beaches, city streets, forests, highways, mountains and offices) were collected. Then line drawings were created by trained artists who traced those contours in the photographs that best captured the scene. We call these data the Ohio State University (OSU) dataset. An elaborated experiment was performed in [3] in which color photographs and line drawings in OSU dataset were presented alternatively to participants. Functional MRI images were recorded when participants passively viewed the OSU dataset. Experimental results showed that despite the marked difference in scene statistics and consideration degradation of information, scene category can be decoded from fMRI data for line drawings just as well as from activity for color photographs.

In this paper, we propose a computational cognitive framework that investigates how a computer program can be used to simulate the human vision system that categorizes natural scenes from line drawings. Our work is based on a recently proposed computational cognition model of Perception, Memory and Judgement (PMJ model) [4]. In the perception stage, we compute a saliency map on color photograph, and map the salient region onto the corresponding line drawing. In the memory stage, we apply a local histogram with circular bins to extract feature instantiations from perceived line drawings. The collected feature instantiations are clustered in a bag-of-word model and forms a visual vocabulary. Then each line drawing is presented by a feature vector that is a set of visual words in the vocabulary. In the judgement stage, a SVM classifier is applied and optimal parameters are trained from feature vectors represented line drawings in six categories in leave-one-out cross-validation. Experimental results show that our proposed cognitive framework in machine vision has a consistent performance in the line with the pattern of brain activity measured with fMRI in observers who viewed line drawings of natural scenes. Our PMJ-inspired cognitive framework achieves 48.4% recognition rate in OSU dataset, which is much higher than fMRI-data-driven decoding accuracy in the visual-processing hierarchy (29% in V1, 27% in V2+VP, 26% in V4, 29% in PPA and 23% in RSC).

Section snippets

Related work

A line drawing is usually referred to as a set of sparse, simple two-dimensional feature lines without hatching or stippling for shading/tone effects [5]. Humans have an innate ability to perceive, recognize and interpret line drawings, e.g., children׳s sketches and line arts by ink on paper. Line drawings had been used for studying objects and scenes in human cognition for more than three decades [6], [7]. Recently a fMRI study found that the neural activation in response to line drawings was

PMJ-inspired cognitive framework

Consensus had been reached in psychological research that most cognitive processes consist of several successive processing stages [22]. In our study, we apply the PMJ model [4] that partitions the cognitive process into the stages of perception, memory and judgement, corresponding to the stages of analysis, modeling and decision in the computation process. In the stage of perception, through pre-attention selection and selective attention, the cognitive load of cognition system is reduced and

Experiments

To the authors׳ best knowledge, the proposed method in this paper is the first computer program that can recognize the category of nature scenes in which a given line drawing belongs to. A previous work [3] that recognized categories of nature scenes in line drawings was based on the fMRI data taken from human observers. The same SVM classifier as in ours was used in [3]. Therefore, when we compared our recognition results with those in [3], the difference in performance was due to the

Conclusion

Natural scene categorization had drawn considerable attention from neuroscience and computer science. In this paper, a PMJCF computational model is proposed to recognize the categories of natural scenes based on line drawings. PMJCF consists of three stages of computation. At the first stage of perception, a saliency map is extracted from color photographs and applied to obtain perceived line drawings. At the second stage of memory, a vocabulary of visual words is obtained by clustering

Acknowledgments

The authors would like to thank Dr. Dirk Bernhardt-Walther for providing us the Ohio State University (OSU) dataset. This work was supported by the Natural Science Foundation of China (61322206, 61379095, 61272228, 61472138), the Beijing Natural Science Foundation (4152055), the Open Projects Program of National Laboratory of Pattern Recognition (201306295) and the National Basic Research Program of China (2011CB302200). The work of Y.J. Liu was supported in part by Tsinghua national laboratory

Minjing Yu received her B.Eng. degree from Wuhan University, China, and she is now a Ph.D. student at Department of Computer Science and Technology, Tsinghua University, China. Her research interests include image processing, computer graphics and cognitive science.

References (24)

  • I. Biederman et al.

    Scene perceptiondetecting and judging objects undergoing relational violations

    Cogn. Psychol.

    (1982)
  • C.-C. Yu et al.

    A global energy optimization framework for 2.1d sketch extraction from monocular images

    Graph. Models

    (2014)
  • L. Zeng et al.

    Sketch2jewelry: semantic feature modeling for sketch-based jewelry design

    Comput. Graph.

    (2014)
  • M.V. Peelen et al.

    Neural mechanisms of rapid natural scene categorization in human visual cortex

    Nature

    (2009)
  • D.B. Walther et al.

    Natural scene categories revealed in distributed patterns of activity in the human brain

    J. Neurosci.

    (2009)
  • D.B. Walther et al.

    Simple line drawings suffice for functional mri decoding of natural scene categories

    Proc. Natl. Acad. Sci. U.S.A.

    (2011)
  • X. Fu et al.

    A computational cognition model of perception, memory and judgment

    Sci. China Inf. Sci.

    (2014)
  • Y.-J. Liu, M. Yu, Q. Fu, W. Chen, Y. Liu, L. Xie, Cognitive mechanism related to line drawings and its applications in...
  • J.G. Snodgrass et al.

    A standardized set of 260 picturesnorms for name agreement, image agreement, familiarity, and visual complexity

    J. Exp. Psychol.: Hum. Learn. Memory

    (1980)
  • Q. Fu, Y.-J. Liu, W. Chen, X. Fu, The time course of natural scene categorization in human brain: simple line-drawings...
  • Y. Cao, C. Wang, L. Zhang, L. Zhang, Edgel index for large-scale sketch-based image search, in: 2011 IEEE Conference on...
  • C.-X. Ma et al.

    Sketch-based annotation and visualization in video authoring

    IEEE Trans. Multimed.

    (2012)
  • Minjing Yu received her B.Eng. degree from Wuhan University, China, and she is now a Ph.D. student at Department of Computer Science and Technology, Tsinghua University, China. Her research interests include image processing, computer graphics and cognitive science.

    Yong-Jin Liu received his B.Eng. degree from Tianjin University, China, in 1998, and his M.Phil. and Ph.D. degrees from the Hong Kong University of Science and Technology, Hong Kong, China, in 2000 and 2004, respectively. He is now an Associate Professor with the Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, China. His research interests include computational geometry, multimedia, computer graphics and computer-aided design. For more information, visit http://cg.cs.tsinghua.edu.cn/people/Yongjin/yongjin.htm.

    Su-Jing Wang received the Master׳s degree from the Software College of Jilin University, China, in 2007. He received the Ph.D. degree from the College of Computer Science and Technology of Jilin University in 2012. He is a Postdoctoral Researcher in Institute of Psychology, Chinese Academy of Sciences. He is One of the Ten Selectees of the Doctoral Consortium at International Joint Conference on Biometrics 2011. He was named Chinese Hawkin by the Xinhua News Agency. His current research interests include pattern recognition, computer vision and machine learning. For more information, visit http://sujingwang.name.

    Qiufang Fu received her Ph.D. degree from Institute of Psychology, Chinese Academy of Sciences, in 2006. She is now an Associate Professor in Institute of Psychology, CAS, with interests in implicit learning and unconscious knowledge. She intends to explore the neural and cognitive mechanisms responsible for the dissociation of implicit and explicit processes.

    Xiaolan Fu received the B.S. and M.S. degrees in psychology from Peking University, Peking, China, in 1984 and 1987, respectively, and the Ph.D. degree from the Institute of Psychology, Chinese Academy of Sciences, Beijing, China, in 1990. Currently she is the Director of Institute of Psychology, Chinese Academy of Sciences and Vice Director of State Key Laboratory of Brain and Cognitive Science. Her research interests focus on visual and computational cognition, including attention and perception, learning and memory, and affective computing. She serves as an Associate Editor of PsyCH Journal and journal of Protein & Cell.

    View full text