Cited By
View all- Prathap PJayakumari J(2023)An improved neural network-based saliency detection scheme for light field imagesMultimedia Tools and Applications10.1007/s11042-023-17683-x83:19(56549-56567)Online publication date: 12-Dec-2023
Light Field Saliency Detection Based on Multi-modal Fusion
Authors: 
Ben Jiang, Dan Xu, Jinlong ShiAuthors Info & Claims
Ben Jiang, Dan Xu, Jinlong ShiAuthors Info & ClaimsPages 253 - 260
Abstract
Compared with RGB images, light field images contain more abundant visual information, which is helpful to accurately detect salient objects in complex scenes. However, most of the existing light field saliency detection methods use single light field data or do not fully consider the differences and complementarities between different light field data, resulting in insufficient multi-modal fusion. To address these issues, a multi-modal feature fusion network is proposed, which makes full use of the rich visual information in the light field images to realize the accurate saliency object detection. The proposed network consists of two parallel subnets, which are used to process the micro-lens image array and all-foucs image respectively. Then the light field refinement module is used to refine the feature map extracted from the micro-lens array stream, and finally the multi-modal feature fusion is realized by the light field attention module to predict saliency objects more accurately. In order to verify the effectiveness of proposed method, extensive comparison with several existing light field saliency detection algorithms is carried on both Lytro-Illum and LFSD datasets. Experimental results show that the proposed method is superior to others in all evaluation metrics on Lytro-Illum dataset, and has desired generalization abilities on LFSD dataset.
References
[1]
Borji A, Frintrop S, Sihite D N, Adaptive object tracking by learning background context [C]// 2012 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, 2012: 23-30.
[2]
Ren S, He K, Girshick R, Faster r-cnn: Towards real-time object detection with region proposal networks [J]. Advances in neural information processing systems, 2015, 28.
[3]
Shafieyan F, Karimi N, Mirmahboub B, Image retargeting using depth assisted saliency map [J]. Signal Processing: Image Communication, 2017, 50: 34-43.
[4]
Zhao Z, Xie X, Wang C, ROI-CSNet: Compressive sensing network for ROI-aware image recovery [J]. Signal Processing: Image Communication, 2019, 78: 113-124.
[5]
Hou Q, Cheng M M, Hu X, Deeply supervised salient object detection with short connections [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3203-3212.
[6]
Kong Y, Wang L, Liu X, Pattern mining saliency [C]// European Conference on Computer Vision. Springer, Cham, 2016: 583-598.
[7]
Li X, Zhao L, Wei L, Deepsaliency: Multi-task deep neural network model for salient object detection [J]. IEEE transactions on image processing, 2016, 25(8): 3919-3930.
[8]
Han J, Chen H, Liu N, CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion [J]. IEEE transactions on cybernetics, 2017, 48(11): 3171-3183.
[9]
Ren J, Gong X, Yu L, Exploiting global priors for RGB-D saliency detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2015: 25-32.
[10]
Li N, Ye J, Ji Y, Saliency detection on light field [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014: 2806-2813.
[11]
Piao Y, Li X, Zhang M, Saliency detection via depth-induced cellular automata on light field [J]. IEEE Transactions on Image Processing, 2019, 29: 1879-1889.
[12]
Piao Y, Rong Z, Zhang M, Exploit and replace: An asymmetrical two-stream architecture for versatile light field saliency detection [C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 11865-11873.
[13]
Piao Y, Li X, Zhang M. Depth-induced cellular automata for light field saliency [C]// Frontiers in Optics. Optica Publishing Group, 2018: FTh3E. 3.
[14]
Wang H, Yan B, Wang X, Accurate saliency detection based on depth feature of 3D images [J]. Multimedia Tools and Applications, 2018, 77(12): 14655-14672.
[15]
Liu T, Yuan Z, Sun J, Learning to detect a salient object [J]. IEEE Transactions on Pattern analysis and machine intelligence, 2010, 33(2): 353-367.
[16]
Yang C, Zhang L, Lu H, Saliency detection via graph-based manifold ranking [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2013: 3166-3173.
[17]
Zhao R, Ouyang W, Li H, Saliency detection by multi-context deep learning [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1265-1274.
[18]
Zhang X, Wang T, Qi J, Progressive attention guided recurrent network for salient object detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 714-722.
[19]
Wang L, Wang L, Lu H, Saliency detection with recurrent fully convolutional networks [C]// European conference on computer vision. Springer, Cham, 2016: 825-841.
[20]
Ren J, Gong X, Yu L, Exploiting global priors for RGB-D saliency detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2015: 25-32.
[21]
Lang C, Nguyen T V, Katti H, Depth matters: Influence of depth cues on visual saliency [C]// European conference on computer vision. Springer, Berlin, Heidelberg, 2012: 101-115.
[22]
Gupta S, Girshick R, Arbeláez P, Learning rich features from RGB-D images for object detection and segmentation [C]// European conference on computer vision. Springer, Cham, 2014: 345-360.
[23]
Li Z, Gan Y, Liang X, Lstm-cf: Unifying context modeling and fusion with lstms for rgb-d scene labeling [C]// European conference on computer vision. Springer, Cham, 2016: 541-557.
[24]
Li N, Sun B, Yu J. A weighted sparse coding framework for saliency detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 5216-5223.
[25]
Zhang J, Wang M, Gao J, Saliency detection with a deeper investigation of light field [C]// Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015.
[26]
Wang T, Piao Y, Li X, Deep learning for light field saliency detection [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 8838-8848.
[27]
Zhang M, Li J, Wei J, Memory-oriented decoder for light field salient object detection [J]. Advances in neural information processing systems, 2019, 32.
[28]
Zhang J, Liu Y, Zhang S, Light field saliency detection with deep convolutional networks [J]. IEEE Transactions on Image Processing, 2020, 29: 4421-4434.
[29]
Chen L C, Papandreou G, Kokkinos I, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(4): 834-848.
[30]
Zhang M, Ji W, Piao Y, LFNet: Light field fusion network for salient object detection [J]. IEEE Transactions on Image Processing, 2020, 29: 6276-6287.
[31]
Everingham M, Eslami S M, Van Gool L, The pascal visual object classes challenge: A retrospective [J]. International journal of computer vision, 2015, 111(1): 98-136.
[32]
Liu N, Han J, Yang M H. Picanet: Learning pixel-wise contextual attention for saliency detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3089-3098.
[33]
Zhang P, Wang D, Lu H, Amulet: Aggregating multi-level convolutional features for salient object detection [C]// Proceedings of the IEEE international conference on computer vision. 2017: 202-211.
[34]
Zhang P, Liu W, Lu H, Salient object detection by lossless feature reflection [J]. arXiv preprint arXiv: 1802.06527, 2018.
[35]
Zhang P, Liu W, Lei Y, Hyperfusion-Net: Hyper-densely reflective feature fusion for salient object detection [J]. Pattern Recognition, 2019, 93: 521-533.
[36]
Zhang J, Wang M, Lin L, Saliency detection on light field: A multi-cue approach [J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2017, 13(3): 1-22.
Recommendations
Object-level saliency detection with color attributes
Recently, saliency detection has been becoming a popular topic in computer vision. In this paper we propose an object-level saliency detection algorithm which explicitly explores bottom-up visual attention and objectness cues. Firstly, some category-...
Exemplar-based image saliency and co-saliency detection
AbstractImage saliency and co-saliency detection that aim to detect salient objects in an image or common salient objects in a group of images are import in computer vision. Researchers often treat saliency and co-saliency as two separate ...
Saliency Detection on Light Field: A Multi-Cue Approach
Saliency detection has recently received increasing research interest on using high-dimensional datasets beyond two-dimensional images. Despite the many available capturing devices and algorithms, there still exists a wide spectrum of challenges that ...
Comments
Information & Contributors
Information
Published In
October 2022
1999 pages
ISBN:9781450397148
DOI:10.1145/3573428
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 March 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
EITCE 2022
EITCE 2022: 2022 6th International Conference on Electronic Information Technology and Computer Engineering
October 21 - 23, 2022
Xiamen, China
Acceptance Rates
Overall Acceptance Rate 508 of 972 submissions, 52%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 25Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)2
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
Cited By
View all- Prathap PJayakumari J(2023)An improved neural network-based saliency detection scheme for light field imagesMultimedia Tools and Applications10.1007/s11042-023-17683-x83:19(56549-56567)Online publication date: 12-Dec-2023
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format