Abstract
In feature learning field, many methods are inspired by advances in neuroscience. Among them, neural network and sparse coding have been broadly studied. Predictive sparse decomposition (PSD) is a practical variant of these two methods. It trains a neural network to estimate the sparse codes. After training, the neural network is fine-tuned to achieve higher performance on object recognition tasks. It is widely believed that introducing discriminative information can make the features more useful for classification task. Hence, in this work, we propose applying the task-driven dictionary learning framework to the PSD and demonstrate that this new model can be optimized by the stochastic gradient descent (SGD) algorithm. Before our work, the semi-supervised auto-encoder framework has already been proposed to guide neural network to extract discriminative representations. But it does not improve the classification performance of neural network. In the experiments, we compare the proposed method with the semi-supervised auto-encoder method. The performance of PSD is used as the baseline for these two methods. On the MNIST and USPS datasets, our method can generate more discriminative and predictable sparse codes than other methods. Furthermore, the recognition accuracy of neural network can be improved.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Wei H, Dong Z. V4 neural network model for shape-based feature extraction and object discrimination. Cogn Comput. 2015:1–10.
Gros C. Cognitive computation with autonomously active neural networks: An emerging field. Cogn Comput. 2009;1(1):77–90.
Kavukcuoglu K, Marc’Aurelio R, LeCun Y. Fast inference in sparse coding algorithms with applications to object recognition. CoRR, abs/1010.3467. 2010.
Kavukcuoglu K, Ranzato MA, Fergus R, LeCun Y. Learning invariant features through topographic filter maps. 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 Jun 2009, Miami, Florida, USA; 2009. pp. 1605–1612.
Kavukcuoglu K, Sermanet P, Boureau Y-L, Gregor K, Mathieu M, LeCun Y. Learning convolutional feature hierarchies for visual recognition. Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 Dec 2010, Vancouver, British Columbia, Canada; 2010. pp. 1090–1098.
He B, Xu D, Nian R, Heeswijk M, Yu Q, Yoan M, Amaury L. Fast face recognition via sparse coding and extreme learning machine. Cogn Comput. 2014;6(2):264–277.
Bengio Y, Courville AC, Pascal V. 2012. Unsupervised feature learning and deep learning: a review and new perspectives. CoRR, abs/1206.5538.
Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia Canada, Dec 4-7 2006; 2006. pp. 153–160.
Mairal J, Bach FR, dictionary JP. Task-driven learning. IEEE Trans Pattern Anal Mach Intell. 2012;34 (4):791–804.
Mallat S, Zhang Z. Matching pursuits with time-frequency dictionaries. IEEE Trans Signal Process. 1993; 41(12):3397–3415.
Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc. 1994;58(1):267–288.
Chen SS, Donoho DL, Saunders MA. Atomic decomposition by basis pursuit. SIAM Rev. 2001;43(1): 129–159.
Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc. 2005:67.
Shalev-Shwartz S, Tewari A. Stochastic methods for l 1-regularized loss minimization. J Mach Learn Res. 2011;12:1865–1892.
Efron B, Tibshirani R. Least angle regression. Ann Stat. 2004;32(2):2004.
Mallat S. A wavelet tour of signal processing (2.ed.). Academic Press. 1999.
Starck J-L, Candès EJ, Donoho DL. The curvelet transform for image denoising. IEEE Trans Image Process. 2002;11(6):670–684.
Do MN, Vetterli M. The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans Image Process. 2005;14(12):2091–2106.
Aharon M, Elad M, Brucstein A. k –svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process. 2006;54(11):4311–4322.
Olshausen BA, Field DJ. Sparse coding with an overcomplete basis set: A strategy employed by v1?. Vis Res. 1997;37(23):3311–3325.
Lee H, Battle A, Raina R, Ng AY. Efficient sparse coding algorithms. Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia Canada, Dec 4-7 2006; 2006. pp. 801–808.
Ba J, Caruana R. Do deep nets really need to be deep? Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada; 2014. pp. 2654– 2662.
Fuchs J-J. Recovery of exact sparse representations in the presence of bounded noise. IEEE Trans Inf Theory. 2005;51(10):3601–3608.
Swersky K, Ranzato MA, Buchman D, Marlin BM, Freitas N. On autoencoders and score matching for energy based models. Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, Jun 28 - Jul 2, 2011; 2011. pp. 1201–1208.
Liu W, Tao D, Cheng J, Tang Y. Multiview hessian discriminative sparse coding for image annotation. Comput Vis Image Underst. 2014;118:50–60.
Xum C, Tao D, Xu C. Robust extreme multi-label learning. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA August 13-17, 2016; 2016. pp. 1275–1284.
Liu W, Zha Z-J, Wang Y, Lu K, Tao D. P-laplacian regularized sparse coding for human activity recognition. IEEE Trans Ind Electron 2016;63(8):5120–5129.
Qiao M, Liu L, Yu J, Xu C, Tao D. Diversified dictionaries for multi-instance learning. Pattern Recognition. 2016.
Acknowledgements
This study was funded by the National Natural Science Foundation of China (NSFC) under Grants No. 61273136, No. 61573353 and No. 61533017.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical Approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
This work is supported by National Natural Science Foundation of China (NSFC) under Grants No. 61273136, No. 61573353 and No. 61533017.
Rights and permissions
About this article
Cite this article
Lv, L., Zhao, D. & Deng, Q. A Semi-Supervised Predictive Sparse Decomposition Based on Task-Driven Dictionary Learning. Cogn Comput 9, 115–124 (2017). https://doi.org/10.1007/s12559-016-9438-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-016-9438-0