Skip to main content

Advertisement

SpringerLink
Log in

Dynamic sparse coding for sparse time-series modeling via first-order smooth optimization

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Sparse coding, often called dictionary learning, has received significant attention in the fields of statistical machine learning and signal processing. However, most approaches assume iid data setup, which can be easily violated when the data retains certain statistical structures such as sequences where data samples are temporally correlated. In this paper we formulate a novel dynamic sparse coding problem, and propose an efficient algorithm that enforces smooth dynamics for the latent state vectors (codes) within a linear dynamic model while imposing sparseness of the state vectors. We overcome the added computational overhead originating from smooth dynamic constraints by adopting the recent first-order smooth optimization technique, adjusted for our problem instance. We demonstrate the improved prediction performance of our approach over the conventional sparse coding on several interesting real-world problems including financial asset return data forecasting and human motion estimation from silhouette videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. It should not incur any theoretical or practical concern because one can pre-scale the entries of B in accordance with the chosen values of c.

  2. They are publicly available from: http://www.mysmu.edu.sg/faculty/chhoi/olps/datasets.html, and the detailed description can be found in [21].

References

  1. Bar-Shalom Y, Li XR (1993) Estimation and tracking: principles, techniques, and software. Artech House, Boston

    MATH  Google Scholar 

  2. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

  3. Bengio Y (2009) Learning deep architectures for ai. Found Trends Mach Learn 2(1):1–127

    Article  Google Scholar 

  4. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Optim 3(1):1–122

    MATH  Google Scholar 

  5. Chen SS, Donoho DL, Saunders MA (1998) Atomic decomposition by basis pursuit. SIAM J Sci Comput 20(1):33–61

    Article  MathSciNet  Google Scholar 

  6. Cho K, Merrienboer BV, Gulcehre C, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv:1406.1078

  7. Chung J, Gulcehre C, Cho K, Bengio Y (2015) Gated feedback recurrent neural networks. International Conference on Machine Learning (ICML)

  8. d’Aspremont A, Banerjee O, Ghaoui LE (2008) First-order methods for sparse covariance selection. SIAM J Matrix Anal Appl 30(56):367–385

    MathSciNet  MATH  Google Scholar 

  9. Deb K, Padhye N (2014) Enhancing performance of particle swarm optimization through an algorithmic link with genetic algorithms. Comput Optim Appl 57(3):761–794

    Article  MathSciNet  Google Scholar 

  10. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc 39:1–38

    MathSciNet  MATH  Google Scholar 

  11. Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306

    Article  MathSciNet  Google Scholar 

  12. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Statist 32(2):407–499

    Article  MathSciNet  Google Scholar 

  13. Fu W, Wang J, Lu H, Ma S (2013) Dynamic scene understanding by improved sparse topical coding. Pattern Recogn 46(7):1841–1850

    Article  Google Scholar 

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  15. Hsaio WH, Liu CL, Wu WL (2017) Locality-constrained max-margin sparse coding. Pattern Recogn 65:285–295

    Article  Google Scholar 

  16. Huang Z, Liu Y, Li X, Li J (2015) An adaptive bimodal recognition framework using sparse coding for face and ear. Pattern Recogn Lett 53(1):69–76

    Article  Google Scholar 

  17. Kim SJ, Koh K, Lustig M, Boyd S, Gorinevsky D (2007) An interior-point method for large-scale l1-regularized least squares. IEEE J Select Topics Signal Process 1(4):606–617

    Article  Google Scholar 

  18. Lee H, Battle A, Raina R, Ng AY (2007) Efficient sparse coding algorithms. Advances in Neural Information Processing Systems (NIPS) 19

  19. Lemaréchal C, Sagastizábal C (1997) Practical aspects of the Moreau–Yosida regularization: theoretical preliminaries. SIAM J Optim 7(2):367–385

    Article  MathSciNet  Google Scholar 

  20. Lewicki MS, Sejnowski TJ (2000) Learning overcomplete representations. Neural Comput 12(2):337–365

    Article  Google Scholar 

  21. Li B, Hoi SC (2012) On-line portfolio selection: a survey. Tech. rep., Nanyang Technological University

  22. Li Y, Ngom A (2013) Sparse representation approaches for the classification of high-dimensional biological data. BMC Syst Biol 6:1–14

    Article  Google Scholar 

  23. Liu S, Liu M, Yang Z (2017) Sparse coding based orientation estimation for latent fingerprints. Pattern Recogn 67:164–176

    Article  Google Scholar 

  24. Lu Z, Wang L (2015) Noise-robust semi-supervised learning via fast sparse coding. Pattern Recogn 48 (2):605–612

    Article  MathSciNet  Google Scholar 

  25. Mairal J, Bach F, Ponce J, Sapiro G (2009) Online dictionary learning for sparse coding. In: Proceedings of the 26th annual international conference on machine learning

  26. Nesterov Y (2005) Smooth minimization of non-smooth functions. Math Program 103(1):127–152

    Article  MathSciNet  Google Scholar 

  27. Nesterov Y, Nemirovsky A (1994) Interior-point polynomial methods in convex programming. Studies in applied mathematics. SIAM, Philadelphia, p 13

    Google Scholar 

  28. Olshausen B, Field D (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14(4):481–487

    Article  Google Scholar 

  29. Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381:607–609

    Article  Google Scholar 

  30. Padhye N, Bhardawaj P, Deb K (2013) Improving differential evolution through a unified approach. J Glob Optim 55(4):771–799

    Article  MathSciNet  Google Scholar 

  31. Parikh N, Boyd S (2014) Proximal algorithms. Found Trends Optim 1(3):123–231

    Google Scholar 

  32. Paul R, Wohlberg B (2009) A generalized vector-valued total variation algorithm. In: Proceedings of the 16th IEEE international conference on image processing

  33. Perkins S, Theiler J (2003) Online feature selection using grafting. In: International conference on machine learning (ICML)

  34. Polyak BT (1987) Introduction to optimization. Optimization Software, New York

    MATH  Google Scholar 

  35. Srinivas S, Subramanya A, Babu RV (2017) Training sparse neural networks. In: IEEE International conference on computer vision and pattern recognition workshops

  36. Tian TP, Li R, Sclaroff S (2005) Articulated pose estimation in a learned smooth space of feasible solutions. In: Proceedings of IEEE workshop in CVPR

  37. Tibshirani R (1994) Regression shrinkage and selection via the lasso. J R Statist Soc Series B 58:267–288

    MathSciNet  MATH  Google Scholar 

  38. Wahlberg B, Boyd S, Annergren M, Wang Y (2012) An ADMM algorithm for a class of total variation regularized estimation problems. In: Proceedings of the 16th IFAC symposium on system identification

    Article  Google Scholar 

  39. Wright S (1997) Primal-dual interior-point methods. Society for industrial and applied mathematics. SIAM, Philadelphia

    Book  Google Scholar 

  40. Yang AY, Zhou Z, Ganesh A, Sastry SS, Ma Y (2013) Fast l 1-minimization algorithms for robust face recognition. IEEE Trans Image Process 22(8):3234–3246

    Article  Google Scholar 

  41. Ye F, Zhang L, Zhang D, Fujita H, Gong Z (2016) A novel forecasting method based on multi-order fuzzy time series and technical analysis. Inf Sci 367–368:41–57

    Article  Google Scholar 

  42. Ye Y (1997) Interior point algorithms: theory and analysis. Wiley, New York

    Book  Google Scholar 

  43. Yu YF, Dai DQ, Ren CX, Huang KK (2017) Discriminative multi-scale sparse coding for single-sample face recognition with occlusion. Pattern Recogn 66:302–312

    Article  Google Scholar 

  44. Yuan Z, Lu T, Tan CL (2017) Learning discriminated and correlated patches for multi-view object detection using sparse coding. Pattern Recogn 69:26–38

    Article  Google Scholar 

  45. Zhu J, Xing EP (2011) Sparse topical coding. In: Proceedings of the conference on uncertainty in artificial intelligence

Download references

Funding

This work is supported by National Research Foundation of Korea (NRF-2016R1A1A1A05921948).

Author information

Authors and Affiliations

  1. Department of Electronics & IT Media Engineering, Seoul National University of Science & Technology, Seoul, 139-743, Korea

    Minyoung Kim

Authors
  1. Minyoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minyoung Kim.

Ethics declarations

Conflict of interests

The authors have no conflict of interest.

Ethical Approval

This research does not involve human participants nor animals.

Consent for Publication

Consent to submit this manuscript has been received tacitly from the authors’ institution, Seoul National University of Science & Technology.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, M. Dynamic sparse coding for sparse time-series modeling via first-order smooth optimization. Appl Intell 48, 3889–3901 (2018). https://doi.org/10.1007/s10489-018-1189-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-018-1189-z

Keywords

Search

Navigation