Skip to main content
SpringerLink
Log in

Deep Neural Networks: Models and Methods

  • Chapter
  • First Online:
Multi-faceted Deep Learning

Abstract

In this chapter we review main principles of Deep Neural Networks models with a specific focus on those used in the next chapters. We first introduce Artificial Neural Networks, then, we discuss Convolutional Neural Networks and Recurrent Neural Networks. The last sections are dedicated to some particular architectures which gained popularity in last few years: Generative Adversarial Networks, Auto-encoders and Siamese Networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20–26 June 2005, San Diego, CA, USA, pages 539–546. IEEE Computer Society, 2005.

    Google Scholar 

  2. Wei-Ta Chu. An introduction to optimization. 2014.

    Google Scholar 

  3. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.

    Google Scholar 

  4. George Cybenko. Approximation by superpositions of a sigmoidal function. MCSS, 2(4):303–314, 1989.

    MathSciNet  MATH  Google Scholar 

  5. Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings. JMLR.org, 2013.

    Google Scholar 

  6. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. 12:2121–2159, July 2011. http://jmlr.org/papers/volume12/duchi11a/duchi11a.pdf.

  7. Yann Dauphin, Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. CoRR, abs/1406.2572, 2014. http://arxiv.org/abs/1406.2572.

  8. Pierre Gillot, Jenny Benois-Pineau, Akka Zemmari, and Yurii E. Nesterov. Increasing training stability for deep CNNS. In 2018 IEEE International Conference on Image Processing, ICIP 2018, Athens, Greece, October 7–10, 2018, pages 3423–3427. IEEE, 2018.

    Google Scholar 

  9. Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR, abs/1311.2524, 2013. http://arxiv.org/abs/1311.2524.

  10. Ross B. Girshick. Fast R-CNN. CoRR, abs/1504.08083, 2015. http://arxiv.org/abs/1504.08083.

  11. Raphael Hauser. Line search methods for unconstrained optimisation. Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, 2007.

    Google Scholar 

  12. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17–22 June 2006, New York, NY, USA, pages 1735–1742. IEEE Computer Society, 2006.

    Google Scholar 

  13. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.

    Article  Google Scholar 

  14. Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning - lecture 6a - overview of mini-batch gradient descent. 2012.

    Google Scholar 

  15. D. H. HUBEL and T. N. WIESEL. Receptive fields and functional architecture of monkey striate cortex. 195:215–243, 1968. http://hubel.med.harvard.edu/papers/HubelWiesel1968Jphysiol.pdf.

  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. http://arxiv.org/abs/1512.03385.

  17. Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., and Bengio Y. Generative adversarial nets. Advances in Neural Information Processing Systems, pages 2672–2680, 2041.

    Google Scholar 

  18. Orlando De Jesus and Martin T. Hagan. Backpropagation through time for general dynamic networks. In Hamid R. Arabnia and Youngsong Mun, editors, Proceedings of the 2008 International Conference on Artificial Intelligence, ICAI 2008, July 14–17, 2008, Las Vegas, Nevada, USA, 2 Volumes (includes the 2008 International Conference on Machine Learning; Models, Technologies and Applications), pages 45–51. CSREA Press, 2008.

    Google Scholar 

  19. Nathalie Japkowicz, Catherine Myers, and Mark A. Gluck. A novelty detection approach to classification. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, IJCAI 95, Montréal Québec, Canada, August 20–25 1995, 2 Volumes, pages 518–523. Morgan Kaufmann, 1995.

    Google Scholar 

  20. Nan Jiang, Wenge Rong, Baolin Peng, Yifan Nie, and Zhang Xiong. An empirical analysis of different sparse penalties for autoencoder in unsupervised feature learning. In 2015 International Joint Conference on Neural Networks, IJCNN 2015, Killarney, Ireland, July 12–17, 2015, pages 1–8. IEEE, 2015.

    Google Scholar 

  21. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. https://arxiv.org/pdf/1412.6980.pdf.

  22. Rafal Kozik, Marek Pawlicki, and Michal Choras. Sparse autoencoders for unsupervised netflow data classification. In Michal Choras and Ryszard S. Choras, editors, Image Processing and Communications Challenges 10 - 10th International Conference, IP&C’2018, Bydgoszcz, Poland, 14–16 November 2018, Proceedings, volume 892 of Advances in Intelligent Systems and Computing, pages 192–199. Springer, 2018.

    Google Scholar 

  23. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.

    Google Scholar 

  24. Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. pages 2278–2324, 1998. http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf.

  25. Yann LeCun. MNIST Demos. Yann LeCun’s website. http://yann.lecun.com/exdb/lenet/index.html.

  26. Romain Lopez, Jeffrey Regier, Michael I. Jordan, and Nir Yosef. Information constraints on auto-encoding variational bayes. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pages 6117–6128, 2018.

    Google Scholar 

  27. Sheng Ma and Chuanyi Ji. A unified approach on fast training of feedforward and recurrent networks using EM algorithm. IEEE Trans. Signal Processing, 46(8):2270–2274, 1998.

    Article  Google Scholar 

  28. W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:115–133, 1943.

    Article  MathSciNet  Google Scholar 

  29. Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k 2). Soviet Mathematics Doklady (Vol. 27), 1983.

    Google Scholar 

  30. Boris Polyak and A B. Juditsky. Acceleration of stochastic approximation by averaging. 30:838–855, 07 1992.

    Google Scholar 

  31. Miltiadis Poursanidis, Jenny Benois-Pineau, Akka Zemmari, Boris Mansencal, and Aymar de Rugy. Move-to-data: A new continual learning approach with deep CNNS, application for image-class recognition, 2020.

    Google Scholar 

  32. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013 [DBL13], pages 1310–1318.

  33. Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs/1506.01497, 2015. http://arxiv.org/abs/1506.01497.

  34. Nitish Srivastava, Geoffrey Hinton, Alex hevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.

    Google Scholar 

  35. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. http://arxiv.org/abs/1409.4842.

  36. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. pages III–1139–III–1147, 2013. http://dl.acm.org/citation.cfm?id=3042817.3043064.

  37. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013 [DBL13], pages 1139–1147.

  38. Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. 28(3):343–351, 2013. https://arxiv.org/pdf/1206.1106.pdf.

  39. J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. Selective search for object recognition. International Journal of Computer Vision, 104(2):154–171, 2013.

    Article  Google Scholar 

  40. Gezheng Wen and Li Fan. Large scale optimization - lecture 4. 2012.

    Google Scholar 

  41. Akka Zemmari and Jenny Benois-Pineau. Deep Learning in Mining of Visual Content. Springer Briefs in Computer Science. Springer, 2020.

    Book  Google Scholar 

  42. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR, abs/1311.2901, 2013. http://arxiv.org/abs/1311.2901.

Download references

Author information

Authors and Affiliations

  1. LaBRI UMR 5800, University of Bordeaux, Talence Cedex, France

    Akka Zemmari & Jenny Benois-Pineau

Authors
  1. Akka Zemmari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jenny Benois-Pineau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Akka Zemmari .

Editor information

Editors and Affiliations

  1. LaBRI UMR 5800, University of Bordeaux, Talence Cedex, France

    Jenny Benois-Pineau

  2. LaBRI UMR 5800, University of Bordeaux, Talence Cedex, France

    Akka Zemmari

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Cite this chapter

Zemmari, A., Benois-Pineau, J. (2021). Deep Neural Networks: Models and Methods. In: Benois-Pineau, J., Zemmari, A. (eds) Multi-faceted Deep Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-74478-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74478-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74477-9

  • Online ISBN: 978-3-030-74478-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation