research-article

DeepTracker: Visualizing the Training Process of Convolutional Neural Networks

Authors:

Huamin QuAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 10, Issue 1

Article No.: 6, Pages 1 - 25

https://doi.org/10.1145/3200489

Published: 28 November 2018 Publication History

Abstract

Deep Convolutional Neural Networks (CNNs) have achieved remarkable success in various fields. However, training an excellent CNN is practically a trial-and-error process that consumes a tremendous amount of time and computer resources. To accelerate the training process and reduce the number of trials, experts need to understand what has occurred in the training process and why the resulting CNN behaves as it does. However, current popular training platforms, such as TensorFlow, only provide very little and general information, such as training/validation errors, which is far from enough to serve this purpose. To bridge this gap and help domain experts with their training tasks in a practical environment, we propose a visual analytics system, DeepTracker, to facilitate the exploration of the rich dynamics of CNN training processes and to identify the unusual patterns that are hidden behind the huge amount of information in training log. Specifically, we combine a hierarchical index mechanism and a set of hierarchical small multiples to help experts explore the entire training log from different levels of detail. We also introduce a novel cube-style visualization to reveal the complex correlations among multiple types of heterogeneous training data, including neuron weights, validation images, and training iterations. Three case studies are conducted to demonstrate how DeepTracker provides its users with valuable knowledge in an industry-level CNN training process; namely, in our case, training ResNet-50 on the ImageNet dataset. We show that our method can be easily applied to other state-of-the-art “very deep” CNN models.

References

[1]

Charu C. Aggarwal. 2013. Outlier Analysis. Springer.

Digital Library

[2]

Wolfgang Aigner, Silvia Miksch, Wolfgang Müller, Heidrun Schumann, and Christian Tominski. 2008. Visual methods for analyzing time-oriented data. IEEE TVCG 14, 1 (2008), 47–60.

Digital Library

[3]

Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, and Christian Tominski. 2011. Visualization of Time-Oriented Data. Springer.

Digital Library

[4]

Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. 2018. Do convolutional neural networks learn class hierarchy? IEEE TVCG 24, 1 (2018), 152–162.

[5]

Benjamin Bach, Pierre Dragicevic, Daniel Archambault, Christophe Hurter, and Sheelagh Carpendale. 2014. A review of temporal data visualizations based on space-time cube operations. In Proceedings of the Eurographics Conference on Visualization. The Eurographics Association, 23–41.

[6]

Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, Grégoire Montavon, Geneviève B. Orr, and Klaus-Robert Müller (Eds.). Springer, 437–478.

[7]

Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE TPAMI 35, 8 (2013), 1798–1828.

Digital Library

[8]

Lior Berry and Tamara Munzner. 2004. Binx: Dynamic exploration of time series datasets across aggregation levels. In Proceedings of the IEEE Symposium on Information Visualization. IEEE Computer Society, Los Alamitos, CA, USA, 5–6.

Digital Library

[9]

Léon Bottou. 1991. Stochastic gradient learning in neural networks. Neuro-Nımes 91, 8 (1991).

[10]

Sunghyo Chung, Cheonbok Park, Sangho Suh, Kyeongpil Kang, Jaegul Choo, and Bum Chul Kwon. 2016. Revacnn: Steering convolutional neural network via real-time visual analytics. In Future of Interactive Learning Machines Workshop at Proceedings of Conference on Neural Information Processing Systems (NIPS'16).

[11]

Alexey Dosovitskiy and Thomas Brox. 2015. Inverting convolutional networks with convolutional networks. arXiv preprint arXiv:1506.02753 (2015).

[12]

Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing Higher-Layer Features of a Deep Network. Technical Report 1341, University of Montreal, QC, Canada.

[13]

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 580–587.

Digital Library

[14]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats, Vol. 9. 249–256.

[15]

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017).

[16]

Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).

[17]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 770–778.

[18]

Jeffrey Heer, Michael Bostock, and Vadim Ogievetsky. 2010. A tour through the visualization zoo. Communications of the ACM 53, 6 (2010), 59–67.

Digital Library

[19]

Jeffrey Heer, Nicholas Kong, and Maneesh Agrawala. 2009. Sizing the horizon: The effects of chart size and layering on the graphical perception of time series visualizations. In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems. 1303–1312.

Digital Library

[20]

Waqas Javed, Bryan McDonnel, and Niklas Elmqvist. 2010. Graphical perception of multiple time series. IEEE TVCG 16, 6 (2010), 927–934.

Digital Library

[21]

Minsuk Kahng, Pierre Andrews, Aditya Kalro, and Duen Horng Chau. 2018. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE TVCG 24, 1 (2018), 88–97.

[22]

Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis. Department of Computer Science, University of Toronto.

[23]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of Conference on Neural Information Processing Systems (NIPS'12). 1097–1105.

Digital Library

[24]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444.

[25]

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002 (2017).

[26]

Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, and Shixia Liu. 2018. Analyzing the training processes of deep generative models. IEEE TVCG 24, 1 (2018), 77–87.

[27]

Mengchen Liu, Jiaxin Shi, Zhen Li, Chongxuan Li, Jun Zhu, and Shixia Liu. 2017. Towards better analysis of deep convolutional neural networks. IEEE TVCG 23, 1 (2017), 91–100.

Digital Library

[28]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15). 3431–3440.

[29]

Eliana Lorch. 2016. Visualizing deep network training trajectories with PCA. In Proceedings of the ICML Workshop on Visualization for Deep Learning.

[30]

Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15). 5188–5196.

[31]

Nicola Pezzotti, Thomas Höllt, Jan van Gemert, Boudewijn PF Lelieveldt, Elmar Eisemann, and Anna Vilanova. 2018. DeepEyes: Progressive visual analytics for designing deep neural networks. IEEE TVCG 24, 1 (2018), 98–108.

[32]

Paulo E. Rauber, Samuel G. Fadel, Alexandre X. Falcao, and Alexandru C. Telea. 2017. Visualizing the hidden activity of artificial neural networks. IEEE TVCG 23, 1 (2017), 101–110.

Digital Library

[33]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252.

Digital Library

[34]

Christin Seifert, Aisha Aamir, Aparna Balagopalan, Dhruv Jain, Abhinav Sharma, Sebastian Grottel, and Stefan Gumhold. 2017. Visualizations of deep neural networks in computer vision: A survey. In Transparent Data Mining for Big and Small Data, Tania Cerquitelli, Daniele Quercia, and Frank Pasquale (Eds.). Springer, 123–144.

[35]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[36]

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014).

[37]

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 (2016).

[38]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 1–9.

[39]

Edward R Tufte. 1983. The Visual Display of Quantitative Information. Graphics Press.

Digital Library

[40]

Andreas Veit, Michael J. Wilber, and Serge Belongie. 2016. Residual networks behave like ensembles of relatively shallow networks. In Proceedings of Conference on Neural Information Processing Systems (NIPS’16). 550–558.

Digital Library

[41]

Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV’14). 818–833.

[42]

Matthew D. Zeiler, Graham W. Taylor, and Rob Fergus. 2011. Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of IEEE International Conference on Computer Vision (ICCV’11). 2018–2025.

Digital Library

[43]

Haipeng Zeng, Hammad Haleem, Xavier Plantaz, Nan Cao, and Huamin Qu. 2017. CNNComparator: Comparative analytics of convolutional neural networks. arXiv preprint arXiv:1710.05285 (2017).

Cited By

Ma YZhang ZCheng RJin YTan K(2025)ParetoLens: A Visual Analytics Framework for Exploring Solution Sets of Multi-Objective Evolutionary Algorithms [Application Notes]IEEE Computational Intelligence Magazine10.1109/MCI.2024.348713420:1(78-94)Online publication date: Feb-2025
https://doi.org/10.1109/MCI.2024.3487134
Zhang TFeng HHuang WLiang LZhang HChen ZTung AChen W(2025)FedCare: towards interactive diagnosis of federated learning systemsFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-3735-719:7Online publication date: 1-Jul-2025
https://dl.acm.org/doi/10.1007/s11704-024-3735-7
Kim BChoi HJang HKim S(2024)Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual NetworksACM Transactions on Intelligent Systems and Technology10.1145/364386015:3(1-20)Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1145/3643860
Show More Cited By

Index Terms

DeepTracker: Visualizing the Training Process of Convolutional Neural Networks
1. Human-centered computing
  1. Visualization
    1. Visualization application domains
      1. Visual analytics

Recommendations

Neural network training fingerprint: visual analytics of the training process in classification neural networks
Abstract
The striking results of deep neural networks (DNN) have motivated its wide acceptance to tackle large datasets and complex tasks such as natural language processing, facial recognition, and artificial image generation. However, DNN parameters are ...
Process Improvement for Software Engineering Training
CSEE '96: Proceedings of the 9th Conference on Software Engineering Education

The training program of a software organization is the combination of activities described in the project training plans and the organizational training plan. The training process is the sequence of activities that are performed whenever a training ...
Deployment of an in-house designed training process in a quaternary care hospital

BACKGROUND: Healthcare providers, such as doctors and nurses, have been famous for high resistance to change. A careful change management plan, particularly training process, is utmost necessary. A quaternary care hospital in India changed its system, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 10, Issue 1

Special Issue on Visual Analytics

January 2019

235 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3295616

Editor:
Yu Zheng
JD Finance, China

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2018

Accepted: 01 February 2018

Revised: 01 December 2017

Received: 01 August 2017

Published in TIST Volume 10, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

ITC Grant
the National Basic Research Program of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
877
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ma YZhang ZCheng RJin YTan K(2025)ParetoLens: A Visual Analytics Framework for Exploring Solution Sets of Multi-Objective Evolutionary Algorithms [Application Notes]IEEE Computational Intelligence Magazine10.1109/MCI.2024.348713420:1(78-94)Online publication date: Feb-2025
https://doi.org/10.1109/MCI.2024.3487134
Zhang TFeng HHuang WLiang LZhang HChen ZTung AChen W(2025)FedCare: towards interactive diagnosis of federated learning systemsFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-3735-719:7Online publication date: 1-Jul-2025
https://dl.acm.org/doi/10.1007/s11704-024-3735-7
Kim BChoi HJang HKim S(2024)Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual NetworksACM Transactions on Intelligent Systems and Technology10.1145/364386015:3(1-20)Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1145/3643860
Zhang ZYang FCheng RMa Y(2024)ParetoTracker: Understanding Population Dynamics in Multi-Objective Evolutionary Algorithms Through Visual AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614231:1(820-830)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456142
Ruan SLiang ZGuan QGriffin PWen XLin YWang Y(2024)VIOLET: Visual Analytics for Explainable Quantum Neural NetworksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338855730:6(2862-2874)Online publication date: 23-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388557
Li GWang JWang YShan GZhao Y(2024)An In-Situ Visual Analytics Framework for Deep Neural NetworksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333958530:10(6770-6786)Online publication date: Oct-2024
https://doi.org/10.1109/TVCG.2023.3339585
Gao LShao ZLuo ZHu HTurkay CChen S(2024)TransforLearn: Interactive Visual Tutorial for the Transformer ModelIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332735330:1(891-901)Online publication date: Jan-2024
https://doi.org/10.1109/TVCG.2023.3327353
Wei YWang ZWang ZDai YOu GGao HYang HWang YCao CWeng LLu JZhu RChen W(2024)Visual Diagnostics of Parallel Performance in Training Large-Scale DNN ModelsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324322830:7(3915-3929)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3243228
Keelawat P(2023)NBGuru: Generating Explorable Data Science Flowcharts to Facilitate Asynchronous Communication in Interdisciplinary Data Science TeamsCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607020(6-11)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3607020
La Rosa BBlasilli GBourqui RAuber DSantucci GCapobianco RBertini EGiot RAngelini M(2023)State of the Art of Visual Analytics for eXplainable Deep LearningComputer Graphics Forum10.1111/cgf.1473342:1(319-355)Online publication date: 6-Feb-2023
https://doi.org/10.1111/cgf.14733
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents