What do Programmers Discuss about Deep Learning Frameworks

Abstract

Deep learning has gained tremendous traction from the developer and researcher communities. It plays an increasingly significant role in a number of application domains. Deep learning frameworks are proposed to help developers and researchers easily leverage deep learning technologies, and they attract a great number of discussions on popular platforms, i.e., Stack Overflow and GitHub. To understand and compare the insights from these two platforms, we mine the topics of interests from these two platforms. Specifically, we apply Latent Dirichlet Allocation (LDA) topic modeling techniques to derive the discussion topics related to three popular deep learning frameworks, namely, Tensorflow, PyTorch and Theano. Within each platform, we compare the topics across the three deep learning frameworks. Moreover, we make a comparison of topics between the two platforms. Our observations include 1) a wide range of topics that are discussed about the three deep learning frameworks on both platforms, and the most popular workflow stages are Model Training and Preliminary Preparation. 2) the topic distributions at the workflow level and topic category level on Tensorflow and PyTorch are always similar while the topic distribution pattern on Theano is quite different. In addition, the topic trends at the workflow level and topic category level of the three deep learning frameworks are quite different. 3) the topics at the workflow level show different trends across the two platforms. e.g., the trend of the Preliminary Preparation stage topic on Stack Overflow comes to be relatively stable after 2016, while the trend of it on GitHub shows a stronger upward trend after 2016. Besides, the Model Training stage topic still achieves the highest impact scores across two platforms. Based on the findings, we also discuss implications for practitioners and researchers.

Appendices

Appendix

A LDA Topics and Keywords

Tables 7, 8, 9, 10, 11 and 12 illustrate the details of the discovered LDA-topics and their top keywords.

Table 7 Topics Generated by LDA for tensorflow on Stack Overflow

Table 8 Topics Generated by LDA for PyTorch on Stack Overflow

Table 9 Topics Generated by LDA for Theano on Stack Overflow

Table 10 Topics Generated by LDA for Tensorflow on GitHub

Table 11 Topics Generated by LDA for PyTorch on GitHub

Table 12 Topics Generated by LDA for Theano on GitHub

B LDA-topic Trends

We further make analyses on the development trends of LDA-topics, we first detemine the dominant topics of the posts/records by applying the dominant topic metric (1). Then, we calculate the topic trends of the LDA-topics using the impact metric (2). Figures 17, 18, 19, 20, 21 and 22 illustrate the top 3 LDA-topics with the largest increases or decreases over time for the six different corpora.

Fig. 17

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of Tensorflow on Stack Overflow, as measured by the percentage change in topic impact scores during November 2015 to March 2018

Fig. 18

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of Tensorflow on GitHub, as measured by the percentage change in topic impact scores during November 2015 to July 2018

Fig. 19

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of PyTorch on Stack Overflow, as measured by the percentage change in topic impact scores during January 2017 to March 2018

Fig. 20

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of PyTorch on GitHub, as measured by the percentage change in topic impact scores during September 2016 to July 2018

Fig. 21

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of Theano on Stack Overflow, as measured by the percentage change in topic impact scores during April 2012 to March 2018

Fig. 22

The top 3 LDA-topics with largest increasing and decreasing trends for the corpus of Theano on GitHub, as measured by the percentage change in topic impact scores during August 2011 to July 2018

C Examples of LDA Topics on Different Workflow Stages

Preliminary Preparation. :

Out of the derived 75 LDA-topics, 29 LDA-topics belong to the preliminary preparation stage, which are: 1). File Operation for Tensorflow on Stack Overflow, 2). Keras for Tensorflow on Stack Overflow, 3). Installation Error for Tensorflow on Stack Overflow, 4). Runtime Error for Tensorflow on Stack Overflow, 5). Build Error for Tensorflow on Stack Overflow, 6). Code Error for PyTorch on Stack Overflow, 7). Installation Error for PyTorch on Stack Overflow, 8). File Operation for Theano on Stack Overflow, 9). Code Error for Theano on Stack Overflow, 10). File Operation for Tensorflow on GitHub, 11). Installation in Linux for Tensorflow on GitHub, 12). Version Problem for Tensorflow on GitHub, 13). Build Error for Tensorflow on GitHub, 14). Fixing Error for Tensorflow on GitHub, 15). File Operation for PyTorch on GitHub, 16). System Installation for PyTorch on GitHub, 17). Version Problem for PyTorch on GitHub, 18). Third Party for PyTorch on GitHub, 19). Build Error for PyTorch on GitHub, 20). Code Error for PyTorch on GitHub, 21). File Operation for Theano on GitHub, 22). Version Problem for Theano on GitHub, 23). Using Numpy for Theano on GitHub, 24). Import Error for Theano on GitHub, 25). Installation Error for Theano on GitHub, 26). Warnings for Theano on GitHub, 27). Errors on Windows for Theano on GitHub, 28). Compile Error for Theano on GitHub, 29). Clang Error for Theano on GitHub. Some example posts/records of these LDA-topics are shown as following:

• Detect object from video stream using Keras .h5 file I am using keras and tensorflow to train a custom model using transfer learning. I was wondering, is there any tutorial which covers custom object detection from live video stream using keras .h5 file? Here is my sample code for training based on https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb Dominant Topic: Tensorflow on Stack Overflow/Keras

• Install PyTorch on Windows I am trying to install PyTorch on Windows8.1. I am using Python 3.6.4 and no GPU. I’ve tried already the Anaconda package provided by peterjc123 by running conda install -c peterjc123 pytorch_legacy cuda80 using a virtual environment. While the installation goes smooth (without errors), after import torch I get the following error. Can somebody help me to install it? Dominant Topic: PyTorch on Stack Overflow/Installation Error

• strides argument, the layer received both the legacy keyword argument subsample and the Keras 2 keyword argument strides when I try to run this code with keras 2.1.3 I get this error https://github.com/marcellacornia/sam Dominant Topic: Theano on Stack Overflow/Code Error

• cannot import name bayesflow Error Hi, I get the error I mentioned in the title. I did a search on Google and I usually found a solution to update dask. I updated Dask to version 0.17.2 but I still get the same error. I can not import BayesFlow. The Tensorflow version is 0.12.1. Thanks for the answers.

  Code : from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets(“/tmp/data/”)

  OS : Ubuntu 16.04 LTS Tensorflow version is 0.12.1 Cuda : 8.0 CuDNN : 5.1 GPU : 4 GB GTX 1050Ti Dask : 0.17.2 Dominant Topic: Tensorflow on GitHub/Version Problem

• Install hpp headers for CPP Extensions With the Cppzation of a few files in ‘TH’/‘THC’, the CPP extensions got broken whenever the user uses feature from ‘THC’ in their files, when pytorch is installed via ‘python setup.py install’.

  This addresses issues such as “/home/me/.conda/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/THC/THCDeviceTensorUtils.cuh:5:25: fatal error: THCTensor.hpp: No such file or directory” Dominant Topic: PyTorch on GitHub/File Operation

• Theano cannot detect clang++ in Mac OS X I am using the dev version of theano under Mac OS X 10.11.3 with command line tools for Xcode 7. Running theano gives me the following warning:

  ‘WARNING (theano.configdefaults): Only clang++ is supported. With g++, we end up with strange g++/OSX bugs.’

  I’ve also got g++ installed. It seems theano cannot detect the ‘clang++’.

  Besides, I had strang NaN problems resulting from a simple calculation ‘T.dot(W, X)’ where ‘W’ and ‘X’ do not have nan value (checked with ‘np.any(np.isnan)’). I doubt this is because of I am not using ‘clang++’ Dominant Topic: Theano on GitHub/Clang Error

Data Preparation. :

We obtain 13 LDA-topics in the data preparation stage, that is: 1). Variable for Tensorflow on Stack Overflow, 2). Data Reading for Tensorflow on Stack Overflow, 3). Tensor Operation for Tensorflow on Stack Overflow, 4). Input Error for Tensorflow on Stack Overflow, 5). Input Size for PyTorch on Stack Overflow, 6). Tensor Error for PyTorch on Stack Overflow, 7). Value Type for Tensorflow on GitHub, 8). Variable Shape for Tensorflow on GitHub, 9). Tensor Operation for PyTorch on GitHub, 10). Tensor Fixing for PyTorch on GitHub, 11). Data Type for Theano on GitHub, 12). Data Shape for Theano on GitHub, 13). Object Error for Theano on GitHub. The following is the examples of posts/records of the LDA-topics.

• Dataset API ‘flat_map’ method producing error for same code which works with ‘map’ method I am trying to create a pipeline to read multiple CSV files using TensorFlow Dataset API and Pandas. However, using the flat_map method is producing errors. However, if I am using map method I am able to build the code and run it in session. This is the code I am using. I already opened #17415 issue in TensorFlow Github repository. Dominant Topic: Tensorflow on Stack Overflow/Data Reading

• Image Captioning Example input size of Decoder LSTM PyTorch I’m new to PyTorch, there is a doubt that am having in the Image Captioning example code. We first embed the captions and then concat the embeddings with the context feature from the EncoderCNN, but the concat increases the size from embed size how we can forward that to the lstm? As the input size of lstm is already defined as embed_size. Dominant Topic: PyTorch on Stack Overflow/Input Size

• Feature request: tf.as_dtype(float) should work just as tf.as_dtype(‘float’) In NumPy, ‘np.dtype(float)’ works just the same as ‘np.dtype(“float”)’. In TensorFlow ‘tf.as_dtype(“float”)’ works but ‘tf.as_dtype(float)’ crashes with ‘TypeError: Cannot convert value <class ‘float’> to a TensorFlow DType.’. Is there a particular reason for this behaviour or was it just overlooked? (same error for other builtins such as ‘int’ and ‘complex’) Dominant Topic: Tensorflow on GitHub/Value Type

• Explicitly specify the output ndim in reshape The whole code expects the shape to be of length 4, and the output to be 4D already. Fixes #5613. Dominant Topic: Theano on GitHub/Data Shape

Model Setup. :

We find 8 model setup LDA-topics in our dataset, namely: 1). Image Classification for Tensorflow on Stack Overflow, 2). Word Embedding for Tensorflow on Stack Overflow, 3). RNN LSTM for Tensorflow on Stack Overflow, 4). Network Layer for Tensorflow on Stack Overflow, 5). Model Saving for Tensorflow on Stack Overflow, 6). Image Training for PyTorch on Stack Overflow, 7). Network Layer for PyTorch on Stack Overflow, 8). Storage Error for Theano on GitHub. Here are the 3 examples of these LDA-topics.

• single neuron layer after softmax (keras) I need to create a neural network (with keras) that has as last layer a single neuron that contains the index of the neuron with the maximum value prediction in the precedent softmax layer. For example my softmax layer gives as result this: [0.1, 0.1, 0.7, 0.0, 0.05, 0.05] And I want that the single neuron layer (after the softmax layer) gives as result 2 (considering a 0 based valutation). How can I do that? Dominant Topic: Tensorflow on Stack Overflow/Network Layer

• In tensorflow deep and wide tutorial, what’s the embedding principle When I played tensorflow tutorial, one embedding trick is used in Wide and Deep tutorial like this. The tutorial shows how transfer sparse feature (usually one hot encoding) to embedding vector. I knew there are some approaches to create this embedding, such as word embedding, PCA or t-SNE or matrix factorization. But in this tutorial, they did not show how to create an embedding for the sparse vector. Or did the tutorial just use neural network to finish the embedding? Dominant Topic: Tensorflow on Stack Overflow/Word Embedding

• Reading multiple images as custom dataset for PyTorch? I want to read in multiple images for the main_image set and blur_image set. For example, 5 main images and 5 blurred images. The goal is determine what values for the kernel in the convolutional layer convert the main images to the blurred images. The assumption is that the same kernel is used to blur each of the 5 original images to produce the 5 blurred images.

  My code is available at: https://pastebin.com/PWf7rjd4 and https://pastebin.com/VxryDb7g

  However, it seems to only be processing the first image, that is “1.png” for the main and blurred images. It is not processing images 2.png, 3.png, 4.png, and 5.png How can I fix this? Dominant Topic: PyTorch on Stack Overflow/Image Training

Model Training. :

We discover 22 model training LDA-topics in our dataset, including: 1). Model Training for Tensorflow on Stack Overflow, 2). Loss Function for Tensorflow on Stack Overflow, 3). Batch for Tensorflow on Stack Overflow, 4). Performance for Tensorflow on Stack Overflow, 5). Training Accuracy for Tensorflow on Stack Overflow, 6). Model Training for PyTorch on Stack Overflow, 7). Gpu Training for PyTorch on Stack Overflow, 8). Loss Function for PyTorch on Stack Overflow, 9). Cuda Error for PyTorch on Stack Overflow, 10). Function Operation for Theano on Stack Overflow, 11). Model Training for Theano on Stack Overflow, 12). Gpu Error for Theano on Stack Overflow, 13). Model Training for Tensorflow on GitHub, 14). Gradient for Tensorflow on GitHub, 15). Performance for Tensorflow on GitHub, 16). Function Operation for PyTorch on GitHub, 17). Distributed Process for PyTorch on GitHub, 18). Theano Composite for Theano on GitHub, 19). Performance for Theano on GitHub, 20). Gpu Error for Theano on GitHub, 21). Cuda Error for Theano on GitHub, 22). Gpuarray Bug for Theano on GitHub. The following are the examples of posts/records in our dataset.

• Try to define pearson correlation as loss function but got error I would like to use pearson correlation as the loss function in Keras with backend of tensorflow. The dimensions of the tensor is (Batch, Coils, Time). The correlation coefficients are to be calculated along time and across coils. For example, if the number of coils is 3, the averaged correlation coefficients will be calculated between coil #1 and #2, #1 and #3, and #2 and #3. Dominant Topic: Tensorflow on Stack Overflow/Loss Function

• RNN is not training (PyTorch) I can’t get what I am doing wrong when training RNN. I am trying to train RNN for AND operation on sequences (to learn how it works on simple task). But my network is not learning, loss stays the same and it can’t event overfit the model. Can you please help me to find the problem? Dominant Topic: PyTorch on Stack Overflow/Model Training

• Enabling GPU with theano generates Exception I have followed the steps from here to enable gpu with theano on an Ubuntu 16.04 machine. I installed cuda toolkit, cudnn, drivers but I am still not able to get it to work. Dominant Topic: Theano on Stack Overflow/Gpu Error

• [Java] Support addition of gradient operations in a graph This calls the C-api ‘TF_AddGradients’ method through a new JNI binding for adding gradient nodes to a graph. It also includes an ‘AddGradients’ wrapper for invoking this operation smoothly while building a graph using the new Java Ops API. Dominant Topic: Tensorflow on GitHub/Gradient

• Use customized python interpreter for distributed launch util For spawning sub-processes, I think it should be quite intuitive to use the interpreter of ‘launch.py’ rather than the default ‘python’. Dominant Topic: PyTorch on GitHub/Distributed Process

Model Evaluation. :

We find 2 model evaluation LDA-topics in our dataset, 1). Tensorboard for Tensorflow on Stack Overflow , 2). Code Graph for Theano on GitHub. The following example post is from these LDA-topics:

• Tensorboard not creating network graph (Python) I really can’t understand why tensorboard is not showing the graph of my network. I have followed the tutorials on Tensorboard Website and other stuff in the web, none of these allowed to display the graph. I am embedding the part of my code related to the network. I’ve tried to remove all the other parts but I did not want to reduce to much otherwise it can create confusion. The only thing it displays on the graph sections is the global_step. Dominant Topic: Tensorflow on Stack Overflow/Tensorboard

Model Tuning. :

In our derived 75 LDA-topics, no LDA-topic is related to the model tuning stage.

Model Prediction. :

Only 1 LDA-topic is associated with the model prediction stage, which is 1). Test Error for Theano on GitHub.