Automatic classification of infant vocalization sequences with convolutional neural networks

Highlights

• Small bottlenecks after the convolutional stage primarily raise CNN performance.

• Global average pooling layers are most efficient for creating small bottlenecks.

• Tuning the convolutional receptive field size is the second most important factor.

• Vocalization classes are primarily confused according to their affective similarity.

Abstract

In this study we investigated Convolutional Neural Networks (CNNs) for the classification of infant vocalization sequences. The target classes were ‘crying’, ‘fussing’, ‘babbling’, ‘laughing’ and ‘vegetative vocalizations’. The general case of this classification task is of importance for applications which require a qualitative evaluation of general infant vocalizations, such as pain assessment or assessment of language acquisition. The classification procedure was based on representing audio segments as spectrograms which are input to an conventional CNN architecture scheme. We systematically analyzed the influence of network features on the classification performance to derive guidelines for designing effective CNN architectures for the task. We show that CNNs should be modeled to have a small bottleneck between the convolutional stage and the fully connected stage, achieved through broad aggregation of convolutional feature maps across the time and frequency axis. The best performing CNN configuration yielded a balanced accuracy of 72%. We conclude that conventional CNN architectures can reach satisfactory performance for this task even with small amounts of training data as long as certain network features are ensured.

Introduction

Automatic classification of infant vocalizations is a promising field of research to support areas which require a qualitative assessment of infant vocal expressions. Examples for such areas are pain assessment in paediatric wards or assessment of language acquisition. Automatic systems can aid in increasing coverage when human surveillance is infeasible.

Most publications in automatic infant vocalization recognition have been based on conventional audio recognition approaches. In these, audio signals are represented through hand-crafted and often task-specific feature sets such as mel frequency cepstral coefficients, fundamental frequency etc. They are used to train conventional classifiers such as support vector machines, hidden Markov models or multilayer perceptrons (Wagner et al., 2018) (Virtanen et al., 2018, chapter 4). These methods have been widely used for infant vocalization recognition, e.g. in the studies (Abdulaziz, Ahmad, 2010, Xie, Ward, Laszlo, 1996, Zhang, Cristia, Warlaumont, Schuller, 2018, Naithani, Kivinummi, Virtanen, Tammela, Peltola, Leppänen, 2018, Rodriguez, Caluya, 2017, Ntalampiras, 2015).

More recently, ‘deep learning’ or ‘end-to-end’ approaches emerged as an alternative to conventional approaches. They are based on replacing hand-crafted feature sets with more basic, task-unspecific audio representations such as spectrograms or raw waveforms. Those representations are fed into neural networks, particularly Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) Wagner et al. (2018) (Virtanen et al., 2018, chapter 4). Recently end-to-end systems significantly outperformed conventional systems for general audio recognition tasks such as audio scene classification or audio event detection, as the DCASE competitions demonstrated Mesaros, Heittola, Virtanen, 2018, Mesaros, Heittola, Diment, Elizalde, Shah, Vincent, Raj, Virtanen, 2017.

Consequently, researchers began to apply end-to-end systems to infant vocalization recognition tasks as well: Chang and Li (2016) employed CNNs with spectrogram inputs for cry reason classification and reached a validation accuracy of 78.5%. Lavner et al. (2016) also employed CNNs with spectrogram inputs for cry sound detection and compared it to a conventional approach based on logistic regression. The CNN outperformed the conventional approach for lower false-positive rates. The Interspeech 2018 computational paralinguistics challenge (Schuller et al., 2018) proposed a competition for classifying infant vocalizations into ‘crying’, ‘fussing’ and ‘neutral’. The organization team entered a baseline system employing a CNN-RNN with raw waveform inputs which reached a test balanced accuracy of 63 %. Two of seven participating teams entered end-to-end system as well: Turan and Erzin (2018) proposed a CNN variant called “capsule net” with spectrogram inputs and reached a balanced accuracy of 71.6%. Wagner et al. (2018) reached 67% through an RNN with spectrogram as well as raw waveform inputs.

These studies established and emphasized the general feasibility of infant vocalization recognition through end-to-end systems. While they provide performance indications for specific system configurations, they did not yet provide in-depth analysis of which system choices and parameterizations have the greatest impact on the performance. Such analysis is necessary to explore the true performance potential of end-to-end systems. For example, analysis and optimization of architectural CNN hyperparameters lead to significant performance improvements in the area of image classification with CNNs (Krizhevsky, Sutskever, Hinton, 2012, Zeiler, Fergus, 2014, Simonyan, Zisserman, 2014). To our knowledge, there is only one study in the area of infant vocalization recognition which provided an in-depth analysis of an end-to-end system component: Wagner et al. (2018) systematically compared various feature set types for a fixed RNN architecture and found that hand-crafted features outperformed more basic audio representations. They concluded that RNNs are more suited for hand-crafted features for infant vocalization classification.

In this study we investigated the classification of infant vocalizations through CNNs with mel-spectrogram inputs. We adapted this approach as it is currently the prevailing end-to-end paradigm for general audio classification tasks (Mesaros, Heittola, Virtanen, 2018, Mesaros, Heittola, Diment, Elizalde, Shah, Vincent, Raj, Virtanen, 2017; Hershey et al., 2017). Our investigation focused on optimizing the CNN architecture to increase the classification performance. Our goal was to not only identify the most performant CNN configurations, but to determine which CNN features have the greatest impact on the classification performance in general. Consequently, our key contribution is the identification of the most relevant CNN traits for infant vocalization classification. While there are additional system components which typically also contribute to raising performance, such as data augmentation techniques (Virtanen et al., 2018, chapter 5), our investigation focuses exclusively on the influence of the CNN architecture as it forms the essential core part of any end-to-end system.

The methodology of the investigation was as follows: We specified a vocalization classification task by defining a target class set and constructing a corresponding acoustic database. We then defined a CNN architecture scheme representative of conventional VGG-like CNNs (Simonyan and Zisserman, 2014) and produced numerous CNN configurations of this scheme drawn from a parameter space. Each configuration was trained and evaluated on the database. We finally analyzed the relation between architectural CNN features and the classification performance through statistical methods to identify the most influential ones.

The remainder of this paper is structured as follows: Section 2.1 describes the target classes and Section 2.2 the acoustic database. Section 2.3 summarizes the deep learning approach. Section 2.4 describes the CNN architecture scheme as well as the configurations. Section 2.5 describes the evaluation procedure for measuring the performance of the CNN configurations. The results Section 3 summarizes the overall system performance and shows the analysis of the relation between CNN features and the performance.