Dilated residual networks with multi-level attention for speaker verification

Abstract

With the development of deep learning techniques, speaker verification (SV) systems based on deep neural network (DNN) achieve competitive performance compared with traditional i-vector-based works. Previous DNN-based SV methods usually employ time-delay neural network, limiting the extension of the network for an effective representation. Besides, existing attention mechanisms used in DNN-based SV systems are only applied to a single level of network architectures, leading to insufficiently extraction of important features.

To address above issues, we propose an effective deep speaker embedding architecture for SV, which combines a residual connection of one-dimensional dilated convolutional layers, called dilated residual networks (DRNs), with a multi-level attention model. The DRNs can not only capture long time-frequency context information of features, but also exploit information from multiple layers of DNN. In addition, the multi-level attention model, which consists of two-dimensional convolutional block attention modules employed at the frame level and the vector-based attention utilized at the pooling layer, can emphasize important features at multiple levels of DNN. Experiments conducted on NIST SRE 2016 dataset show that the proposed architecture achieves a superior equal error rate (EER) of 7.094% and a better detection cost function (DCF16) of 0.552 compared with state-of-the-art methods. Furthermore, the ablation experiments demonstrate the effectiveness of dilated convolutions and the multi-level attention on SV tasks.

Introduction

Speaker verification (SV) serves as a common speech-related research topic to determine if a given speech utterance belongs to a specific speaker, which can be applied in various fields, such as criminal investigation, judicial forensics and telephone identification. According to the difference of recording texts in speech utterances, SV can be generally classified into two typical categories: text-dependent SV and text-independent SV. Since text-independent SV is more challenging and has greater practical significance [1], [2], this work mainly focuses on text-independent SV which has no phonetic restrictions on the recording texts.

Over the past decade, i-vector-based representation [3] using a probabilistic linear discriminant analysis (PLDA) backend classifier [4] dominates the SV area, which consists of a gaussian mixture model-universal background model (GMM-UBM) to collect statistics and a large projection matrix to map the high-dimensional statistics into a low-dimensional representation.

Recently, with the development of deep learning techniques, a lot of works introduced deep neural network (DNN) into SV, and achieved competitive performance compared with traditional i-vector-based works. In detail, the DNN-based SV methods exploit a deep speaker learning architecture instead of the i-vector model to extract speaker representations. The process can be divided into three phases [5] in Fig. 1, which is also adopted in this work:

• Training phase: First, the utterances in training dataset are cut into segments with the length of 2 to 4 s as training samples of the DNN. The samples are extracted into acoustic features, e.g., mel frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LFCC) and enhanced teager energy cepstral coefficients (ETECC) [6]. Then, the DNN is trained with acoustic features to discriminate between speakers with a loss function.

• Enrollment phase: The acoustic features of integral utterances in training dataset are fed into the trained DNN to extract speaker-dependent bottleneck features, known as speaker embeddings. The speaker embeddings in enrolled speaker dataset are used to train a separate PLDA classifier model for the following test phase.

• Test phase: The test utterances are evaluated with the DNN embeddings and the PLDA classifier.

In DNN-based SV systems, it is essential to construct an efficient deep embedding learning architecture for discriminating between speakers. The architecture can be bulit by time-delay neural network (TDNN) [7], [8], convolutional neural network (CNN) [9], [10] or long short-term memory network (LSTM) [11]. Among those methods, x-vector [8] using a TDNN architecture, which has superior performance than traditional i-vector, becomes the state-of-the-art DNN-based method in current SV works.

The x-vector architecture contains three typical parts: five TDNN layers for frame-level processing, a statistics pooling layer which aggregates the frame-level outputs across all frames, and two fully-connected (FC) layers for utterance-level processing. Although x-vector shows great performance on SV, the TDNN layers in x-vector limit the performance improvement. The works in [12], [13] show that exploiting information from multiple layers is beneficial for an effective representation since there exists a gradual transition from low to high layers in a DNN. However, it drastically increases computational complexity to concatenate the outputs of five high-dimensional TDNN layers in x-vector (512, 512, 512, 512, and 1500, respectively). In addition, the frequency dimension of input acoustic features is set only between 20 and 30, leading to a huge difference in dimensions between the input and the first TDNN layer, and resulting in leakage of corresponding feature information.

In recent years, to make the networks focus on important features, attention mechanisms have been gradually introduced to the DNN-based SV systems and applied to different levels of DNN architecture, such as the pooling layer [14], [15], [16], [17], the frame level [1] and the FC layer [18]. However, those works only employ the attention mechanism at a single level of the DNN and emphasize important features along a single dimension, ignoring the impacts from multiple levels and other feature dimensions.

Based on above observations, we propose an effective deep embedding learning architecture called Dilated Residual Networks with Multi-level Attention (DRES-MA for short), which combines dilated residual networks with a multi-level attention model, for the SV tasks. Inspired by traditional x-vector, we utilize one-dimensional dilated convolutional layers to capture long time-frequency context information for frame-level features. In order to exploit information from multiple layers, the residual connection [19] is employed to build the dilated residual networks. Besides, a multi-level attention model is proposed to focus on important features at multiple network levels, including the frame level and the pooling layer. At the frame level, we propose a two-dimensional convolutional block attention module (CBAM2D) to capture important features along temporal and channel axes. At the pooling layer, we introduce a vector-based attention (VA) mechanism and incorporate it into a weighted statistics pooling layer, called the VA-based pooling layer. The CBAM2D and VA-based pooling layer form the multi-level attention model, applying attention mechanisms to multiple network levels. Experiments conducted on NIST SRE 2016 datasets [20] verify the effectiveness of the proposed architecture. The DRES-MA architecture achieves an equal error rate (EER) of 7.094% and a detection cost function 2016 (DCF16) score of 0.552. Compared to the x-vector baseline, DRES-MA improves the EER by 17% and DCF16 score by 9% with little increase of computational cost.

The rest parts are organized as follows. Section 2 introduces related works of DNN-based SV methods. Section 3 describes the network construction of our proposed DRES-MA architecture. Section 4 presents the experimental setup. Experimental results and analysis of the proposed techniques are presented in Section 5. Section 6 concludes this work.