Keywords

1 Introduction

Recently, with the spread of network devices, such as personal computers and smartphones, various pieces of information can be easily obtained, and the opportunity to obtain information by talking with people has decreased. The number of people who are living alone and are reclusive is increasing, and the lack of communication between humans has become a big problem.

The creation of some communities has been promoted to support the communication between the elderly, especially disaster-stricken elderly people. However, it is not easy to provide communication support to people who once missed the opportunity for interaction with others. When they are not confident in communicating with others or they have a negative impression of the community themselves, it is difficult to get people to immediately participate in communication support activities. Therefore, a different approach to solving this problem is necessary.

Many robots that communicate with humans have been developed for elderly persons [1,2,3]. They are effective at being interested in conversation; however, they have a problem in that the user becomes tired of their conversation for reasons such as lack of conversational flexibility and adaptability. It would be difficult to change their topics, expressions, utterance timings, or talking speeds according to the situation of the user and the conversation atmosphere. We have already proposed a conversation support system for the public community [4]. The system can understand whether or not the human-to-human communication proceeds smoothly. When sensing there has been little progress during the conversation, the system attempts to provide a topic to lead a smoother discussion and good atmosphere. We have already confirmed that the fundamental frequency (F0) and sound power (SP) values for each utterance are effective in estimating the conversation atmosphere using a free conversation database recorded in a recording studio. Figure 1 shows the structure of the conversation smoothness estimation process. The system extracts F0 values from input utterances and calculated the standard deviation of the F0 values (SD-F0). When the SD-F0 values of some utterances in the conversation are detected to be under the threshold value, the system decides that the conversation is not progressing smoothly and provides a new topic to liven up the conversation. However, the system was evaluated using conversations recorded in a studio, and we have not solved the problems of conversation support in real-world use.

Fig. 1
figure 1

Conventional process of a conversion support system

Full size image

We think that if we use our system in a real restaurant or lounge, the following fatal problems will occur and decrease the estimation performance:

  1. 1.

    Ambient noise decreases the estimation performance. In particular, it would be difficult to exclude nonstationary noise, such as a person’s speech from behind the target speaker, even when we use current noise cancelling methods.

  2. 2.

    Non-language utterances (e.g., laughter, cough, and clicking tongue) are often included in daily conversation more than in the conversations recorded in studios. The error rate of conversation atmosphere estimation increases when using real daily conversation because the F0 characteristics are different from those of normal speech.

In this paper, we describe an identification method for target normal speech utterances from ambient speech utterances and laughter utterances, which are frequently included in daily conversation.

2 Our Conversation Support System

The left side of Fig. 2 shows a scene of use of our conversation support system listening to a conversation between two speakers and estimating whether the conversation is progressing smoothly. The plush-doll on the table is our conversation support system. The two right pictures in Fig. 3 show the microphone for recording speech. The microphone is a small directional condenser microphone. Several microphones are placed just in front of the chairs on the sides of the table. Furthermore, it is desirable that the position of the microphone is adjustable according to the sitting positions of the speakers. In such locations, even if the microphone is installed near the user, background conversations of other people are frequently inputted.

Fig. 2
figure 2

Usage scene of conversation support system (left)

Full size image
Fig. 3
figure 3

Location of the microphone for recording conversation utterances (right)

Full size image

3 Identification Method for Differentiating Between Ambient and Target Speech

We used our conversation support system in a restaurant during lunchtime to evaluate it for practical use. The system regarded signals as target utterances when the power level of signals exceeded the threshold. Over a three-minute period, over seventy utterances were extracted as target voices. However, the rate of target speech extraction was only 65%. The extracted utterances included several inputs from people sitting at the neighboring table and employees of the restaurant. It is necessary to distinguish between the speech of ambient speakers and the target speaker.

To exclude background speech, several sound source separation methods have been proposed using microphone-arrays [5, 6]. There are also several identification methods for the separation of sound sources based on the use of a single microphone, such as binary masking using a Bayesian network and non-negative matrix factorization [7, 8]. These methods are effective for separating target sounds from background noise. However, the associated system tends to be complex, and the separation of target sound sources from nonstationary noise, such as ambient speech, is typically insufficient.

We use the standard deviation values of SP (SD-SP) for each utterance to identify the target speaker located near a microphone. In previous our paper [9], we found that the SD-SP of each utterance when speakers talk near the microphone tends to be larger than that when speakers talk far from the microphone. However, we confirmed this using only male speakers. In this paper, we report the experience results using both males and females.

3.1 Free Conversation Recording in a Restaurant Environment

To analyze the difference between the acoustic characteristics of speech originating near a microphone and that originating far from a microphone, we recorded several conversations in an experimental room.

To reproduce the acoustic environment of a restaurant, two loudspeakers were placed behind the speaker and noises from actual restaurants were played. We asked a speaker and a partner to talk to each other freely amidst this noise. The partner stood behind the microphone, and its position was fixed. One speaker (Speaker1) stood at two different positions: one was 30 cm from the microphone and the other (Speaker2) was 120 cm from it. We recorded the speaker’s utterances for each spot. When the speaker’s distance is 30 cm from the microphone, the system regards the speaker as the target, whereas when the distance is 120 cm, the system does not regard the speaker as the target due to environmental noise. The former is the conversation that should be accepted by the system, and the latter is the conversation that should be excluded.

Table 1 lists the recording conditions. Figure 4 shows the experimental layout used to record the conversations.

Fig. 4
figure 4

Schematic diagram of experimental layout

Full size image
Table 1. Conversation recording conditions
Full size table

3.2 Acoustic Analysis and Extraction of Each Utterance

We extracted the speaker’s utterances from the recorded conversation. The threshold level was decided using Eq. (1), and only parts above the threshold level are extracted as utterance parts.

$$ Threshold\; \, level \, = \, \left( {Average\;of\; \, ambient \, \;sound \, \;power} \right) + 5\;{\text{dB}} $$
(1)

We calculated the F0 values and SP values of each utterance as acoustic parameters. The analysis conditions and specifications are listed in Table 2.

Table 2. Parameter extraction details
Full size table

3.3 Standard Deviation of F0 or SP Value of Each Utterance

We calculated standard deviation values of SP (SD-SP) as well as the SD-F0 of each utterance. Their relationship is shown in Fig. 5. A comparison was made between the values obtained for utterances originating near the microphone (distance of 30 cm) and those for utterances originating far from the microphone (distance of 120 cm). The line in Fig. 5 represents the boundary calculated using the LDA method. Figure 5 indicates the following:

  • The SD-SP values at 30 cm are higher than those at 120 cm. The difference between the two sets of values is significant, based on the F-value analysis (confidence level of 95%). The tendency of the relationship is independent of male and female.

  • The difference in the SD-F0 values between the 30 cm and 120 cm cases is small, and the difference is not significant based on the F-value analysis.

Fig. 5
figure 5

Relationship of the standard deviation between F0 and SP

Full size image

We confirmed that the SD-SP values at 30 cm were higher than those at 120 cm as follows:

  1. (1)

    The SD-SP values for each utterance decrease as the distance between the microphone and the loudspeaker is increased.

  2. (2)

    The SD-SP values of loud utterances tend to be smaller than those of normal utterances.

To clarify the reason why the SD-SP values of loud utterances are smaller than that of normal utterances, we compared the changes in the time axis of SP values between loud utterances and normal utterances. We recorded two kinds of utterances. One was the normal utterance recorded in the room without noise, and the other was the loud utterance spoken while listening to the restaurant’s noises with headphones

Figure 6 shows examples of the two kinds of utterances. The left figure shows an example of a loud utterance’s SP, and the right figure shows an example of a normal utterance’s SP. The dotted lines in these figures represent the threshold levels when each utterance is spoken in a noisy restaurant. When comparing the shapes of the changes on the time axis of SP values, the flat part indicated dotted circle is found only in the upper threshold of the loud utterance for the Lombard effect. In a normal utterance, the shape of the SP has a tendency to change constantly. This is the reason the SD-SP of the loud utterance is smaller than that of the normal utterance.

Fig. 6
figure 6

Examples of waveform and sound power for loud utterance and normal utterance

Full size image

3.4 Evaluation of Identification of Target Speech Using SD-SP and SD-F0

We estimated the distance between the speaker and the microphone for each speaker using the support vector machine (SVM) function for the other three speakers, and we evaluated an estimation performance by calculating the recall rates, precision rates, and F-measure rates (harmonic average value). Table 3 lists the values of each rate. The talker A–D are males and the talker E and F are females. Although the recall rate and precision rate varied depending on the speaker, the average rate across all four speakers was 80.5%. This result suggests that the realization of a conversation support system that can extract only the utterances of a target speaker near a microphone is feasible, even in an environment with ambient noise.

Table 3. Estkmation performance of each speaker
Full size table

4 Differentiation Between Laughter Utterances and Normal Speech Utterances

Daily conversation includes several non-language utterances, such as laughter, coughing, and tongue clicking. The rates of non-language utterances in the free conversation database are 19%. The utterances of the elderly include many types of non-language utterances, but 83% of utterances are “laughter”. We have already proposed an identification method between laughter utterances and normal speech utterances using the standard deviations of F0 values for each utterance [8]. However, the identification performance was not high. The F0 values and SP values characteristic of laughter utterances depend on the speaker, and it was not easy to detect the boundary between normal speech and laughter utterances independent of speakers. In this paper, we report the identification experiment results for laughter speech using the MFCC in addition to F0 and SP parameters.

4.1 Free Conversation Recording

We recorded six sets of daily conversations by six males as participating speakers. They met for the first time with each other. Figure 7 shows the location of the conversation recording. We used two microphones and a video camera for this purpose. Figure 8 shows an example photo extracted from the video data. The conditions of the recordings are listed in Table 4.

Fig. 7
figure 7

Location of recording conversation

Full size image
Fig. 8
figure 8

Example photo extracted from the video data

Full size image
Table 4. Conditions of conversation: estimation performance of each speaker
Full size table

4.2 Laughter Utterance Extraction

We extracted both normal speech utterances and laugher utterances from the recording data. The laughter utterances were classified into three types according to social function in Tanaka’s paper [10]. Table 5 indicates the number of utterances for each type of extracted utterance.

Table 5. Number of extracted utterances of laughter and normal speech
Full size table

4.3 Acoustic Analysis Conditions

We calculated the F0, SP, and MFCC values of both normal speech and laughter utterances shown in Table 5 as acoustic parameters, and the standard deviations of these parameters were calculated. In a practical noisy environment, the threshold should be set above the SP of noise, and only the part of the input signal over the threshold is regarded as utterances. In the experiment, we calculated the average SP value of each utterance and regarded it as a threshold. We calculated the acoustic parameters of only the utterance parts above the threshold. The analysis conditions and specifications are listed in Table 6.

Table 6. Parameter extraction details
Full size table

4.4 Normal Speech Identification from Laughter Utterances by SVM

We confirmed the identification performance between the normal speech and laughter utterances using F0, SP, and MFCC. We used an SVM for identification. In particular, to confirm the effectiveness of MFCC, we compared the performance using only F0 and SP with that using F0, SP, and MFCC. Tables 7 and 8 show the identification performances in terms of recall, precision, F-measure, and accuracy of normal speech extraction from the database shown in Table 4.

To improve the conversation-atmosphere estimation performance, it is important that all extracted utterances are normal speech utterances. When the “True Positive (TP)”, “True Negative (TN)”, “False Positive (FP)”, and “False Negative (FN)” are counted on the side of normal speech utterances, the accuracy rate is defined by Eq. (2).

$$ Accuracy \, = \, \left( {TP \, + \, FN} \right) \, / \, \left( {TP \, + \, FP \, + \, FN \, + \, TN} \right) $$
(2)
Table 7. Identification rates of each speaker using F0, SP, and MFCC
Full size table
Table 8. Identification rates of each speaker using F0 and SP
Full size table

Tables 7 and 8 show the following:

  • The accuracy rate was approximately 69.23%. When using MFCC parameters in addition to F0 and PS, the accuracy rate increases compared to using only F0 and PS (from 61.55% to 69.23%).

  • For our conversation atmosphere estimation, a high precision rate is desirable. The precision rate was improved by approximately 10% by using MFCC (from 62.89% to 73.16%).

  • The performance depends on the speaker. In particular, the accuracy rates are decreased for speakers B and C. However, the differences in the accuracy tend to decrease when using MFCC.

4.5 Discussion

For the conversation atmosphere estimation, all extracted utterances should be normal speech utterances without laughter utterances, and the precision rates are important. As a result, the precision rates were 73.16%. This is insufficient for identification performance. However, if we extract all the utterances as normal speech, the precision rate is 53.29% (number of normal speech utterances/number of all utterances shown in Table 5). The results suggest that the precision rate would be improved drastically. In addition, the MFCC parameters are quite effective in improving the precision rates.

To clarify the reason for the effectiveness of MFCC, we compared the standard deviation of MFCC (SD-MFCC) values between the normal speech utterances and laughter utterances.

Figure 9 shows the comparison results between normal and laughter utterances using SD-MFCC values of the 2nd dimension values and the 4th dimension values. The SVM method was used for identification. The curved line in Fig. 9 indicates the boundary between normal speech and laughter by SVM.

This figure indicates that the area plotted by the MFCC values of laughter utterances is small. Almost all values of both the 2nd dimension and 4th dimension are plotted within the SVM boundary. However, the area in the case of normal speech is wide. The results indicate that it is difficult to separate both areas, but it is possible to decide the area plotting only normal speech utterances. The results suggest that MFCC values are only useful for extracting normal speech utterances.

The identification performance shown in Fig. 9 depends on the speaker. In the future, it is necessary to develop a speaker adaptation method to improve its performance.

Fig. 9
figure 9

Comparison between normal and laughter utterances using SD-MFCC values (the 2nd dimension and the 4th dimension).

Full size image

5 Conclusion

We studied an identification method of target speech utterance to use a conversation support system in a lounge environment. We found that the standard deviation of the SP values is effective for identifying the target speech utterance from ambient speech. As a result of an identification experiment using the SVM method, although the recall rate and precision rate varied depending on the speaker, the average rate across all six speakers was 80.5%. We also confirmed the identification performance between normal speech and laughter utterances using F0, SP, and MFCC. The identification rate of normal speech utterances was 73.1% as the precision rate.

These results indicate that these acoustic characteristics would be effective in conversation atmosphere estimation and suggest that our conversation support system could be useful in practical scenarios.