Skip to main content
SpringerLink
Log in

MOS estimation model development using ACR listening-opinion tests with Thai users referring to loss effects: a case of G.726 and G.729

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

This paper proposes two models of Mean Opinion Score (MOS) estimation based on Thai users and the Thai language, referring to packet loss effects, for G.726 and G.729 codecs. Based on Thai users and Thai speech referring to packet loss effects in this work, the Absolute Category Rate (ACR) listening tests were conducted with 89 participants and 107 participants for the MOS estimation model development of G.726 and G.729 respectively, while the same tests were conducted with totally 60 participants for the model evaluation of both codecs. Packet loss rates were 0–15% for G.726 with 5 test conditions and G.729 with 6 test conditions; each condition was conducted with at least 16 participants. After gathering the data, the MOS estimation models for both codecs were simply created and then evaluated with the test sets, comparing Perceptual Evaluation of Speech Quality (PESQ), a popular measurement method. For one of the contributions of this study, after the models were evaluated using Mean Absolute Percentage Error (MAPE), it was found that the proposed models for G.726 and G.729 provided better performance than PESQ, particularly by reducing the MAPE by about 30% and 17% respectively, compared to PESQ.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. De Rango, F., Tropea, M., Fazio, P., Marano, S.: Overview on VoIP: subjective and objective measurement methods. Int. J. Comput. Sci. Netw. Secur. 6(1B), 140–153 (2006)

    Google Scholar 

  2. ITU-T.: ITU-T Recommendation P.800 Methods for subjective determination of transmission quality (1996)

  3. Streijl, R.C., Winkler, S., Hands, D.: Mean opinion score (MOS) revisited: methods and applications limitations and alternatives. Multimed. Syst. 22(2), 213–227 (2016)

    Article  Google Scholar 

  4. Baharudin, M.A.B., Quang, T.M., Kamioka, E.: Improvement of handover performance based on bio-inspired approach with received signal strength and mean opinion score. Arab J. Sci. Eng. 40, 1623–1636 (2015)

    Article  Google Scholar 

  5. Daengsi, T., Wuttidittachotti, P.: VoIP-Quality of experience modeling: E-model and simplified E-model enhancement using bias factor. Multimed. Tools Appl. http://dx.doi.org/10.1007/s11042-016-3389-z. Accessed 12 April 2016

  6. ITU-T.: Question 7/12—methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions. http://www.itu.int/ITU-T/studygroups/com12/sg12-q7.html. Accessed 12 April 2016

  7. Cai, Z., Kitawaki, N., Yamada, T., Makino, S.: Comparison of MOS evaluation characteristics for Chinese, Japanese and English in IP telephony. In: Proceedings of IUCS 2010, Beijing, pp. 1–4 (2010)

  8. Wuttidittachotti, P., Khaoduang, P., Daengsi, T.: Development of a MOS estimation model for G.729 using listening-opinion tests with Thai speech referring to packet loss effects. In: Proceedings of ISCAIE 2014, Penang, pp. 29–32 (2014)

  9. Daengsi, T., Wutiwiwatchai, C., Preechayasomboon, A., Sukparungsee, S.: IP telephony: comparison of subjective assessment methods for voice quality evaluation. Walailak J. Sci. Technol. 11(2), 87–92 (2014)

    Google Scholar 

  10. Daengsi, T., Khitmoh, N., Wattidittachotti, P.: VoIP quality measurement: subjective VoIP quality estimation model for G.711 and G.729 based on native Thai users. Multimed. Syst. http://dx.doi.org/10.1007/s00530-015-0468-3. Accessed 12 April 2016

  11. Sodanil, M., Nitsuwat, S., Haruechaiyasak, C.: Thai word recognition using hybrid MLP-HMM. Int. J. Comput. Sci. Netw. Secur. 10, 103–110 (2010)

    Google Scholar 

  12. Daengsi, T., Wutiwiwatchai, C., Preechayasomboon, A., Sukparungse S.: A study of VoIP quality evaluation: user perception of voice quality from G.729, G.711 and G.722. In: Proceedings of IEEE CCNC—SS-QoE, Las Vegas, pp. 342–345 (2012)

  13. Lindeberg, M., Kristiansen, S., Plagemann, T., Goebel, V.: Challenges and techniques for video streaming over mobile ad hoc networks. Multimed. Syst. 17, 51–82 (2011)

    Article  Google Scholar 

  14. Goudarzi. M.: Evaluation of Voice Quality in 3G Mobile Networks. Thesis, University of Plymouth (2008)

  15. Al-Akhras, M., Zedan, H., John, R., Almomani, I.: Non-intrusive speech quality prediction in VoIP networks using a neural network approach. Neurocomputing 72, 2595–2608 (2009)

    Article  Google Scholar 

  16. Mahdi, A.E., Picovici, D.: Advances in voice quality measurement in modern telecommunications. Dig. Signal Process. 19, 79–103 (2009)

    Article  Google Scholar 

  17. Ding, L., Lin, Z., Radwan, A., El-Hennaway, M., Goubran, R.: Non-intrusive single-ended speech quality assessment in VoIP. Speech Commun. 49, 477–489 (2007)

    Article  Google Scholar 

  18. Karapantazis, S., Pavlidou, F.-N.: VoIP: a comprehensive survey on a promising technology. Comput. Netw. 53(12), 2050–2090 (2009)

    Article  Google Scholar 

  19. ITU-T.: ITU-T Recommendation P.800.1 Mean opinion score (MOS) terminology (2006)

  20. Telchemy.: Voice quality measurement. http://www.telchemy.com/appnotes/TelchemyVoiceQualityMeasurement.pdf. Accessed April 12 2016

  21. ITU-T.: ITU-T Recommendation G.729, coding of speech at 8 kb/s using conjugate-structure algebraic-code-excited linear prediction (CS-ACELP) (2012)

  22. ITU-T.: ITU-T Recommendation G.726, 40, 32, 24, 16 kbit/s adaptive differential PulseCode modulation (ADPCM) (1990)

  23. ITU-T.: ITU-T Recommendation P.862, Perceptual evaluation of speech quality (PESQ): an objective method for end-to-end speech quality assessment of narrow-band telephone (2001)

  24. Voznak, M., Rozhon, J.: Influence of atmospheric parameters on speech quality in GSM/UMTS. Int. J. Math. Model. Method. Appl. Sci. 6(4), 575–582 (2012)

    Google Scholar 

  25. Wuttidittachotti, P., Daengsi, T.: Quality evaluation of mobile networks using VoIP applications: a Case Study with Skype and LINE based-on Stationary Tests in Bangkok. Int. J. Comput. Netw. Inform. Security. 7(12), 28–41 (2015)

    Article  Google Scholar 

  26. Goudarzi, M., Sun, L.: Performance analysis and comparison of PESQ and 3SQM in live 3G mobile networks. http://www.tech.plym.ac.uk/spmc/staff/mgoudarzi/Performance%20analysis%20and%20comparison%20of%20PESQ%20and%203SQM.pdf. Accessed 12 April 2016

  27. Jiang, W., Schulzrinne, H.: Comparison and optimization of packet loss repair methods on VoIP perceived quality under bursty loss. In: Proceedings of NOSSDAV’02, Miami, pp. 73–81 (2002)

  28. Zhang, H., Xie, L., Byun, J., Flynn, P., Shim, Y.: Packet loss burstiness and enhancement to the E-model. In: Proceedings of SNPD/SAWN 2005. Towson, pp. 214–219 (2005)

  29. ITU-T.: ITU-T Recommendation G.107 The E-model: a computational model for use in transmission planning (2011)

  30. Ding, L., Goubran, R.A.: Speech quality prediction in VoIP using the extended E-model. In: Proceedings of IEEE GLOBECOM 2003, San Francisco, vol. 7, pp. 3974–3978 (2003)

  31. Sun, L., Ifeachor, E.C.: Voice quality prediction models and their application in VoIP networks. IEEE Trans. Multimed. 8(4), 809–820 (2006)

    Article  Google Scholar 

  32. Ren, J., Zhang, H., Zhu, Y., Gao, C. Assessment of effects of different language in VOIP. In: Proceedings of ICALIP 2008, Shanghai, pp. 1624–1628 (2008)

  33. Ren, J., Zhang, C., Huang, W., Mao, D.: Enhancement to E-model on standard deviation of packet delay. In: Proceedings of ICIS 2010, Chengdu, pp. 256–259 (2010)

  34. Raja, A., Azad, R.M.A., Flanagan, C., Ryan, C.: Evolutionary speech quality estimation in VoIP. Soft. Comput. 15, 89–94 (2011)

    Article  Google Scholar 

  35. Jiang, C., Huang, P.: Research of monitoring VoIP voice QoS. In: Proceedings of ICICIS 2011, Hong Kong, pp. 499–502 (2011)

  36. Assem, H., Malone, D., Dunne, J., O’Sullivan, P.: Monitoring VoIP call quality using improved simplified E-model. In: Proceedings of ICNC 2013, San Diego, pp. 927–931 (2013)

  37. Adel, M. et al.: Improved E-model for monitoring quality of multi-party VoIP communications. In: Proceedings of IEEE Globecom Workshops 2013, Atlanta, pp. 1180–1185 (2013)

  38. Hines, A., Skoglund, J., Kokaram, A.C., Harte, N.: ViSQOL: an objective speech quality model. EURASIP J. Audio, speech, and music processing, http://dx.doi.org/10.1186/s13636-015-0054-9. Accessed 12 April 2016

  39. Jung, Y., Manzano, C.: Burst packet loss and enhanced packet loss-based quality model for mobile voice-over internet protocol applications. IET Commun. 8(1), 41–49 (2014)

    Article  Google Scholar 

  40. Rahdari, F., Eftekhari, M., Akbari, A., Zeinalkhani, M.: Developing fuzzy models for estimating the quality of VoIP. Iran. J. Fuzzy Syst. 11(1), 49–73 (2014)

    MathSciNet  Google Scholar 

  41. Triyason, T., Kanthamanon, P.: E-model modification for multi-languages over IP. Elektronika ir Elektrotechnika. 21(1), 82–87 (2015)

    Article  Google Scholar 

  42. Takahashi, A., Kurashima, A., Yoshino, H.: Objective assessment methodology for estimating conversational quality in VoIP. IEEE Audio Speech Lang Process. 14(6), 1983–1993 (2006)

    Google Scholar 

  43. Tsiaras, C., Rösch, M., Stiller, B.: VoIP-based calibration of the DQX model. In: Proceedings of IFIP Networking 2015. Toulouse, pp. 1–9 (2015)

  44. Daengsi, T., Preechayasomboon, A., Sukparungsee, S., Chootrakoo, P., Wutiwiwatchai, C.: The development of a Thai speech set for telephonometry. In: Proceedings of oriental-COCOSDA 2010, Kathmandu, Nepal, paper 53 (2010)

  45. Carbone, M., Rizzo, L.: Dummynet revisited. ACM SIGCOMM Comput. Commun. Rev. 40(2), 12–20 (2010)

    Article  Google Scholar 

  46. Daengsi, T., Yochanang, K., Wuttiditachotti, P.: A study of perceptual VoIP quality evaluation with Thai users and codec selection using voice quality: bandwidth tradeoff analysis. In: Proceedings of ICTC 2013, Jeju, pp. 691–696 (2013)

  47. ITU-T.: Annex B Speech Files, ITU-T Recommendation P.501. http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=10000501. Accessed 12 April 2016

  48. Aydin, G., Karakurt, I., Hamzacebi, C.: Performance prediction of diamond sawblades using artificial neural network and regression analysis. Arab. J. Sci. Eng. 40, 2003–2012 (2015)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to dedicate the contributions of this paper to His Majesty the late King Bhumibol Adulyadej of Thailand who was called the “Father of Thai Invention,” the “Father of Thai Technology,” the “Father of Thai Innovation” and the “Father of Royal Rainmaking” on the occasion of his passing away. Last but not least, thank you to all participants for the ACR tests and the Speech and Audio Laboratory, NECTEC for TSST. Thanks to the VoIP Laboratory, School of Information Technology, KMUTT for PESQ measurement tool, particularly Dr. Tuul Triyason and Asst. Prof. Dr. Vajirasak Vanijja for support. Finally, thank you to Mr. Gary Sherriff for editing.

Author information

Authors and Affiliations

  1. Department of Data Communication and Networking, Faculty of Information Technology, KMUTNB, Bangkok, Thailand

    Pongpisit Wuttidittachotti

  2. Department of Information Technology, Faculty of Industry and Technology, RMUTR (Wangkraikangwon Campus), Prachuap Khiri Khan, Thailand

    Phisit Khaoduang

  3. Enterprise Services, JADS Comm Limited, Bangkok, Thailand

    Therdpong Daengsi

Authors
  1. Pongpisit Wuttidittachotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Phisit Khaoduang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Therdpong Daengsi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Therdpong Daengsi.

Additional information

Communicated by T. Plagemann.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wuttidittachotti, P., Khaoduang, P. & Daengsi, T. MOS estimation model development using ACR listening-opinion tests with Thai users referring to loss effects: a case of G.726 and G.729. Multimedia Systems 24, 285–295 (2018). https://doi.org/10.1007/s00530-017-0549-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-017-0549-6

Keywords

Search

Navigation