Skip to main content
SpringerLink
Log in

Evaluation of a dynamic classification method for multimodal ambiguities based on Hidden Markov Models

  • Original Paper
  • Published:
Evolving Systems Aims and scope Submit manuscript

Abstract

The wide interest in ambiguities is because it represents uncertainty but also a fundamental item of discussion for who is interested in the interpretation of languages also considering that it is functional for communicative purposes. This paper addresses ambiguity issues in terms of identification of the meaningful features of multimodal ambiguities and it evaluates a dynamic HMM-based classification method that is able to classify ambiguities by learning, and progressively adapting the model to the evolution of the interaction, refining the existing classes, or identifying new ones. The comparative evaluation of the considered method of the considered method with other surveyed methods demonstrates an improvement considering the performance evaluation measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://nlp.stanford.edu/software/

  2. https://tec.citius.usc.es/stac/

References

  • Akshay S, Bazille H, Fabre E, Genest B Classification among Hidden Markov Models. In: FSTTCS 2019—39th IARCS annual conference on. foundations of software technology and theoretical computer science, Dec 2019, Bombay, India. pp. 1–14, ff10.4230/LIPIcs.FSTTCS.2019.29ff. ffhal-02350252f

  • An A (2003) Learning classification rules from data. Comput Math Appl 45:737–748

    Article  MathSciNet  Google Scholar 

  • Angelov P, Zhou X (2008) On line learning fuzzy rule-based system structure from data streams. In: IEEE international conference on fuzzy systems (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp 915–922.

  • Antal M (2004) Speaker independent phoneme classification in continuous speech. Stud Univ Babes-Bolyal Inform 49(2):55–64

    MATH  Google Scholar 

  • Aoki PM, Woodruff A (2005) Making space for stories: ambiguity in the design of personal communication systems. Proc CHI 2005:181–190

    Google Scholar 

  • Argyropoulos S, Moustakas K, Karpov A, Aran O, Tzovaras D, Tsakiris T, Varni G, Kwon B (2008) Multimodal user interface for the communication of the disabled. J Multimodal User Interfaces 2(2):105–116 (Springer-Verlag)

    Article  Google Scholar 

  • Benesch T (2001) The Baum-Welch algorithm for parameter estimation of Gaussian autoregressive mixture models. J Math Sci (New York) 105:2515–2518

    Article  MathSciNet  Google Scholar 

  • Ben-Gal I (2007) Bayesian networks. In: Ruggeri F, Faltin F, Kenett R (eds) Encyclopedia of statistics in quality & reliability. Wiley, Hoboken

    Google Scholar 

  • Berry DM, Gacitua R, Sawyer P, Tjong SF (2012) The case for dumb requirements engineering tools. In: REFSQ, ser. LNCS vol 7195, Springer, Pp 211–217

  • Berry DM, Kamsties E, and Krieger MM, (2003) From contract drafting to software specification: linguistic sources of ambiguity, Technical Report, University of Waterloo, Waterloo, ON, Canada, https://cs.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf Accessed 12 July 2017.

  • Berry DM, Kamsties E, Kay DG, Krieger MM (2001) From contract drafting to software specification: linguistic sources of ambiguity. Technical Report, University of Waterloo, Waterloo, ON, Canada.

  • Burges C (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):1–47

    Article  Google Scholar 

  • Caschera MC (2009) Interpretation methods and ambiguity management in multimodal systems. In: Grifoni P (ed) Multimodal human computer interaction and pervasive services. IGI Global (USA), pp 87–102. https://doi.org/10.4018/978-1-60566-386-9.ch005

  • Caschera MC, D’Ulizia A, Ferri F, Grifoni P, (2012) Towards evolutionary multimodal interaction. In: OTM 2012 workshops proceedings, 10–14 September 2012, Rome, Springer-Verlag, Lecture Notes in Computer Science 7567: 608-616

  • Caschera MC, Ferri F, Grifoni P (2007b) An approach for managing ambiguities in multimodal interaction. OTM 2007 Ws, Part I, LNCS 4805. Springer-Verlag Berlin Heidelberg 2007: 387–397

  • Caschera MC, Ferri F, Grifoni P (2008) Ambiguity detection in multimodal systems. In: Levialdi S (ed) AVI 2008—Proceedings of the working conference on advanced visual interfaces May 28–30, 2008, Napoli, Italy: 331–334.

  • Caschera MC, Ferri F, Grifoni P (2013a) From modal to multimodal ambiguities: a classification approach. JNIT 4(5):87–109

    Article  Google Scholar 

  • Caschera MC, Ferri F, Grifoni P (2013b) InteSe: an integrated model for resolving ambiguities in multimodal sentences. IEEE Trans Syst Man Cybern 43(4):911–931

    Article  Google Scholar 

  • Caschera MC, Ferri F, Grifoni P, (2007a) The management of ambiguities. Visual languages for interactive computing: definitions and formalizations. IGI Publishing, pp 129–140.

  • Caschera MC, Ferri F, Grifoni P (2007) Multimodal interaction systems: information and time features. Int J Web Grid Services IJWGS 3(1):82–99

    Article  Google Scholar 

  • Chen HS, Tsai WJ (2016) Incorporating frequent pattern analysis into multimodal HMM event classification for baseball videos. Multimed Tools Appl 75(9):4913–4932

    Article  Google Scholar 

  • Cheng J, Greiner R, Kelly J, Bell D, Liu W (2002) Learning Bayesian networks from data: an information-theory based approach. Artif Intell 137:43–90

    Article  MathSciNet  Google Scholar 

  • D’Ulizia A, Ferri F, Grifoni P (2010) Generating multimodal grammars for multimodal dialogue processing. IEEE Trans Syst Man Cybern Part A Syst Hum 40(6):1130–1145

    Article  Google Scholar 

  • El-yacoubi A, Sabourin R, Gilloux M, Suen CY (1999). Off-Line Handwritten Word Recognition Using Hidden Markov Models –, Ecole De Technologie Supérieure, Département Reconnaissance, Modélisation Optimisation (rmo, Catolica Parana)

  • Favetta F, Aufaure-Portier MA, (2000) About ambiguities in visual GIS query languages: a taxonomy and solutions. In: Proceedings of the 4th international conference on advances in visual information systems, Springer-Verlag, pp 154–165.

  • Futrelle RP, (1999) Ambiguity in visual language theory and its role in diagram parsing. In: IEEE symposium on visual languages, Tokyo, IEEE Computer Soc. 172–175.

  • Gleich B, Creighton O, and Kof L (2010) Ambiguity detection: towards a tool explaining ambiguity sources. In: Proc. of REFSQ’10, ser. LNCS, vol. 6182. Springer, pp 218–232.

  • Gong S, Loy CC, Xiang T (2011) Security and surveillance. Vis Anal Hum 2011:455–472

    Article  Google Scholar 

  • Grifoni P, Caschera MC, Ferri F (2020) DAMA: a dynamic classification of multimodal ambiguities. Int J Comput Intell Syst 13(1):178–192. https://doi.org/10.2991/ijcis.d.200208.001

    Article  Google Scholar 

  • Hegde V (2012) Multi-perspective comparative study: common context based knowledge integration in word sense disambiguation for information retrieval. P.hD thesis in Computer Science and Engineering from Avinashilingam University, Coimbatore India.

  • Hodges JL, Lehmann EL (1962) Ranks methods for combination of independent experiments in analysis of variance. Ann Math Stat 33:482–497

    Article  MathSciNet  Google Scholar 

  • Jablonka E, Ginsburg S, Dor D (2012) The co-evolution of language and emotions. Philos Trans R Soc 367(1599):2152–2159

    Article  Google Scholar 

  • Jamil U, Khalid S (2014) Comparative study of classification techniques used in skin lesion detection systems. 266–271. 10.1109/INMIC.2014.7097349.

  • Josinski H, Kostrzewa D, Michalczuk A, Switonski A, Wojciechowski KW (2013) Feature extraction and HMM-based classification of gait video sequences for the purpose of human identification. Vision Based Systems for UAV Applications. Volume 481 of the series Studies in Computational Intelligence: pp 233–245

  • Kessous L, Castellano G, Caridakis G (2010) Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J Multimodal User Interfaces 3(1):33–48

    Article  Google Scholar 

  • Kishansingh R, Bhavesh AO (2017) A comparative study of classification techniques in data mining. Int J Creat Res Thoughts (IJCRT) 5(3):154–163

    Google Scholar 

  • Kiyavitskaya N, Zeni N, Mich L, and Berry DM, (2007) Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. In: Proc. of WER’07, pp 197–206

  • Kotsianti. SB (2007) Supervised machine learning: a review of classification techniques. In: Proceedings of the 2007 conference on emerging artificial intelligence applications in computer engineering: real word ai systems with applications in eHealth, HCI, Information Retrieval and Pervasive Technologies. pp 3–24

  • Kotsiantis S (2007) Supervised machine learning: a review of classification techniques. Informatica (Ljubljana). Informatica 31(3):249–268

    MathSciNet  MATH  Google Scholar 

  • Lim TS, Loh WY, Shih YS (2000) A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach Learn 40:203–228

    Article  Google Scholar 

  • Liu N, Lovell BC (2003) Gesture Classification using hidden markov models and viterbi path counting. In: Sun C, Talbot H, Ourselin S, Adriaansen T (eds) Proceedings of the seventh biennial Australian pattern recognition society conference. The seventh biennial australian pattern recognition society conference, Sydney: 273–282. 10–12 December

  • Lu C, Drew MS, Au J (2001) Classification of summarized videos using hidden markov models on compressed chromaticity signatures. In: MULTIMEDIA '01: Proceedings of the ninth ACM international conference on multimedia, October 2001, pp 479–482. https://doi.org/10.1145/500141.500217

  • Malcangi M, Grew P (2017) Evolving connectionist method for adaptive audiovisual speech recognition. Evol Syst 8(1):85–94. https://doi.org/10.1007/s12530-016-9156-6

    Article  Google Scholar 

  • Marcus MP, Santorini B, Marcinkiewicz MA (1994) Building a large annotated corpus of english: the penn treebank. Comput Linguist 19(2):313–330

    Google Scholar 

  • Martalo’ A, Novielli N, de Rosis F (2008) Attitude display in dialogue patterns. In: Proceedings of AISB’08, symposium on ‘affective language in human and machine’

  • Massey AK, Rutledge RL, Anton AI and Swire PP (2014) Identifying and classifying ambiguity for regulatory requirements. In: Requirements engineering conference (RE), 2014 IEEE 22nd International: pp 83–92

  • Maurya HC, Gupta P, Choudhary N (2015) Natural language ambiguity and its effect on machine learning. Int J Modern Eng Res (IJMER) 5(4):25–30

    Google Scholar 

  • Mavrogiorgou A, Kiourtis A, Kyriazis D (2017) A Comparative study of classification techniques for managing IoT devices of common specifications. In: Pham C, Altmann J, Bañares J (eds) Economics of grids, clouds, systems, and services. GECON 2017. Lecture notes in computer science, vol 10537. Springer, Cham

    Google Scholar 

  • McLuhan M, Fiore Q (1967) The medium is the massage. Random House, New York

    Google Scholar 

  • Mitchell TM (1997) Machine learning. McGraw-hill, New York

    MATH  Google Scholar 

  • Mittal P, Gill NS (2014) A comparative analysis of classification techniques on medical data sets. IJRET 03(06):454–460

    Article  Google Scholar 

  • Mouret M, Solnon C, Wolf C (2008) Classification of images based on Hidden Markov Models. In: IEEE workshop on content based multimedia indexing, pp 169–174

  • Nikam SS (2015) A comparative study of classification techniques in data mining algorithms. Orient J Comp Sci Technol 8(1):13–19

    Google Scholar 

  • Novielli N (2010) HMM modeling of user engagement in advice-giving dialogues. J Multimodal User Interf 3(1):131–140

    Article  Google Scholar 

  • Oliver N, Horvitz E (2005) A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In: Ardissono L, Brna P, Mitrovic A (eds) User modeling 2005. UM 2005. Lecture notes in computer science, vol 3538, Springer, Berlin, Heidelberg

  • Patel JA (2015) Classification algorithms and comparison in data mining. Int J Innovations Adv Comput Sci ISSN 2347—8616, Volume 4, Special Issue

  • Quinlan R (1996) Improved Use of Continuous Attributes in C4.5. Journal of Artificial Intelligence Research 4:77–90

    Article  Google Scholar 

  • Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–285

    Article  Google Scholar 

  • Rodríguez-Fdez I, Canosa A, Mucientes M, Bugarín A, (2015) STAC: a web platform for the comparison of algorithms using statistical tests, In: Proceedings of the 2015 IEEE international conference on fuzzy systems (FUZZ-IEEE), 2015.

  • Skowron A, Wang H, Wojna A, Bazan J (2006) Multimodal classification: case studies. In: Peters JF, Skowron A (eds) Transactions on rough sets V. Lecture notes in computer science, vol 4100. Springer, Berlin, Heidelberg, pp 224–239. https://doi.org/10.1007/11847465_11

    Chapter  Google Scholar 

  • Stacey M, Eckert C (2003) Against ambiguity. Comput Support Coop Work 12:153–183

    Article  Google Scholar 

  • Stolcke A, Coccaro N, Bates R, Taylor P, Van Ess-Dykema C, Ries K, Shriberg E, Jurafsky D, Martin R, Meteer M (2000) Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput Linguist 26:3

    Article  MathSciNet  Google Scholar 

  • Tan PN, Steinbach M, Kumar V (2005) Classification: basic concepts, decision trees, and model evaluation. In: Introduction to data mining, 1st edn. Addison-Wesley, pp 145–205

  • Tharwat A Classification assessment methods. Appl Comput Informatics, 2018, ISSN 2210–8327, https://doi.org/10.1016/j.aci.2018.08.003.

  • Tung T, Gomez R, Kawahara T, Matsuyama T (2014) (2014) Multiparty interaction understanding using smart multimodal digital signage. IEEE Trans Hum Mach Syst 44(5):625–637

    Article  Google Scholar 

  • Twitchell DP, Adkins M, Nunamaker JF, Burgoon JK (2004) Using speech act theory to model conversations for automated classification and retrieval. In: Procs of the 9th international working conference on the language-action perspective on communication modeling: pp 121–130

  • Vigliocco G, Perniss P, Vinson D (2014) Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philos Trans R Soc B 369(1651):1–7. https://rstb.royalsocietypublishing.org/content/royptb/369/1651/20130292.full.pdf Accessed 12 July 2017

  • Yang H, Roeck AND, Gervasi V, Willis A, Nuseibeh B (2011) Analysing anaphoric ambiguity in natural language requirements. Requir Eng 16(3):163–189

    Article  Google Scholar 

  • Zhang GP (2000) Neural networks for classification: a survey. IEEE Trans Syst Man Cybern Part C 30(4):451–462

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Istituto Di Ricerche Sulla Popolazione E Le Politiche Sociali, Consiglio Nazionale Delle Ricerche, Via Palestro 32, 00185, Rome, Italy

    Patrizia Grifoni, Maria Chiara Caschera & Fernando Ferri

Authors
  1. Patrizia Grifoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Chiara Caschera
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fernando Ferri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Chiara Caschera.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Grifoni, P., Caschera, M.C. & Ferri, F. Evaluation of a dynamic classification method for multimodal ambiguities based on Hidden Markov Models. Evolving Systems 12, 377–395 (2021). https://doi.org/10.1007/s12530-020-09344-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12530-020-09344-3

Keywords

Search

Navigation