skip to main content
10.1145/3555776.3577637acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
research-article

Multimodal Semantographic Metalanguage (MSM): A novel methodology for digital enablement of semi-literates

Published: 07 June 2023 Publication History

Abstract

People in developing countries without tertiary education, face hurdles in using digital platforms for communication. The linguistic diversity of this section of population makes design of near-universal digital enablement methodology a challenging task. It is therefore pivotal to build a language agnostic methodology with bare minimum text to achieve digital communication across language boundaries. This would also help in bridging the "Digital Divide". In this paper, we illustrate building a Multimodal Semantographic Metalanguage (MSM) using Machine Learning (ML), Natural Language Processing (NLP) and Natural Semantic Metalanguage (NSM). The proposed methodology uses pictographs and ideographs, which are visually more distinctive, simpler to understand, have a reduced learning time and appropriate for achieving digital literacy for semi-literates. We establish our claim on a dataset compiled from text messages by semi-literates. We have observed that using the proposed approach, we can successfully communicate semantic elements across semi-literates with different linguistic backgrounds with an accuracy of more than 80%.

References

[1]
Stephen R Anderson. 2010. How many languages are there in the world. Linguistic Society of America (2010).
[2]
Lynda A Archer. 1977. Blissymbolics---A nonverbal communication system. Journal of Speech and Hearing Disorders (1977).
[3]
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.
[4]
Susana Bautista, Raquel Hervás, Agustín Hernández-Gil, Carlos Martínez-Díaz, Sergio Pascua, and Pablo Gervás. 2017. Aratraductor: text to pictogram translation using natural language processing techniques. In Proceedings of the XVIII International Conference on Human Computer Interaction.
[5]
Martin Böcker. 1996. A multiple index approach for the evaluation of pictograms and icons. Computer Standards & Interfaces (1996).
[6]
John Brooke. 1996. Sus: a "quick and dirty'usability. Usability evaluation in industry 189, 3 (1996).
[7]
Christopher Burke. 2009. Isotype representing social facts pictorially. Information Design Journal (2009).
[8]
Richard Oliver Collin. 2010. Ethnologue. Ethnopolitics (2010).
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[10]
David Gittins. 1986. Icon-based human-computer interaction. International Journal of Man-Machine Studies (1986).
[11]
Cliff Goddard. 2006. Natural semantic metalanguage. (2006).
[12]
Cliff Goddard. 2012. Semantic primes, semantic molecules, semantic templates: Key concepts in the NSM approach to lexical typology. Linguistics (2012).
[13]
Abdel Ghani Karkar. 2018. An ontology based text-to-picture multimedia m-learning system. Ph.D. Dissertation.
[14]
Abdel Ghani Karkar, Jihad Mohamad Aja'am, and Arif Mahmood. 2017. Illustrate it! an arabic multimedia text-to-picture m-learning system. (2017).
[15]
Adam Kilgarriff, Pavel Rychly, Pavel Smrz, and David Tugwell. 2004. Itri-04-08 the sketch engine. Information Technology 105, 116 (2004), 105--116.
[16]
Maria Kunilovskaya and Marina Koviazina. 2017. Sketch engine: A toolbox for linguistic discovery. Jazykovedny Casopis 68, 3 (2017), 503.
[17]
Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and usability engineering group. Springer, 63--76.
[18]
Alon Lavie, Kenji Sagae, and Shyamsundar Jayaraman. 2004. The significance of recall in automatic metrics for MT evaluation. In Conference of the Association for Machine Translation in the Americas. Springer, 134--143.
[19]
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning. PMLR, 1188--1196.
[20]
Haojie Li, Jinhui Tang, Guangda Li, and Tat-Seng Chua. 2013. Word2Image: A System for Visual Interpretation of Concepts. Internet Multimedia Search and Mining (2013).
[21]
Kenneth N Lodding. 1983. Iconic interfacing. IEEE Computer graphics and applications (1983).
[22]
Martin C Maguire. 1985. A review of human factors guidelines and techniques for the design of graphical human-computer interfaces. Computers & graphics (1985).
[23]
Alexis Michaud. 2011. Pictographs and the language of Naxi rituals.
[24]
Milan Randic. 2010. Nobel Universal Graphical Language. Xlibris Corporation.
[25]
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
[26]
Horacio Saggion, Elena Gómez-Martínez, Esteban Etayo, Alberto Anula, and Lorena Bourg. 2011. Text simplification in simplext: Making texts more accessible. Procesamiento del lenguaje natural (2011).
[27]
Leen Sevens, Vincent Vandeghinste, Ineke Schuurman, and Frank Van Eynde. 2015. Natural language generation from pictographs. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG). 71--75.
[28]
Prawaal Sharma, Navneet Goyal, and MR Vinay. 2021. Semi-literate Texting (SLT): Survey based text message dataset from digitally semi-literate users in India. Data in Brief (2021).
[29]
Mohamed Ali Hadj Taieb, Torsten Zesch, and Mohamed Ben Aouicha. 2020. A survey of semantic relatedness evaluation datasets and procedures. Artificial Intelligence Review (2020).
[30]
Kristen Tatti. 2016. New IConji Language for the Symbol-minded-BizWest. BizWest (2016).
[31]
Andries Van Dam. 1984. Computer software for graphics. Scientific American (1984).
[32]
Anna Wierzbicka. 1996. Semantics: Primes and universals: Primes and universals. Oxford University Press, UK.
[33]
Xingsi Xue, Haolin Wang, Jie Zhang, Yikun Huang, Mengting Li, and Hai Zhu. 2021. Matching Transportation Ontologies with Word2Vec and Alignment Extraction Algorithm. Journal of Advanced Transportation (2021).
[34]
Jezia Zakraoui, Moutaz Saleh, and Jihad Al Ja'am. 2019. Text-to-picture tools, systems, and approaches: a survey. Multimedia Tools and Applications (2019).
[35]
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. 1997. BIRCH: A new data clustering algorithm and its applications. Data Mining and Knowledge Discovery (1997).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing
March 2023
1932 pages
ISBN:9781450395175
DOI:10.1145/3555776
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. semantographics
  2. human centered computing
  3. digital enablement
  4. semi-literate enablement

Qualifiers

  • Research-article

Conference

SAC '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25
The 40th ACM/SIGAPP Symposium on Applied Computing
March 31 - April 4, 2025
Catania , Italy

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 46
    Total Downloads
  • Downloads (Last 12 months)19
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media