Skip to main content
SpringerLink
Log in

Empirical evaluation and study of text stemming algorithms

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Text stemming is one of the basic preprocessing step for Natural Language Processing applications which is used to transform different word forms into a standard root form. For Arabic script based languages, adequate analysis of text by stemmers is a challenging task due to large number of ambigious structures of the language. In literature, multiple performance evaluation metrics exist for stemmers, each describing the performance from particular aspect. In this work, we review and analyze the text stemming evaluation methods in order to devise criteria for better measurement of stemmer performance. Role of different aspects of stemmer performance measurement like main features, merits and shortcomings are discussed using a resource scarce language i.e. Urdu. Through our experiments we conclude that the current evaluation metrics can only measure an average conflation of words regardless of the correctness of the stem. Moreover, some evaluation metrics favor some type of languages only. None of the existing evaluation metrics can perfectly measure the stemmer performance for all kind of languages. This study will help researchers to evaluate their stemmer using right methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Built in SMART system.

  2. Extensively modified version Lovens (1968) included in SMART system.

  3. https://antiflux.org/dictionary?dict=moby-thesaurus

References

  • Ababneh M, Al-Shalabi R, Kanaan G, Al-Nobani A (2012) Building an effective rule-based light stemmer for arabic language to improve search effectiveness. Int Arab J Inf Technol 9(4):368–372

    Google Scholar 

  • Abainia K, Ouamour S, Sayoud H (2017) A novel robust Arabic light stemmer. J Exp Theor Artif Intell 29(3):557–573

    Google Scholar 

  • Abu-Errub A, Odeh A, Shambour Q, Hassan OAH (2014) Arabic roots extraction using morphological analysis. Int J Comput Sci Issues (IJCSI) 11(2):128

    Google Scholar 

  • Ali M, Khalid S, Aslam MH (2018) Pattern-based comprehensive Urdu stemmer and short text classification. IEEE Access 6:7374–7389

    Google Scholar 

  • Ali M, Khalid S, Saleemi M (2019) Comprehensive stemmer for morphologically rich urdu language. Int Arab J Inf Technol 16(1):138–147

    Google Scholar 

  • Alotaibi FS, Gupta V (2018) A cognitive inspired unsupervised language-independent text stemmer for Information retrieval. Cognit Syst Res 52:291–300

    Google Scholar 

  • Al-Kabi MN, Kazakzeh SA, Ata BMA, Al-Rababah SA, Alsmadi IM (2015) A novel root based Arabic stemmer. J King Saud Univ-Comput Inf Sci 27(2):94–103

    Google Scholar 

  • Al-Omari A, Abuata B (2014) Arabic light stemmer (ARS). J Eng Sci Technol 9(6):702–717

    Google Scholar 

  • AlSerhan HM, Alqrainy S, Ayesh A (2008, November). Is paice method suitable for evaluating Arabic stemming algorithms? In: International conference on computer engineering & systems, 2008 (ICCES 2008). IEEE, pp 131–135

  • Al-Shammari ET, Lin J. (2008, October). Towards an error-free Arabic stemming. In Proceedings of the 2nd ACM workshop on Improving non English web searching. ACM, pp 9–16

  • Al-Sughaiyer IA, Al-Kharashi IA (2004) Arabic morphological analysis techniques: A comprehensive survey. J American Soc Inf Sci Tech 55(3):189–213

    Google Scholar 

  • Alvares RV, Garcia AC, Ferraz I (2005) December) STEMBR: a stemming algorithm for the Brazilian Portuguese language. Portuguese conference on artificial intelligence. Springer, Berlin, pp 693–701

    Google Scholar 

  • Aronoff M, Fudeman K (2011) What is morphology? vol. 8. Wiley, pp 2–3

  • Bimba A, Idris N, Khamis N, Noor NF (2016) Stemming Hausa text: using affix-stripping rules and reference look-up. Lang Resour Eval 50(3):687–703

    Google Scholar 

  • Bölücü, Necva and Burcu Can. (2019). Unsupervised Joint PoS Tagging and Stemming for Agglutinative Languages. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 18, 3, Article 25 (January 2019), 21 pages. https://doi.org/10.1145/3292398

  • Boudchiche M, Mazroui A (2015, December). Evaluation of the ambiguity caused by the absence of diacritical marks in Arabic texts: statistical study. In: 2015 5th international conference on information and communication technology and accessibility (ICTA). IEEE, pp 1–6

  • Boukhalfa I, Mostefai S, Chekkai N (2018, March) A study of graph based stemmer in Arabic extrinsic plagiarism detection. In: Proceedings of the 2nd mediterranean conference on pattern recognition and artificial intelligence. ACM, pp 27–32

  • Brychcín T, Konopík M (2015) HPS: high precision stemmer. Inf Process Manag 51(1):68–91

    Google Scholar 

  • Buckley C (1985) Implementation of the smart information retrieval system. Technical report 85–686, Cornell University.

  • Cambria E, White B (2014) Jumping NLP curves: a review of natural language processing research. IEEE Comput Intell Mag 9(2):48–57

    Google Scholar 

  • Chintala DR, Reddy EM (2013) An approach to enhance the CPI using Porter stemming algorithm. Int J Adv Res Comput Sci Softw Eng 3(7):1148–1156

    Google Scholar 

  • CISI Collection https://ir.dcs.gla.ac.uk/resources/test_collections/cisi/. Accessed 30 Dec 2019. Developed by University of Glasgow

  • Dahab MY, Ibrahim A, Al-Mutawa R (2015) A comparative study on Arabic stemmers. Int J Comput Appl 125(8):38–47

    Google Scholar 

  • Dang Q, Zhang J, Lu Y, Zhang K (2013) WordNet-based suffix tree clustering algorithm. In: International conference on information science and computer applications (ISCA 2013)

  • Dey A, Paul A, Purkayastha BS (2014) Named entity recognition for Nepali language: a semi hybrid approach. Int J Eng Innov Technol (IJEIT) 3:21–25

    Google Scholar 

  • Dianati MH, Sadreddini MH, Hossein RA, Fakhrahmad SM, Taghi-Zadeh H (2014) Words stemming based on structural and semantic similarity. Comp Eng Appl J 3(2):89–99

    Google Scholar 

  • de Oliveira RAN, Junior MC (2018) Experimental analysis of stemming on jurisprudential documents retrieval. Information 9(2):28

    Google Scholar 

  • Dukes K, Habash N (2010) Morphological annotation of Quranic Arabic. In Lrec, pp 2530–2536

  • El-Defrawy M, El-Sonbaty Y, Belal NA (2016) A rule-based subject-correlated Arabic stemmer. Arab J Sci Eng 41(8):2883–2891

    Google Scholar 

  • Fattah MA, Ren F, Kuroiwa S (2006) Stemming to improve translation lexicon creation form bitexts. Inf Process Manag 42(4):1003–1016

    Google Scholar 

  • Flores FN, Moreira VP (2016) Assessing the impact of stemming accuracy on information retrieval–a multilingual perspective. Inf Process Manag 52(5):840–854

    Google Scholar 

  • Frakes WB, Fox CJ (2003) Strength and similarity of affix removal stemming algorithms. In ACM SIGIR forum, vol 37, no 1. ACM, pp 26–30.

  • Gaidhane MS, Gondhale MD, Talole MP (2015) A comparative study of stemming algorithms for natural language processing. J Eng Educ Technol (ARDIJEET) 3(2):1–6

    Google Scholar 

  • Giachanou A, Crestani F (2016) Like it or not: a survey of twitter sentiment analysis methods. ACM Comput Surv (CSUR) 49(2):28

    Google Scholar 

  • Harman D (1991) How effective is suffixing. J Am Soc Inf Sci 42(1):7–15

    MathSciNet  Google Scholar 

  • Hassani K, Lee WS (2016) Visualizing natural language descriptions: a survey. ACM Comput Surv (CSUR) 49(1):17

    Google Scholar 

  • Husain MS, Ahamad F, Khalid S (2013) A language independent approach to develop Urdu stemmer. Advances in computing and information technology. Springer, Berlin, pp 45–53

    Google Scholar 

  • Hull DA (1996) Stemming algorithms—a case study for detailed evaluation. J Am Soc Inf Sci 47:70–84

    Google Scholar 

  • Hussain Z, Iqbal S, Saba T, Almazyad AS, Rehman A (2017) Design and development of dictionary-based stemmer for the urdu language. J Theor Appl Inf Technol 95(15):3560–3569

    Google Scholar 

  • Islam Md, Uddin Md, Khan M (2007) A light weight stemmer for Bengali and its use in spelling checker. Retrieved 24 March, 2019, from http://hdl.handle.net/10361/328

  • Ismailov A, Jalil MA, Abdullah Z, Rahim NA (2016) A comparative study of stemming algorithms for use with the Uzbek language. In: 3rd international conference on computer and information sciences (ICCOINS), 2016. IEEE, pp 7–12

  • Jaafar Y, Namly D, Bouzoubaa K, Yousfi A (2017) Enhancing Arabic stemming process using resources and benchmarking tools. J King Saud Univ-Comput Inf Sci 29(2):164–170

    Google Scholar 

  • Jabbar A, Iqbal S, Khan MUG (2016a) Analysis and development of resources for Urdu text stemming. In: Proceedings of the 6th annual international conference on language and technology, KICS-CLE, UET Lahore

  • Jabbar A, Iqbal S, Akhunzada A, Abbas Q (2018a) An improved Urdu stemming algorithm for text mining based on multi-step hybrid approach. J Exp Theor Artif Intell. https://doi.org/10.1080/0952813X.2018.1467495

    Article  Google Scholar 

  • Jabbar A, Iqbal S, Khan MUG, Hussain S (2018b) A survey on Urdu and Urdu like language stemmers and stemming techniques. Artif Intell Rev 49(3):339–373

    Google Scholar 

  • Jabbar A, Iqbal S, Khan MUG, Hussain S (2018b) A survey on Urdu and Urdu like language stemmers and stemming techniques. Artif Intell Rev 49(3):339–373

    Google Scholar 

  • Jivani AG (2011) A comparative study of stemming algorithms. Int J Comp Tech Appl 2(6):1930–1938

    Google Scholar 

  • Karaa WBA (2013) A new stemmer to improve information retrieval. Int J Netw Secur Appl 5(4):143

    MathSciNet  Google Scholar 

  • Karimi S, Wang C, Metke-Jimenez A, Gaire R, Paris C (2015) Text and data mining techniques in adverse drug reaction detection. ACM Comput Surv (CSUR) 47(4):56

    Google Scholar 

  • Kastner I (2019) Templatic morphology as an emergent property. Nat Lang Linguist Theory 37(2):571–619

    Google Scholar 

  • Khalid A, Hussain Z, Baig MA (2016) Arabic stemmer for search engines information retrieval. Int J Adv Comput Sci Appl 1(7):407–411

    Google Scholar 

  • Khan S, Waqas A, Usama B, Xuan W (2015) Template based affix stemmer for a morphologically rich language. Int Arab J Inf Tech 12(2):146–154

  • Khoja S, Garside R (1999) Stemming arabic text. Lancaster University, Lancaster, UK, Computing Department

    Google Scholar 

  • Krovetz R (2000) Viewing morphology as an inference process. Artif intel 118(1–2):277–294

    MATH  Google Scholar 

  • Larkey LS, Ballesteros L, Connell ME (2007) Light stemming for Arabic information retrieval. Arabic computational morphology. Springer, Dordrecht, pp 221–243

    Google Scholar 

  • Lemur (2016) https://www.lemurproject.org. Accessed 14 Aug 2018

  • Lennon M, Peirce DS, Tarry BD, Willett P (1981) An evaluation of some conflation algorithms for information retrieval. Inf Sci 3(4):177–183

    Google Scholar 

  • Lovins JB (1968) Development of a stemming algorithm. Mech Transl Comput Linguist 11(1–2):22–31

    Google Scholar 

  • Lucene (2018) https://lucene.apache.org. Accessed 12 Aug 2018

  • Mateen A, Malik MK, Nawaz Z, Danish HM, Siddiqui MH, Abbas Q (2017) A hybrid stemmer of punjabi shahmukhi script. Int J Comput Sci Netw Secur 17(8):90–97

    Google Scholar 

  • McCormick C (2016) Word2Vec tutorial—the skip-gram model. https://www.mccormickml.com

  • Mishra U, Prakash C (2012) MAULIK: an effective stemmer for Hindi language. Int J Comput Sci Eng 4(5):711–717

    Google Scholar 

  • Mochizuki M, Aizawa K (2000) An affix acquisition order for EFL learners: an exploratory study. System 28(2):291–304

    Google Scholar 

  • Moghadam FM, MohammadReza K (2015) Comparative study of various Persian stemmers in the field of information retrieval. J Inf Proc Syst 11(3):450–464

    Google Scholar 

  • Momenipour F, Keyvanpour MR (2016) PHMM: stemming on Persian texts using statistical stemmer based on hidden Markov Model. Int J Inf Sci Manag 14(2):107–117

  • Mustafa AM, Rashid TA (2018) Kurdish stemmer pre-processing steps for improving information retrieval. J Inf Sci 44(1):15–27

    Google Scholar 

  • Nguyen, (2013) Nguyen DT, Leveling J (2013) Exploring domain-sensitive features for extractive summarization in the medical domain. International conference on application of natural language to information systems. Springer, Berlin, pp 90–101

    Google Scholar 

  • Nwesri AFA, Alyagoubi HAH (2015). Applying arabic stemming using query expansion. In 2015 26th international workshop on database and expert systems applications (DEXA) (pp. 299–303). IEEE

  • Orengo VM, Huyck C (2001) a stemming algorithm for the portuguese language. In; SPIRE '01: Proceedings of eigth symposium on string processing and information retrieval, pp 186–193.

  • Paice CD (1990) Another stemmer. SIGIR Forum 24(3):56–61

    Google Scholar 

  • Paice CD (1996) Method for evaluation of stemming algorithms based on error counting. J Am Soc Inf Sci 47(8):632–649

    Google Scholar 

  • Paice CD (1994) An evaluation method for stemming algorithms. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. Springer, New York, pp 42–50

  • Pande BP, Tamta P, Dhami HS (2018) Generation, implementation and appraisal of an N-gram based stemming algorithm. Digit Scholarsh Humanit. https://doi.org/10.1093/llc/fqy053

    Article  Google Scholar 

  • Paik JH, Pal D, Parui SK (2011) A novel corpus-based stemming algorithm using co-occurrence statistics. In: Proceedings of the 34th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR’11). ACM, New York, pp 863–872

  • Patil CG, Patil SS (2013) Use of Porter stemming algorithm and SVM for emotion extraction from news headlines. Int J Electron Commun Soft Comput Sci Eng 2(7):9–13

    Google Scholar 

  • Porter MF (2006) https://snowball.artarus.org/algorithms/english/ stemmer.html

  • Porter MF (1980) An algorithm for suffix stripping. Program 14(3):130–137

    Google Scholar 

  • Qureshi AH, Hassan MU, Akhter S (2018) Towards description of derivation in Urdu: morphological perspective. Al-Qalam 23(2):96–100

    Google Scholar 

  • Rani SPR, Ramesh B, Anusha M, Rani SJGR (2015) Evaluation of stemming techniques for text classification. Int J Comput Sci Mobile Comput 4(3):165–171

    Google Scholar 

  • Rashid TA, Mohamad SO (2016) Enhancement of detecting wicked website through intelligent methods. International symposium on security in computing and communication. Springer, Singapore, pp 358–368

    Google Scholar 

  • Rashidi A, Lighvan MZ (2014) HPS: a hierarchical Persian stemming method. arXiv preprint arXiv:1403.2837.

  • Rehman Z, Anwar W, Bajwa UI, Xuan W, Chaoying Z (2013) Morpheme matching based text tokenization for a scarce resourced language. PLoS ONE 8(8):e68178

    Google Scholar 

  • Saad MK, Ashour W (2010) Arabic morphological tools for text mining. Corpora 18:19

    Google Scholar 

  • Saeed AM, Rashid TA, Mustafa AM, Al-Rashid Agha RA, Shamsaldin AS, Al-Salihi NK (2018a) An evaluation of Reber stemmer with longest match stemmer technique in Kurdish Sorani text classification. Iran J Comput Sci 1(2):99–107

    Google Scholar 

  • Saeed AM, Rashid TA, Mustafa AM, Fattah P, Ismael B (2018b) Improving Kurdish web mining through tree data structure and Porter’s Stemmer algorithms. UKH J Sci Eng 2(1):48–54

    Google Scholar 

  • Sarma B, Purkayastha BS (2013) An affix based word classification method of assamese text. Int J Adv Res Comput Sci 4(9):213–216

    Google Scholar 

  • Schofield A, Mimno D (2016) Comparing apples to apple: the effects of stemmers on topic models. Trans Assoc Comput Linguist 4:287–300

    Google Scholar 

  • Setiawan R, Kurniawan A, Budiharto W, Kartowisastro IH, Prabowo H (2016) Flexible affix classification for stemming Indonesian Language. In: 2016 13th international conference on electrical engineering/electronics, computer, telecommunications and information technology (ECTI-CON). IEEE, pp 1–6

  • Singh J, Gupta V (2016) Text stemming: approaches, applications, and challenges. ACM Comput Surv (CSUR) 49(3):45

    Google Scholar 

  • Singh J, Gupta V (2017) An efficient corpus-based stemmer. Cognit Comput 9(5):671–688

    Google Scholar 

  • Sirsat SR, Chavan V, Mahalle HS (2013) Strength and accuracy analysis of affix removal stemming algorithms. Int J Comput Sci Inf Technol 4(2):265–269

    Google Scholar 

  • Sulaiman S, Omar K, Omar N, Murah MZ, Abdul Rahman HD (2014) The effectiveness of a Jawi stemmer for retrieving relevant Malay documents in Jawi characters. ACM Trans Asian Lang Inf Process (TALIP) 13(2):6

    Google Scholar 

  • Suryani AA, Widyantoro DW, Purwarianti A, Sudaryat Y (2018) The rule-based sundanese stemmer. ACM Trans Asian Low-Resour Lang Inf Process (TALLIP) 17(4):27

    Google Scholar 

  • Taghi-Zadeh H, Sadreddini MH, Diyanati MH, Rasekh AH (2015) A new hybrid stemming method for persian language. Digit Scholarsh Humanit 32(1):209–221

    Google Scholar 

  • Thangarasu M, Manavalan R (2013) Design and development of stemmer for Tamil language: cluster analysis. Int J Adv Res Comput Sci Softw Eng 3(7):812–818

    Google Scholar 

  • The free dictionary (2018) https://www.thefreedictionary.com/. Accessed 03 Aug 2018

  • Qunis I, Amati G, Plachouras V, He B, Macdonald C, Lioma C (2006) A high performance and scalable information retrieval plateform. In: SIGR workshop on open source information retrieval

  • Urdu L (2006) https://182.180.102.251:8081/oud/help_3.htm. Accessed 04 Aug 2018

  • Xapian (2018) https://xapian.org. Accessed 07 Aug 2018

  • Xer (1994) Xeror linguistic database reference, English version 1.1.4 ed.s

  • Yadollahi A, Shahraki AG, Zaiane OR (2017) Current state of text sentiment analysis from opinion to emotion mining. ACM Comput Surv (CSUR) 50(2):25

    Google Scholar 

  • Zerrouki T (2016) Tashaphyne 0.2 (Online). https://pypi.python.org/pypi/Tashaphyne. Accessed 14 Apr 2016

  • Zhou D, Mark T, Brailsford T, Wade V, Ashman H (2012) Translation techniques in cross-language information retrieval. ACM Comput Surv (CSUR) 45(1):1

    Google Scholar 

Download references

Funding

Funding was provided by Bahauddin Zakariya University (PK) (Grant No: 2019-05).

Author information

Authors and Affiliations

  1. Department of Computer Science, COMSATS University Islamabad (CUI), Main campus, Park Road, Tarlai Kalan, Islamabad, 45550, Pakistan

    Abdul Jabbar, Manzoor Ilahi Tamimy & Adnan Akhunzada

  2. Department of Computer Science, Bahauddin Zakariya University Multan, Multan, Punjab, Pakistan

    Sajid Iqbal

  3. Bahauddin Zakariya University Multan (Sahiwal Sub-Campus), Multan, Punjab, Pakistan

    Shafiq Hussain

Authors
  1. Abdul Jabbar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sajid Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manzoor Ilahi Tamimy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shafiq Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Adnan Akhunzada
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sajid Iqbal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jabbar, A., Iqbal, S., Tamimy, M.I. et al. Empirical evaluation and study of text stemming algorithms. Artif Intell Rev 53, 5559–5588 (2020). https://doi.org/10.1007/s10462-020-09828-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-020-09828-3

Keywords

Search

Navigation