Abstract
Indonesian is classified as underrepresented in the Natural Language Processing (NLP) field, despite being the tenth most spoken language in the world with 198 million speakers. The paucity of datasets is recognized as the main reason for the slow advancements in NLP research for underrepresented languages. Significant attempts were made in 2020 to address this drawback for Indonesian. The Indonesian Natural Language Understanding (IndoNLU) benchmark was introduced alongside IndoBERT pre-trained language model. The second benchmark, Indonesian Language Evaluation Montage (IndoLEM), was presented in the same year. These benchmarks support several tasks, including Named Entity Recognition (NER). However, all NER datasets are in the public domain and do not contain domain-specific datasets. To alleviate this drawback, we introduce IndQNER, a manually annotated NER benchmark dataset in the religious domain that adheres to a meticulously designed annotation guideline. Since Indonesia has the world’s largest Muslim population, we build the dataset from the Indonesian translation of the Quran. The dataset includes 2475 named entities representing 18 different classes. To assess the annotation quality of IndQNER, we perform experiments with BiLSTM and CRF-based NER, as well as IndoBERT fine-tuning. The results reveal that the first model outperforms the second model achieving 0.98 F1 points. This outcome indicates that IndQNER may be an acceptable evaluation metric for Indonesian NER tasks in the aforementioned domain, widening the research’s domain range.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
All English translations are the sahih international version from https://corpus.quran.com/translation.jsp.
- 9.
- 10.
- 11.
We used the Asmaul Husna reference that can be seen at https://github.com/dice-group/IndQNER/blob/main/Asmaul_Husna_Reference.pdf.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
References
Aji, A.F., et al.: One Country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 7226–7249. Association for Computational Linguistics (2022). https://doi.org/10.18653/v1/2022.acl-long.500, https://aclanthology.org/2022.acl-long.500
Alfina, I., Manurung, R., Fanany, M.I.: DBpedia entities expansion in automatically building dataset for Indonesian NER. 2016 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2016, pp. 335–340 (2017). https://doi.org/10.1109/ICACSIS.2016.7872784
Khairunnisa, S.O., Imankulova, A., Komachi, M.: Towards a standardized dataset on indonesian named entity recognition. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pp. 64–71. Association for Computational Linguistics (2020). https://www.tempo.co/, https://www.aclweb.org/anthology/2020.aacl-srw.10
Koto, F., Rahimi, A., Lau, J.H., Baldwin, T.: IndoLEM and IndoBERT: a benchmark dataset and pre-trained language model for indonesian NLP. In: Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 757–770. International Committee on Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.coling-main.66, https://www.aclweb.org/anthology/2020.coling-main.66
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, pp. 260–270. Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/N16-1030, https://aclanthology.org/N16-1030
Luthfi, A., Distiawan, B., Manurung, R.: Building an Indonesian named entity recognizer using Wikipedia and DBPedia. In: Proceedings of the International Conference on Asian Language Processing 2014, IALP 2014, pp. 19–22 (2014). https://doi.org/10.1109/IALP.2014.6973520
Martinez-Rodriguez, J.L., Hogan, A., Lopez-Arevalo, I.: Information extraction meets the semantic web: a survey (2020). https://doi.org/10.3233/SW-180333, http://prefix.cc
Syaifudin, Y., Nurwidyantoro, A.: Quotations identification from Indonesian online news using rule-based method. In: Proceeding - 2016 International Seminar on Intelligent Technology and Its Application, ISITIA 2016: Recent Trends in Intelligent Computational Technologies for Sustainable Energy, pp. 187–194 (2017). https://doi.org/10.1109/ISITIA.2016.7828656
Wilie, B., et al.: IndoNLU: benchmark and resources for evaluating Indonesian natural language understanding (2020). http://arxiv.org/abs/2009.05387
Acknowledgements
We acknowledge the support of the German Federal Ministry for Economic Affairs and Climate Action (BMWK) within the project SPEAKER (01MK20011U), the German Federal Ministry of Education and Research (BMBF) within the project KIAM (02L19C115) and the EuroStars project PORQUE (01QE2056C), and Mora Scholarship from the Ministry of Religious Affairs, Republic of Indonesia. Furthermore, we would like to thank our amazing annotators, including Anggita Maharani Gumay Putri, Muhammad Destamal Junas, Naufaldi Hafidhigbal, Nur Kholis Azzam Ubaidillah, Puspitasari, Septiany Nur Anggita, Wilda Nurjannah, and William Santoso. We also thank Khodijah Hulliyah, Lilik Ummi Kultsum, Jauhar Azizy, and Eva Nugraha for the valuable feedback.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gusmita, R.H., Firmansyah, A.F., Moussallem, D., Ngonga Ngomo, AC. (2023). IndQNER: Named Entity Recognition Benchmark Dataset from the Indonesian Translation of the Quran. In: Métais, E., Meziane, F., Sugumaran, V., Manning, W., Reiff-Marganiec, S. (eds) Natural Language Processing and Information Systems. NLDB 2023. Lecture Notes in Computer Science, vol 13913. Springer, Cham. https://doi.org/10.1007/978-3-031-35320-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-35320-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35319-2
Online ISBN: 978-3-031-35320-8
eBook Packages: Computer ScienceComputer Science (R0)