short-paper

Sentence boundary disambiguation for Indonesian language

Authors:

Syopiansyah Jaya Putra,

Muhamad Nur Gunawan,

Teddy MantoroAuthors Info & Claims

iiWAS '17: Proceedings of the 19th International Conference on Information Integration and Web-based Applications & Services

Pages 587 - 590

https://doi.org/10.1145/3151759.3156474

Published: 04 December 2017 Publication History

Abstract

Sentence boundary detection is essential for natural language processing (NLP). Sentence boundary detection in the Indonesian language has lots of problems, which includes punctuation, abbreviation, and character in the bracket. The disambiguation should be detected as sentence boundary. Thus the sentence boundary system can divide the sentences accurately. This study presents the development of a training dataset for the existing model to optimize supervised sentence boundary detection for the Indonesian language. Indonesian Translation of the Quran (ITQ) data set was used in this study by using the supervised method. The following is the process briefly: create the training data, apply sentence detection to separate sentences on ITQ, and calculate precision, recall, and F-measure. The result is quite promising, it gives as follows: Precision of 91.7%, Recall 81.6%, and F-Measure 86.4 %, respectively.

References

[1]

M. Parakh, N. Rajesha, and M. Ramya, "Sentence Boundary Disambiguation in Kannada Texts," Language in India, www.languageinindia.com, Special Volume: Problems of Parsing in Indian Languages, pp. 17--19, 2011.

[2]

N. Wanjari, G. Dhopavkar, and N. B. Zungre, "Sentence Boundary Detection for Marathi Language," Procedia Computer Science, vol. 78, pp. 550--555, 2016.

Digital Library

[3]

B. Jurish and K.-M. Würzner, "Word and Sentence Tokenization with Hidden Markov Models," JLCL, vol. 28, pp. 61--83, 2013.

[4]

J. Read, R. Dridan, S. Oepen, and L. J. Solberg, "Sentence boundary detection: A long solved problem?," COLING (Posters), vol. 12, pp. 985--994, 2012.

[5]

H. P. Le and T. V. Ho, "A maximum entropy approach to sentence boundary detection of Vietnamese texts," in IEEE International Conference on Research, Innovation and Vision for the Future-RIVF 2008, 2008.

[6]

F. Šarić, J. Šnajder, and B. Dalbelo Bašić, "Optimizing Sentence Boundary Detection for Croatian," in Text, Speech and Dialogue, 2012, pp. 105--111.

[7]

P. P. Mazur, "Text segmentation in Polish," in Intelligent Systems Design and Applications, 2005. ISDA'05. Proceedings. 5th International Conference on, 2005, pp. 43--48.

Digital Library

[8]

Y. Yang and N. Xue, "Chinese comma disambiguation for discourse analysis," in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, 2012, pp. 786--794.

Digital Library

[9]

N. Xue and Y. Yang, "Chinese sentence segmentation as comma classification," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, 2011, pp. 631--635.

Digital Library

[10]

T. Kiss and J. Strunk, "Unsupervised multilingual sentence boundary detection," Computational Linguistics, vol. 32, pp. 485--525, 2006.

Digital Library

[11]

J. C. Reynar and A. Ratnaparkhi, "A maximum entropy approach to identifying sentence boundaries," in Proceedings of the fifth conference on Applied natural language processing, 1997, pp. 16--19.

Digital Library

[12]

D. Rusu, L. Dali, B. Fortuna, M. Grobelnik, and D. Mladenic, "Triplet extraction from sentences," in Proceedings of the 10th International Multiconference" Information Society-IS, 2007, pp. 8--12.

[13]

M. Collins, "Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms," in Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, 2002, pp. 1--8.

Digital Library

[14]

A. Gruber, Y. Weiss, and M. Rosen-Zvi, "Hidden topic Markov models," in Artificial intelligence and statistics, 2007, pp. 163--170.

[15]

A. Ratnaparkhi, "A simple introduction to maximum entropy models for natural language processing," IRCS Technical Reports Series, p. 81, 1997.

[16]

J. Huang and G. Zweig, "Maximum entropy model for punctuation annotation from speech," in INTERSPEECH, 2002.

[17]

A. Ratnaparkhi, "Maximum entropy models for natural language ambiguity resolution," A Dissertation, University of Pennsylvania, 1998.

Digital Library

[18]

N. Agarwal, K. H. Ford, and M. Shneider, "Sentence boundary detection using a maxEnt classifier," in Proceedings of MISC, 2005, pp. 1--6.

[19]

K. Tomanek, J. Wermter, and U. Hahn, "Sentence and token splitting based on conditional random fields," in Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics, 2007, pp. 49--57.

[20]

D. J. Walker, D. E. Clements, M. Darwin, and J. W. Amtrup, "Sentence boundary detection: A comparison of paradigms for improving MT quality," in Proceedings of the MT Summit VIII, 2001.

Cited By

Putra SGunawan MSuryatno A(2018)Tokenization and N-Gram for Indexing Indonesian Translation of the Quran2018 6th International Conference on Information and Communication Technology (ICoICT)10.1109/ICoICT.2018.8528762(158-161)Online publication date: May-2018
https://doi.org/10.1109/ICoICT.2018.8528762

Index Terms

Sentence boundary disambiguation for Indonesian language
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Information extraction

Recommendations

Sentence Boundary Detection For Marathi Language

Detecting the sentence boundary forms the basic step for many natural language applications. A lot of work has been done in this direction for English and other foreign languages. But not much work has been done for Indian languages. This paper proposes ...
Adaptive multilingual sentence boundary disambiguation

The sentence is a standard textual unit in natual language processing applications. In many language the punctuation mark that indicates the end-of-sentence boundary is ambiguous; thus the tokenizers of most NLP systems must be equipped with special ...
Graph-based word sense disambiguation in Telugu language

In Natural Language Processing, word sense disambiguation (WSD) is an open challenge which improves the performance of the applications such as machine translation and information retrieval system. Many verbal languages will have many ambiguous words. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

iiWAS '17: Proceedings of the 19th International Conference on Information Integration and Web-based Applications & Services

December 2017

609 pages

ISBN:9781450352994

DOI:10.1145/3151759

Editors:
Maria Indrawan-Santiago
Monash University, Australia
,
Matthias Steinbauer
Johannes Kepler University Linz, Austria
,
Ivan Luiz Salvadori
Federal University of Santa Catarina, Brazil
,
Ismail Khalil
Johannes Kepler University Linz, Austria
,
Gabriele Anderst-Kotsis
Johannes Kepler University Linz, Austria

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

iiWAS2017

iiWAS2017: The 19th International Conference on Information Integration and Web-based Applications & Services

December 4 - 6, 2017

Salzburg, Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
72
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Putra SGunawan MSuryatno A(2018)Tokenization and N-Gram for Indexing Indonesian Translation of the Quran2018 6th International Conference on Information and Communication Technology (ICoICT)10.1109/ICoICT.2018.8528762(158-161)Online publication date: May-2018
https://doi.org/10.1109/ICoICT.2018.8528762

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten