Abstract
The paper describes a new Text Preprocessing Pipeline based on a Hybrid approach which involve rule-based and stochastic approaches. The presented pipeline is part of a larger project titled Big Data for Multi-Agent Specialized System developed by Network Contacts in collaboration with University of Salerno and other institutional partners. The aim of the project is to build an Hybrid Question Answering System composed by sets of Dialog Bots able to process great volumes of data. Due to the importance of unstructured textual data, a particular focus of the project is on automatic processing of Text. The paper will describe the three main modules of the preprocessing pipeline, which involve a Style Correction Module, a Clitic Decomposition Module and a POS Tagging and Lemmatization Module.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Network Contacts is one of the major national player in BPO services (Business Process Outsourcing), CRM (Customer Relationship Management), Digital Interaction and Call & Contact Center.
- 2.
With MWE we make reference both to lemmatized MWE, e.g. carta di credito, which are listed in the Knowledge base, and to a set of MWE which are domain specific, e.g. all inclusive unlimited, for the domain of telecommunications.
References
Amato, F., Colace, F., Greco, L., Moscato, V., Picariello, A.: Semantic processing of multimedia data for e-government applications. J. Vis. Lang. Comput. 32, 35–41 (2016). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84949036166&doi=10.1016%2fj.jvlc.2015.10.012&partnerID=40&md5=77d8fbfd4293af91d382e770553e159c.
Amato, F., Mazzeo, A., Penta, A., Picariello, A.: Knowledge representation and management for e-government documents. In: IFIP International Federation for Information Processing, vol. 280, pp. 31–40 (2008). https://www.scopus.com/inward/record.uri?eid=2-s2.0-47249122655&doi=10.1007%2f978-0-387-09712-1_4&partnerID=40&md5=d8e43aeb328d1fdcb45aa323689f7010.
Amato, F., Moscato, F.: A model driven approach to data privacy verification in e-health systems. Trans. Data Privacy 8(3), 273–296 (2015). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84949775424&partnerID=40&md5=2ca02581d8482fbc2f8489a423a80290.
Attardi, G., Fuschetto, A., Tamberi, F., Simi, M., Vecchi, E.M.: Experiments in tagger combination: arbitrating, guessing, correcting, suggesting. In: Proceedings of Workshop Evalita, p. 10 (2009)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Choi, J.D.: Dynamic feature induction: the last gist to the state-of-the-art. In: Proceedings of NAACL-HLT, pp. 271–281 (2016)
Chrupała, G., Dinu, G., Van Genabith, J.: Learning morphology with Morfette (2008)
Collins, M.: Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, vol. 10, pp. 1–8. Association for Computational Linguistics (2002)
Constant, M., Tellier, I., Duchier, D., Dupont, Y., Sigogne, A., Billot, S.: Intégrer des connaissances linguistiques dans un crf: application à l’apprentissage d’un segmenteur-étiqueteur du français. In: TALN, vol. 1, p. 321 (2011)
Dahlmeier, D., Ng, H.T.: Better evaluation for grammatical error correction. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 568–572. Association for Computational Linguistics (2012)
Dell’Orletta, F.: Ensemble system for part-of-speech tagging. In: Proceedings of EVALITA, vol. 9, pp. 1–8 (2009)
Delmonte, R.: Generating and parsing clitics with getarun. In: CLIN (1999)
Favretti, R.R., Tamburini, F., De Santis, C.: CORIS/CODIS: a corpus of written Italian based on a defined and a dynamic model. A Rainbow of Corpora: Corpus Linguistics and the Languages of the World. Munich: Lincom-Europa (2002)
Felice, M., Yuan, Z., Andersen, Ø.E., Yannakoudakis, H., Kochmar, E.: Grammatical error correction using hybrid systems and type filtering. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 15–24 (2014)
Giménez, J., Marquez, L.: Svmtool: a general POS tagger generator based on support vector machines. In: Proceedings of the 4th International Conference on Language Resources and Evaluation. Citeseer (2004)
Han, N.R., Chodorow, M., Leacock, C.: Detecting errors in English article usage with a maximum entropy classifier trained on a large, diverse corpus. In: LREC (2004)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
Ingason, A.K., Helgadóttir, S., Loftsson, H., Rögnvaldsson, E.: A mixed method lemmatization algorithm using a hierarchy of linguistic identities (HOLI). In: Advances in Natural Language Processing, pp. 205–216. Springer, Heidelberg (2008)
Junczys-Dowmunt, M., Grundkiewicz, R.: The AMU system in the CoNLL-2014 shared task: grammatical error correction by data-intensive and feature-rich statistical machine translation. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 25–33 (2014)
Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys. doklady 10, 707–710 (1966)
Lyding, V., Stemle, E., Borghetti, C., Brunello, M., Castagnoli, S., Dell’Orletta, F., Dittmann, H., Lenci, A., Pirrelli, V.: The PAISA corpus of Italian web texts. In: Proceedings of the 9th Web as Corpus Workshop (WaC-9), pp. 36–43 (2014)
Manning, C.D.: Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In: International Conference on Intelligent Text Processing and Computational Linguistics, pp. 171–189. Springer, Heidelberg (2011)
Mizumoto, T., Hayashibe, Y., Komachi, M., Nagata, M., Matsumoto, Y.: The effect of learner corpus size in grammatical error correction of ESL writings. In: Proceedings of COLING 2012: Posters, pp. 863–872 (2012)
Monachesi, P.: A grammar of Italian clitics (1996)
Monachesi, P.: A lexical approach to Italian cliticization (1999)
Palmero Aprosio, A., Moretti, G.: Italy goes to Stanford: a collection of CoreNLP modules for Italian. ArXiv e-prints (2016)
Pianta, E., Zanoli, R.: TagPro: a system for Italian PoS tagging based on SVM. Intelligenza Artificiale 4(2), 8–9 (2007)
Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)
Rozovskaya, A., Roth, D.: Algorithm selection and model adaptation for ESL correction tasks. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 924–933. Association for Computational Linguistics (2011)
Russi, C.: Italian clitics: an empirical study, vol. 193. Walter de Gruyter (2008)
Schmid, H.: Treetagger—a language independent part-of-speech tagger. Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart 43, 28 (1995)
Shen, L., Satta, G., Joshi, A.: Guided learning for bidirectional sequence classification. In: ACL, vol. 7, pp. 760–767. Citeseer (2007)
Smedt, T.D., Daelemans, W.: Pattern for python. J. Mach. Learn. Res. 13(Jun), 2063–2067 (2012)
Srinivasan, A., Compton, P., Malor, R., Edwards, G., Lazarus, L.: Knowledge acquisition in context for a complex domain. Pre-print of Proceedings of the Fifth EKAW91 (1991)
Sun, C., Jin, X., Lin, L., Zhao, Y., Wang, X.: Convolutional neural networks for correcting English article errors. In: Natural Language Processing and Chinese Computing, pp. 102–110. Springer, Heidelberg (2015)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)
Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 173–180. Association for Computational Linguistics (2003)
Vietri, S.: Dizionari elettronici e grammatiche a stati finiti: metodi di analisi formale della lingua italiana. Plectica (2008)
Vietri, S.: The Italian module for NooJ. In: Proceedings of the First Italian Conference on Computational Linguistics, CLiC-it, pp. 389–393 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Maisto, A., Pelosi, S., Polito, M., Stingo, M. (2019). Automatic Text Preprocessing for Intelligent Dialog Agents. In: Barolli, L., Takizawa, M., Xhafa, F., Enokido, T. (eds) Web, Artificial Intelligence and Network Applications. WAINA 2019. Advances in Intelligent Systems and Computing, vol 927. Springer, Cham. https://doi.org/10.1007/978-3-030-15035-8_78
Download citation
DOI: https://doi.org/10.1007/978-3-030-15035-8_78
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-15034-1
Online ISBN: 978-3-030-15035-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)