Abstract
Recent work based on Deep Learning presents state-of-the-art (SOTA) performance in the named entity recognition (NER) task. However, such models still have the performance drastically reduced in noisy data (e.g., social media, search engines), when compared to the formal domain (e.g., newswire). Thus, designing and exploring new methods and architectures is highly necessary to overcome current challenges. In this paper, we shift the focus of existing solutions to an entirely different perspective. We investigate the potential of embedding word-level features extracted from images and news. We performed a very comprehensive study in order to validate the hypothesis that images and news (obtained from an external source) may boost the task on noisy data, revealing very interesting findings. When our proposed architecture is used: (1) We beat SOTA in precision with simple CRFs models (2) The overall performance of decision trees-based models can be drastically improved. (3) Our approach overcomes off-the-shelf models for this task. (4) Images and text consistently increased recall over different datasets for SOTA, but at cost of precision. All experiment configurations, data and models are publicly available to the research community at horus-ner.org
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
For the sake of fair comparison, 3-MUC is also the base for experiments.
- 3.
41 experiment configurations, 4 training sets (Ritter, WNUT-15, WNUT-16 and WNUT-17), 3 test sets (WNUT-15, WNUT-16 and WNUT-17) and 9 NER architectures (DT, RF, CRF, CRF-PA, LSTM, B-LSTM+CRF, Char+B-LSTM+CRF and B-LSTM+CNN+CRF).
- 4.
3-fold cross-validation.
- 5.
\(\mathcal {B}_{best}\), cfg30-41.
- 6.
References
Baldwin, T., de Marneffe, M.C., Han, B., Kim, Y.B., Ritter, A., Xu, W.: Shared tasks of the 2015 workshop on noisy user-generated text: twitter lexical normalization and named entity recognition. In: Proceedings of the Workshop on Noisy User-generated Text, pp. 126–135 (2015)
Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3), 346–359 (2008). https://doi.org/10.1016/j.cviu.2007.09.014
Bontcheva, K., Derczynski, L., Funk, A., Greenwood, M.A., Maynard, D., Aswani, N.: Twitie: an open-source information extraction pipeline for microblog text. In: RANLP, pp. 83–90 (2013)
Brown, P.F., Desouza, P.V., Mercer, R.L., Pietra, V.J.D., Lai, J.C.: Class-based n-gram models of natural language. Comput. Linguist. 18(4), 467–479 (1992)
Chang, Y.S., Sung, Y.H.: Applying name entity recognition to informal text. Recall 1(1) (2005)
Cunningham, H., et al.: Text Processing with GATE (Version 6). University of Sheffield Department of Computer Science 15 (2011)
Derczynski, L., Bontcheva, K.: Passive-aggressive sequence labeling with discriminative post-editing for recognising person entities in tweets. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pp. 69–73 (2014)
Derczynski, L., Chester, S., Bøgh, K.S.: Tune your brown clustering, please. In: International Conference Recent Advances in Natural Language Processing, RANLP. Association for Computational Linguistics, vol. 2015, pp. 110–117 (2015)
Derczynski, L., et al.: Analysis of named entity recognition and linking for tweets. Inf. Process. Manage. 51(2), 32–49 (2015)
Derczynski, L., Nichols, E., van Erp, M., Limsopatham, N.: Results of the wnut2017 shared task on novel and emerging entity recognition. In: Proceedings of the 3rd Workshop on Noisy User-generated Text. Association for Computational Linguistics, pp. 140–147 (2017). https://doi.org/10.18653/v1/W17-4418
Derczynski, L., Nichols, E., van Erp, M., Limsopatham, N.: Results of the wnut2017 shared task on novel and emerging entity recognition. In: Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140–147 (2017)
Esteves, D., Peres, R., Lehmann, J., Napolitano, G.: Named entity recognition in twitter using images and text. In: Garrigós, I., Wimmer, M. (eds.) ICWE 2017. LNCS, vol. 10544, pp. 191–199. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-74433-9_17
Esteves, D., Reddy, A.J., Chawla, P., Lehmann, J.: Belittling the source: trustworthiness indicators to obfuscate fake news on the web. In: Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pp. 50–59 (2018)
Gattani, A., et al.: Entity extraction, linking, classification, and tagging for social media: a wikipedia-based approach. Proc. VLDB Endow. 6(11), 1126–1137 (2013). https://doi.org/10.14778/2536222.2536237
He, H., Sun, X.: A unified model for cross-domain and semi-supervised named entity recognition in Chinese social media. In: AAAI, pp. 3216–3222 (2017)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 1746–1751 (2014). https://doi.org/10.3115/v1/D14-1181
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 (2016)
Limsopatham, N., Collier, N.: Bidirectional LSTM for named entity recognition in twitter messages. WNUT 2016, 145 (2016)
Liu, X., Zhang, S., Wei, F., Zhou, M.: Recognizing named entities in tweets. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Vol. 1, pp. 359–367 (2011)
Liu, X., Zhou, M., Wei, F., Fu, Z., Zhou, X.: Joint inference of named entity recognition and normalization for tweets. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Vol. 1, pp. 526–535. Association for Computational Linguistics (2012)
Lopez, C., et al.: Cap 2017 challenge: twitter named entity recognition. arXiv preprint arXiv:1707.07568 (2017)
Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international Conference on Computer Vision, Vol. 2, pp. 1150–1157. IEEE (1999)
Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF. arXiv preprint arXiv:1603.01354 (2016)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Moussallem, D., Usbeck, R., Röder, M., Ngonga Ngomo, A.C.: MAG: a multilingual, knowledge-base agnostic and deterministic entity linking approach. In: Proceedings of the Knowledge Capture Conference, K-CAP 2017, p. 8. ACM (2017)
Peres, R., Esteves, D., Maheshwari, G.: Bidirectional LSTM with a context input window for named entity recognition in tweets. In: Proceedings of the Knowledge Capture Conference, p. 42. ACM (2017)
Ritter, A., et al.: Named entity recognition in tweets: an experimental study. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, pp. 1524–1534 (2011)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Strauss, B., Toma, B., Ritter, A., de Marneffe, M.C., Xu, W.: Results of the wnut16 named entity recognition shared task. In: Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pp. 138–144 (2016)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and pattern Recognition, CVPR, pp. 1–9 (2015)
Szegedy, C., et al.: Going deeper with convolutions. In: CVPR, pp. 1–9. IEEE (2015)
Tkachenko, M., Simanovsky, A.: Named entity recognition: exploring features. In: KONVENS, pp. 118–127 (2012)
Wallach, H.M.: Topic modeling: beyond bag-of-words. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 977–984. ACM (2006)
Zhang, Q., Fu, J., Liu, X., Huang, X.: Adaptive co-attention network for named entity recognition in tweets. In: AAAI (2018)
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, pp. 649–657 (2015)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Esteves, D., Marcelino, J., Chawla, P., Fischer, A., Lehmann, J. (2021). HORUS-NER: A Multimodal Named Entity Recognition Framework for Noisy Data. In: Abreu, P.H., Rodrigues, P.P., Fernández, A., Gama, J. (eds) Advances in Intelligent Data Analysis XIX. IDA 2021. Lecture Notes in Computer Science(), vol 12695. Springer, Cham. https://doi.org/10.1007/978-3-030-74251-5_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-74251-5_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74250-8
Online ISBN: 978-3-030-74251-5
eBook Packages: Computer ScienceComputer Science (R0)