Skip to main content
SpringerLink
Log in

Improved Identification of Imbalanced Multiple Annotation Intent Labels with a Hybrid BLSTM and CNN Model and Hybrid Loss Function

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track (ECML PKDD 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12461))

Abstract

Payment or fund transfer transactions can be annotated by users when they are made through a mobile banking app, for example, SCB Easy app—a mobile banking app by Siam Commercial Bank—allows users to annotate transactions with 40 character texts. The AI\(^2\) framework was used to identify user intentions with the transactions, so that the bank can offer the right product to the right customer at the right time. The framework employed Long Short-Term Memory (LSTM). Commonly, one annotated sample can be interpreted as representing multiple intents, thus we had a multiple label classification problem. However, the original model did not consider the class imbalance, that caused the model to bias toward the majority class. We introduced a new hybrid Bidirectional LSTM and Convolutional Neural Network model in conjunction with a new hybrid loss function to tackle the imbalance. Our model with hybrid loss function performed better than the AI\(^2\) framework with a 4.5% improvement in \(F_1\)-score. Moreover, our hybrid loss function enabled the model to classify minority classes better, when the imbalance ratio became higher, compared with a conventional cross-entropy loss function. In other words, our hybrid loss function made the model to be more efficient in real-world multiple label imbalance problem.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017). https://doi.org/10.1162/tacl_a_00051

    Article  Google Scholar 

  2. Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249–259 (2018). https://doi.org/10.1016/j.neunet.2018.07.011

    Article  Google Scholar 

  3. Chumwatana, T.: Using sentiment analysis technique for analyzing Thai customer satisfaction from social media. In: Proceedings of the 5th International Conference on Computing and Informatics (ICOCI 2015), 11–13 August 2015, Istanbul, Turkey (2015)

    Google Scholar 

  4. Crum, W.R., Camara, O., Hill, D.L.: Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans. Med. Imaging 25(11), 1451–1461 (2006). https://doi.org/10.1109/TMI.2006.880587

    Article  Google Scholar 

  5. Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)

    Article  Google Scholar 

  6. Elman, J.L.: Finding structure in time. Cognitive Sci. 14(2), 179–211 (1990). https://doi.org/10.1207/s15516709cog1402_1

    Article  Google Scholar 

  7. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., Mikolov, T.: Learning word vectors for 157 languages. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan (2018)

    Google Scholar 

  8. Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., Bing, G.: Learning from class-imbalanced data: review of methods and applications. Expert Syst. Appl. 73, 220–239 (2017). https://doi.org/10.1016/j.eswa.2016.12.035

    Article  Google Scholar 

  9. Hartmann, J., Huppertz, J., Schamp, C., Heitmann, M.: Comparing automated text classification methods. Int. J. Res. Mark. 36(1), 20–38 (2019). https://doi.org/10.1016/j.ijresmar.2018.09.009

    Article  Google Scholar 

  10. He, H., Garcia, E.A.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21(9), 1263–1284 (2009). https://doi.org/10.1109/TKDE.2008.239

    Article  Google Scholar 

  11. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  12. Hospedales, T.M., Gong, S., Xiang, T.: Finding rare classes: active learning with generative and discriminative models. IEEE Trans. Knowl. Data Eng. 25(2), 374–386 (2011). https://doi.org/10.1109/TKDE.2011.231

    Article  Google Scholar 

  13. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018)

  14. Jaccard, P.: Distribution de la flore alpine dans le bassin des dranses et dans quelques régions voisines. Bulletin de la Société Vaudoise des Sciences Naturelles 37, 241–272 (1901)

    Google Scholar 

  15. Johnson, J.M., Khoshgoftaar, T.M.: Survey on deep learning with class imbalance. J. Big Data 6(1), 1–54 (2019). https://doi.org/10.1186/s40537-019-0192-5

    Article  Google Scholar 

  16. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016)

  17. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)

  18. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  19. Kittinaradorn, R., et al.: Deepcut: a Thai word tokenization library using deep neural network (2019). https://doi.org/10.5281/zenodo.3457707

  20. Kudisthalert, W., Pasupa, K.: A coefficient comparison of weighted similarity extreme learning machine for drug screening. In: Proceedings of 2016 8th International Conference on Knowledge and Smart Technology (KST), Chiangmai, Thailand, pp. 43–48 (2016). https://doi.org/10.1109/KST.2016.7440525

  21. Lee, K., Palsetia, D., Narayanan, R., Patwary, M.M.A., Agrawal, A., Choudhary, A.: Twitter trending topic classification. In: Proceedings of the 11th International Conference on Data Mining Workshops (ICDM 2011), Vancouver, BC, Canada, pp. 251–258 (2011). https://doi.org/10.1109/ICDMW.2011.171

  22. Li, D., Qian, J.: Text sentiment analysis based on long short-term memory. In: Proceedings of the International Conference on Computer Communication and the Internet (ICCCI 2016), Wuhan, China, pp. 471–475 (2016). https://doi.org/10.1109/CCI.2016.7778967

  23. Li, Y., Guo, H., Zhang, Q., Gu, M., Yang, J.: Imbalanced text sentiment classification using universal and domain-specific knowledge. Knowl. Based Syst. 160, 1–15 (2018). https://doi.org/10.1016/j.knosys.2018.06.019

    Article  Google Scholar 

  24. Lin, T., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42(2), 318–327 (2020)

    Article  Google Scholar 

  25. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: Proceedings of the 4th International Conference on 3D Vision (3DV 2016), Stanford, CA, USA, pp. 565–571 (2016). https://doi.org/10.1109/3DV.2016.79

  26. Nam, J., Kim, J., Mencía, E.L., Gurevych, I., Fürnkranz, J.: Large-scale multi-label text classification-revisiting neural networks. In: Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD 2014), Berlin, Heidelberg, pp. 437–452 (2014). https://doi.org/10.1007/978-3-662-44851-9_28

  27. Pasupa, K., Kudisthalert, W.: Virtual screening by a new clustering-based weighted similarity extreme learning machine approach. PLoS ONE 13(4), e0195478 (2018). https://doi.org/10.1371/journal.pone.0195478

  28. Pasupa, K., Ayutthaya, T.S.N.: Thai sentiment analysis with deep learning techniques: a comparative study based on word embedding, POS-tag, and sentic features. Sustain. Cities Soc. 50, 101615 (2019). https://doi.org/10.1016/j.scs.2019.101615

    Article  Google Scholar 

  29. Pasupa, K., Vatathanavaro, S., Tungjitnob, S.: Convolutional neural networks based focal loss for class imbalance problem: a case study of canine redblood cells morphology classification. J. Ambient Intell. Humanized Comput. (2020). https://doi.org/10.1007/s12652-020-01773-x

  30. Pratama, B.Y., Sarno, R.: Personality classification based on Twitter text using Naive Bayes, KNN and SVM. In: Proceedings of 2015 International Conference on Data and Software Engineering (ICoDSE 2015), Yogyakarta, Indonesia, pp. 170–174 (2015). https://doi.org/10.1109/ICODSE.2015.7436992

  31. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39 (06 2015). https://doi.org/10.1109/TPAMI.2016.2577031

  32. Sarker, A., et al.: Data and systems for medication-related text classification and concept normalization from Twitter: insights from the social media mining for health (smm4h)-2017 shared task. J. Am. Med. Inform. Assoc. 25(10), 1274–1283 (2018). https://doi.org/10.1093/jamia/ocy114

    Article  Google Scholar 

  33. Ayutthaya, T.S.N., Pasupa, K.: Thai sentiment analysis via bidirectional LSTM-CNN model with embedding vectors and sentic features. In: Proceedings of 2018 International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP), Pattaya, Thailand, pp. 1–6 (2018). https://doi.org/10.1109/iSAI-NLP.2018.8692836

  34. Sirirattanajakarin, S., Suntisrivaraporn, B.: Annotation intent identification toward enhancement of marketing campaign performance. In: Proceedings of the 11th International Conference on Knowledge and Systems Engineering (KSE 2019), Da Nang, Vietnam, pp. 1–5 (2019). https://doi.org/10.1109/KSE.2019.8919386

  35. Srinilta, C., Sunhem, W., Tungjitnob, S., Thasanthiah, S., Vatathanavaro, S., et al.: Lyric-based sentiment polarity classification of Thai songs. In: Proceedings of the International Multi Conference of Engineers and Computer Scientists (IMECS 2017), Hong Kong (2017)

    Google Scholar 

  36. The Siam Commercial Bank: SCB EASY Changes Digital-platform Scenes Once Again, Makes Tomorrow’s Services Happen Today, Highlights Moment Banking to Fulfil the Needs in Each of Your Moments. https://www.scb.co.th/en/about-us/news/oct-2018/nws-scbeasy-moment-banking.html (2018). Accessed Feb 2020

  37. Vincent, G., Guillard, G., Bowes, M.: Fully automatic segmentation of the prostate using active appearance models. MICCAI Grand Challenge Prostate MR Image Segmentation 2012, 2 (2012)

    Google Scholar 

  38. Wang, J.H., Liu, T.W., Luo, X., Wang, L.: An LSTM approach to short text sentiment classification with word embeddings. In: Proceedings of the 30th Conference on Computational Linguistics and Speech Processing (ROCLING 2018), Hsinchu, Taiwan, pp. 214–223 (2018)

    Google Scholar 

  39. Wikipedia Developers: Wikipedia. https://www.wikipedia.org/

  40. Xu, G., Meng, Y., Qiu, X., Yu, Z., Wu, X.: Sentiment analysis of comment texts based on BiLSTM. IEEE Access 7, 51522–51532 (2019). https://doi.org/10.1109/ACCESS.2019.2909919

    Article  Google Scholar 

  41. Yan, Y., Wang, Y., Gao, W.-C., Zhang, B.-W., Yang, C., Yin, X.-C.: LSTM\(^{2}\): multi-label ranking for document classification. Neural Proc. Lett. 47(1), 117–138 (2017). https://doi.org/10.1007/s11063-017-9636-0

    Article  Google Scholar 

  42. Zhang, C., Tan, K.C., Li, H., Hong, G.S.: A cost-sensitive deep belief network for imbalanced classification. IEEE Trans. Neural Netw. Learn. Syst. 30(1), 109–122 (2018). https://doi.org/10.1109/TNNLS.2018.2832648

    Article  Google Scholar 

  43. Zhang, M.L., Zhou, Z.H.: Multilabel neural networks with applications to functional genomics and text categorization. IEEE Trans. Knowl. Data Eng. 18(10), 1338–1351 (2006). https://doi.org/10.1109/TKDE.2006.162

    Article  Google Scholar 

  44. Zhou, C., Sun, C., Liu, Z., Lau, F.: A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630 (2015)

  45. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., Xu, B.: Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. arXiv preprint arXiv:1611.06639 (2016)

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology King Mongkut’s Institute of Technology Ladkrabang, Bangkok, 10520, Thailand

    Supawit Vatathanavaro & Kitsuchart Pasupa

  2. Data Analytics, Chief Data Office Siam Commercial Bank, Bangkok, 10900, Thailand

    Sorratat Sirirattanajakarin & Boontawee Suntisrivaraporn

Authors
  1. Supawit Vatathanavaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kitsuchart Pasupa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sorratat Sirirattanajakarin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Boontawee Suntisrivaraporn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kitsuchart Pasupa .

Editor information

Editors and Affiliations

  1. Microsoft Research, Redmond, WA, USA

    Yuxiao Dong

  2. University College Dublin, Dublin, Ireland

    Georgiana Ifrim

  3. Jožef Stefan Institute, Ljubljana, Slovenia

    Dunja Mladenić

  4. Amazon Alexa Knowledge, Cambridge, UK

    Craig Saunders

  5. Ghent University, Kotrijk, Belgium

    Sofie Van Hoecke

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vatathanavaro, S., Pasupa, K., Sirirattanajakarin, S., Suntisrivaraporn, B. (2021). Improved Identification of Imbalanced Multiple Annotation Intent Labels with a Hybrid BLSTM and CNN Model and Hybrid Loss Function. In: Dong, Y., Ifrim, G., Mladenić, D., Saunders, C., Van Hoecke, S. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12461. Springer, Cham. https://doi.org/10.1007/978-3-030-67670-4_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67670-4_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67669-8

  • Online ISBN: 978-3-030-67670-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation