Skip to main content
SpringerLink
Log in

Deep Refinement: capsule network with attention mechanism-based system for text classification

  • Deep Learning & Neural Computing for Intelligent Sensing and Control
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Most of the text in the questions of community question–answering systems does not consist of a definite mechanism for the restriction of inappropriate and insincere content. A given piece of text can be insincere if it asserts false claims or assumes something which is debatable or has a non-neutral or exaggerated tone about an individual or a group. In this paper, we propose a pipeline called Deep Refinement which utilizes some of the state-of-the-art methods for information retrieval from highly sparse data such as capsule network and attention mechanism. We have applied the Deep Refinement pipeline to classify the text primarily into two categories, namely sincere and insincere. Our novel approach ‘Deep Refinement’ provides a system for the classification of such questions in order to ensure enhanced monitoring and information quality. The database used to understand the real concept of what actually makes up sincere and insincere includes quora insincere question dataset. Our proposed question classification method outperformed previously used text classification methods, as evident from the F1 score of 0.978.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Liu B (2012) Sentiment analysis and opinion mining. Synth Lect Hum Lang Technol 5(1):1–167

    Article  Google Scholar 

  2. Novozhilov D, Kotenko I, Chechulin A (2016) Improving the categorization of web sites by analysis of html-tags statistics to block inappropriate content. In: Intelligent distributed computing IX. Springer, Berlin, pp 257–263

  3. Belinkov Y, Mohtarami M, Cyphers S, Glass J (2015) Vectorslu: a continuous word vector approach to answer selection in community question answering systems. In: Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp 282–287

  4. Gabbard S, Yang J, Liu J (2018) Quora insincere question classification. Baskin Engineering, University of California, Santa Cruz

  5. Smith LN (2017) Cyclical learning rates for training neural networks. In: 2017 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 464–472

  6. Wang Z-Q, Sun X, Zhang D-X, Li X (2006) An optimal SVM-based text classification algorithm. In: 2006 International conference on machine learning and cybernetics, IEEE, pp 1378–1381

  7. Liu Z, Lv X, Liu K, Shi S (2010) Study on svm compared with the other text classification methods. In: 2010 Second international workshop on education technology and computer science, vol 1, pp 219–222, IEEE

  8. Haryanto AW, Mawardi EK et al (2018) Influence of word normalization and chi squared feature selection on support vector machine (svm) text classification. In: 2018 International seminar on application for technology of information and communication, IEEE, pp 229–233

  9. Huang Z, Thint M, Qin Z (2008) Question classification using head words and their hypernyms. In: Proceedings of the conference on empirical methods in natural language processing, association for computational linguistics, pp 927–936

  10. Haniewicz K, Rutkowski W, Adamczyk M, Kaczmarek M (2013) Towards the lexicon-based sentiment analysis of polish texts: polarity lexicon. In: International conference on computational collective intelligence. Springer, pp 286–295

  11. Zhang H, Wei H, Tang Y, Pu Q (2019) Research on classification of scientific and technological documents based on naive Bayes. In: Proceedings of the 2019 11th international conference on machine learning and computing. ACM, pp 327–331

  12. Qiang G (2010) An effective algorithm for improving the performance of naive Bayes for text classification. In: 2010 second international conference on computer research and development

  13. Narayanan V, Arora I, Bhatia A (2013) Fast and accurate sentiment classification using an enhanced naive Bayes model. In: International conference on intelligent data engineering and automated learning. Springer, pp 194–201

  14. Pratama BY, Sarno R (2015) Personality classification based on twitter text using naive Bayes, KNN and SVM. In: 2015 International conference on data and software engineering (ICoDSE), IEEE, pp 170–174

  15. Santos CND, Xiang B, Zhou B Classifying relations by ranking with convolutional neural networks. arxiv preprint arXiv:1504.06580

  16. Georgakopoulos SV, Tasoulis SK, Vrahatis AG, Plagianakos VP (2018) Convolutional neural networks for toxic comment classification. In: Proceedings of the 10th Hellenic conference on artificial intelligence. ACM, p 35

  17. Kim Y Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882

  18. Dachapally PR, Ramanam S In-depth question classification using convolutional neural networks. arxiv preprint arXiv:1804.00968

  19. Zhou C, Sun C, Liu Z, Lau F A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630

  20. Yenala H, Jhanwar A, Chinnakotla MK, Goyal J (2018) Deep learning for detecting inappropriate content in text. Int J Data Sci Anal 6(4):273–286

    Article  Google Scholar 

  21. Sabour S, Frosst N, Hinton G (2018) Matrix capsules with EM routing. In: 6th international conference on learning representations, ICLR

  22. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, pp 3856–3866

  23. Zhao W, Ye J, Yang M, Lei Z, Zhang S, Zhao Z Investigating capsule networks with dynamic routing for text classification. arXiv preprint arXiv:1804.00538

  24. Yang M, Zhao W, Chen L, Qu Q, Zhao Z, Shen Y (2019) Investigating the transferring capability of capsule networks for text classification. Neural Netw 118:247–261

    Article  Google Scholar 

  25. Zhao W, Peng H, Eger S, Cambria E, Yang M Towards scalable and reliable capsule networks for challenging NLP applications. arXiv preprint arXiv:1906.02829

  26. Zhang N, Deng S, Sun Z, Chen X, Zhang W, Chen H Attention-based capsule networks with dynamic routing for relation extraction. arXiv preprint arXiv:1812.11321

  27. Li J, Yang B, Dou Z-Y, Wang X, Lyu MR, Tu Z Information aggregation for multi-head attention with routing-by-agreement. arXiv preprint arXiv:1904.03100

  28. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  29. Mungekar A, Parab N, Nima P, Pereira S (2019) Quora insincere question classification. National College of Ireland

  30. Chen S, Song B, Guo J (2018) Attention alignment multimodal LSTM for fine-gained common space learning. IEEE Access 6:20195–20208

    Article  Google Scholar 

  31. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  32. Zhou J, Lu Y, Dai HN, Wang H, Xiao H (2019) Sentiment analysis of Chinese microblog based on stacked bidirectional LSTM. IEEE Access 7:38856–38866

    Article  Google Scholar 

  33. Long F, Zhou K, Ou W (2019) Sentiment analysis of text based on bidirectional LSTM with multi-head attention. IEEE Access

  34. Bin Y, Yang Y, Shen F, Xie N, Shen HT, Li X (2018) Describing video with attention-based bidirectional LSTM. IEEE Trans Cybern 49(7):2631–2641

    Article  Google Scholar 

  35. Kowsari K, Brown DE, Heidarysafa M, Meimandi KJ, Gerber MS, Barnes LE (2017) Hdltex: hierarchical deep learning for text classification. In: 2017 16th IEEE international conference on machine learning and applications (ICMLA), IEEE, pp 364–371

  36. Bahdanau D, Cho K, Bengio Y Neural machine translation by jointly learning to align and translate. arxiv preprint arXiv:1409.0473

  37. Lin A, Li J, Ma Z (2019) On learning and learned data representation by capsule networks. IEEE Access 7:50808–50822

    Article  Google Scholar 

  38. Li S, Li M, Xu Y, Bao Z, Fu L, Zhu Y (2018) Capsules based Chinese word segmentation for ancient Chinese medical books. IEEE Access 6:70874–70883

    Article  Google Scholar 

  39. Paoletti ME, Haut JM, Fernandez-Beltran R, Plaza J, Plaza A, Li J, Pla F (2018) Capsule networks for hyperspectral image classification. IEEE Trans Geosci Remote Sens 57(4):2145–2160

    Article  Google Scholar 

  40. Keren G, Sabato S, Schuller B (2018) Fast single-class classification and the principle of logit separation. In: 2018 IEEE international conference on data mining (ICDM). IEEE, pp 227–236

  41. Rodriguez JD, Perez A, Lozano JA (2009) Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans Pattern Anal Mach Intell 32(3):569–575

    Article  Google Scholar 

  42. Ketkar N (2017) Introduction to pytorch. In: Deep learning with python. Springer, Berlin, pp 195–208

Download references

Acknowledgements

This work was supported in part by the Key Laboratory of Intelligent Air-Ground Cooperative Control for Universities in Chongqing and the Key Laboratory of Industrial IoT and Networked Control, Ministry of Education, College of Automation, Chongqing University of Posts and Telecommunications, Chongqing, China, and the work of Xiangyuan Lan was supported by Hong Kong Baptist University Tier 1 Start-up Grant.

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Air-Ground Cooperative Control for Universities in Chongqing, College of Automation, Chongqing University of Posts and Telecommunications, Chongqing, China

    Deepak Kumar Jain

  2. Department of Computer Science and Engineering, Bharati Vidyapeeth’s College of Engineering, New Delhi, India

    Rachna Jain, Yash Upadhyay & Abhishek Kathuria

  3. Department of Computer Science, Hong Kong Baptist University, Kowloon, Hong Kong

    Xiangyuan Lan

Authors
  1. Deepak Kumar Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rachna Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yash Upadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abhishek Kathuria
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiangyuan Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deepak Kumar Jain.

Ethics declarations

Conflict of interest

There is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jain, D.K., Jain, R., Upadhyay, Y. et al. Deep Refinement: capsule network with attention mechanism-based system for text classification. Neural Comput & Applic 32, 1839–1856 (2020). https://doi.org/10.1007/s00521-019-04620-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04620-z

Keywords

Search

Navigation