Skip to main content
SpringerLink
Log in

Algorithm for Double-Layer Structure Multi-label Classification with Optimal Sequence Based on Attention Mechanism

  • Conference paper
  • First Online:
Simulation Tools and Techniques (SIMUtools 2020)

Included in the following conference series:

  • 857 Accesses

Abstract

A common approach to multi-label classification is to perform problem transformation, whereby a multi-label problem is transformed into one or more single-label problems. Problem transformation considers label correlations by extending the attributes, but ignores the importance of each feature attribute for different classification targets, weakens the sensitivity of the classifier, and reduces the classification accuracy. Attention mechanism is a model that simulates the mechanism of human brain attention. It mainly emphasizes the influence of some crucial inputs on the output by calculating the attention probability distribution, which has a good optimization effect on the traditional model. Based on this, this paper proposes a two-layer chain structure multi-label classification (ATDCC-OS) algorithm, which incorporates the attention mechanism. This algorithm constructs a two-layer multi-label classification model in order to realize the correlation between labels through inter-layer and intra-layer interaction. At the same time, the attention mechanism is introduced to focus selectively on the sample features, identify more important information for the current task, and further improve the classification performance of the algorithm. Furthermore, an optimal sequence selection algorithm (OSS) is proposed, seeking to label the pecking order, solving the problem of reduced classification accuracy caused by randomly selecting the class label sequence to train the binary classifier by the chain classification model. The OSS will be used to optimize the second-layer chain classification model of ATDCC-OS. Comparisons on seven benchmark data sets with related algorithms verify the effectiveness of ATDCC-OS.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Shao, H., Li, G., Liu, G., Wang, Y.: Symptom selection for multi-label data of inquiry diagnosis in traditional Chinese medicine. Sci. China Inf. Sci. 56(5), 1–3 (2013)

    Article  MathSciNet  Google Scholar 

  2. Glinka, K., Wosiak, A., Zakrzewska, D.: Improving children diagnostics by efficient multi-label classification method. In: Piętka, E., Badura, P., Kawa, J., Wieclawek, W. (eds.) Information Technologies in Medicine. AISC, vol. 471, pp. 253–266. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39796-2_21

    Chapter  Google Scholar 

  3. Yao, L., Poblenz, E., Dagunts, D., Covington, B., Bernard, D., Lyman, K.: Learning to diagnose from scratch by exploiting dependencies among labels. arXiv preprint arXiv:1710.10501 (2017)

  4. Zhu, H., et al.: Learning tree-based deep model for recommender systems. In: The 2018 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2018)

    Google Scholar 

  5. Son, J., Kim, S.B., Kim, H., Cho, S.: Review and analysis of recommender systems. J. Korean Inst. Ind. Eng. 41(2), 185–208 (2015)

    Google Scholar 

  6. Bogaert, M., Lootens, J., Van den Poel, D., Ballings, M.: Evaluating multi-label classifiers and recommender systems in the financial service sector. Eur. J. Oper. Res. 279(2), 620–634 (2019)

    Article  Google Scholar 

  7. Ray, J., Heng Wang, D., Tran, Y.W., Feiszli, M., Torresani, L., Paluri, M.: Scenes-objects-actions: a multi-task, multi-label video dataset. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XIV, pp. 660–676. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_39

    Chapter  Google Scholar 

  8. Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: CVPR (2019)

    Google Scholar 

  9. Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label classification. In: Buntine, W., Grobelnik, M., Mladenić, D., Shawe-Taylor, J. (eds.) ECML PKDD 2009. LNCS (LNAI), vol. 5782, pp. 254–269. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04174-7_17

    Chapter  Google Scholar 

  10. Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Patt. Recogn. 37(9), 1757–1771 (2004)

    Article  Google Scholar 

  11. Godbole, S., Sarawagi, S.: Discriminative methods for multi-labeled classification. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS (LNAI), vol. 3056, pp. 22–30. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24775-3_5

    Chapter  Google Scholar 

  12. Liu, G.Q., Guo, T.: Algorithm for multi-label classification with optimal sequence based on double layers. Comput. Eng. Des. 37(4), 921-927+948 (2016)

    Google Scholar 

  13. Pan, C., Tan, J., Feng, D., Li, Y.: Very short-term solar generation forecasting based on LSTM with temporal attention mechanism. In: ICCC (2019)

    Google Scholar 

  14. Chen, W.J., Shao, Y.H., Li, C.N., Deng, N.Y.: MLTSVM: a novel twin support vector machine to multi-label learning. Patt. Recogn. 52, 61–74 (2016)

    Article  Google Scholar 

  15. Xu, X., Shan, D., Li, S., Sun, T., Xiao, P., Fan, J.: Multi-label learning method based on ML-RBF and laplacian ELM. Neurocomputing 331, 213–219 (2019)

    Article  Google Scholar 

  16. Zhang, N., Ding, S., Zhang, J.: Multi layer ELM-RBF for multi-label learning. Appl. Soft. Comput. 43, 535–545 (2016)

    Article  Google Scholar 

  17. Roseberry, M., Krawczyk, B., Cano, A.: Multi-label punitive kNN with self-adjusting memory for drifting data streams. ACM Trans. Knowl. Disc. Data 13(6), 1–31 (2019). https://doi.org/10.1145/3363573

    Article  Google Scholar 

  18. Yapu, D.: An Improved ML-KNN Approach for Weibo text classification. Chin. Comput. Commun. 7, 18 (2018)

    Google Scholar 

  19. Wu, G., Zheng, R., Tian, Y., Liu, D.: Joint ranking SVM and binary relevance with robust Low-rank learning for multi-label classification. Neural Netw. 122, 24–39 (2020)

    Article  Google Scholar 

  20. Charte, F., Rivera, A., del Jesus, M.J., Herrera, F.: Improving multi-label classifiers via label reduction with association rules. In: Corchado, E., Snášel, V., Abraham, A., Woźniak, M., Graña, M., Cho, S.-B. (eds.) HAIS 2012. LNCS (LNAI), vol. 7209, pp. 188–199. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28931-6_18

    Chapter  Google Scholar 

  21. Luo, F., Guo, W., Yu, Y., Chen, G.: A multi-label classification algorithm based on kernel extreme learning machine. Neurocomputing 260, 313–320 (2017)

    Article  Google Scholar 

  22. Kulessa, M., Mencía, E.L.: Dynamic classifier chain with random decision trees. In: Soldatova, L., Vanschoren, J., Papadopoulos, G., Ceci, M. (eds.) DS 2018. LNCS (LNAI), vol. 11198, pp. 33–50. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01771-2_3

    Chapter  Google Scholar 

  23. Ali, T., Asghar, S.: Efficient label ordering for improving multi-label classifier chain accuracy. J. Nat. Sci. Found. Sri Lanka 47(2), 175 (2019). https://doi.org/10.4038/jnsfsr.v47i2.9159

    Article  MathSciNet  Google Scholar 

  24. Firat, O., Cho, K., Bengio, Y.: Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint. arXiv:1601.0107 (2016)

  25. Mnih, V., Heess, N., Graves, A.: Recurrent models of visual attention. In: NIPS, pp. 2204–2212. MIT Press, US (2014)

    Google Scholar 

  26. Borji, A., Itti, L.: State-of-the-art in visual attention modeling. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 185–207 (2012)

    Article  Google Scholar 

  27. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint. arXiv:1409.0473 (2014)

  28. Mnih, V., Heess, N., Graves, A.: Recurrent models of visual attention. In:NIPS, pp. 2204–2212 (2014)

    Google Scholar 

  29. Rush, A.M., Chopra, S., Weston, J.: A neural attention model for abstractive sentence summarization. arXiv preprint. arXiv:1509.00685 (2015)

  30. Chen, H., Sun, M., Tu, C., Lin, Y., Liu, Z.: Neural sentiment classification with user and product attention. In: EMNLP (2016)

    Google Scholar 

  31. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In:NAACL HLT, pp. 1480–1489. ACL, California (2016)

    Google Scholar 

  32. He, R., Lee, W. S., Ng, H. T., Dahlmeier, D.: An unsupervised neural attention model for aspect extraction. In: the 55th Annual Meeting of the Association for Computational Linguistics, pp. 388–397. ACL, Canada (2017)

    Google Scholar 

  33. Yin, W., Schütze, H., Xiang, B., Zhou, B.: Abcnn: Attention-based convolutional neural network for modeling sentence pairs. Trans. Assoc. Comput. Linguist. 4, 259–272 (2016)

    Article  Google Scholar 

  34. Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008. MIT Press, US (2017)

    Google Scholar 

  35. Coelho, F., Braga, A.P., Verleysen, M.: A mutual information estimator for continuous and discrete variables applied to feature selection and classification problems. Int. J. Comput. Intell. Syst. 9(4), 726–733 (2016)

    Article  Google Scholar 

  36. Sefidian, A.M., Daneshpour, N.: Missing value imputation using a novel grey based fuzzy c-means, mutual information based feature selection, and regression model. Exp. Syst. Appl. 115, 68–94 (2019)

    Article  Google Scholar 

  37. Cao, X., Cong, G., Jensen, C.S.: Mining significant semantic locations from GPS data. Proc. VLDB Endow. 3(1–2), 1009–1020 (2010)

    Article  Google Scholar 

  38. Yanagisawa, T., et al.: Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann. Neurol. 71(3), 353–361 (2012)

    Article  Google Scholar 

  39. Liu, X.S., et al.: High-resolution peripheral quantitative computed tomography can assess microstructural and mechanical properties of human distal tibial bone. J. Bone Miner. Res. 25(4), 746–756 (2010)

    Google Scholar 

  40. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30(1–7), 107–117 (1998). https://doi.org/10.1016/S0169-7552(98)00110-X

    Article  Google Scholar 

  41. Singhal, R., Srivastava, S.R.: Enhancing the page ranking for search engine optimization based on weightage of in-linked web pages. In: ICRAIE (2016)

    Google Scholar 

  42. Gupta, P., Goel, A., Lin, J., Sharma, A., Wang, D., Zadeh, R.: Wtf: the who to follow service at Twitter. In: IW3C2 (2013)

    Google Scholar 

  43. Fletcher, P., Hoyle, H., Patty, C.W.: Foundations of Discrete Mathematics. PWS-KENT Pub. Co., Boston (1991)

    MATH  Google Scholar 

  44. Jiang, D., Wang, Y., Lv, Z., Qi, S., Singh, S.: Big data analysis based network behavior insight of cellular networks for industry 4.0 applications. IEEE Trans. Ind. Inf. 16(2), 1310–1320 (2020). https://doi.org/10.1109/TII.2019.2930226

    Article  Google Scholar 

  45. Jiang, D., Huo, L., Lv, Z., Song, H., Qin, W.: A joint multi-criteria utility-based network selection approach for vehicle-to-infrastructure networking. IEEE Trans. Intell. Transp. Syst. 19(10), 3305–3319 (2018)

    Article  Google Scholar 

  46. Wang, Y., Jiang, D., Huo, L., Zhao, Y.: A new traffic prediction algorithm to software defined networking. Mob. Netw. Appl., 1–10 (2019).https://doi.org/10.1007/s11036-019-01423-3

  47. Wang, F., Jiang, D., Qi, S.: An adaptive routing algorithm for integrated information networks. China Commun. 16(7), 195–206 (2019)

    Article  Google Scholar 

  48. Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proc. Am. Math. Soc. 7(1), 48–50 (1956)

    Article  MathSciNet  Google Scholar 

  49. Tsoumakas, G., Spyromitros-Xioufis, E., Vilcek, J., Vlahavas, I.: Mulan: a java library for multi-label learning. J. Mach. Learn. Res. 12, 2411–2414 (2011)

    MathSciNet  MATH  Google Scholar 

  50. Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data Warehouse. Min. 3(3), 1–3 (2007)

    Article  Google Scholar 

  51. Zhang, M.L., Zhou, Z.H.: A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng. 26(8), 1819–1837 (2013)

    Article  Google Scholar 

  52. Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36 (1982)

    Article  Google Scholar 

  53. Da, K.: A method for stochastic optimization. arXiv preprint. arXiv:1412.6980 (2014)

  54. DemšCar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7(1), 30 (2006)

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge the financial support of the Planning Subject for the 13th Five Year Plan of National Education Sciences under Grant No. DCA160258 and the Key Research Project of Education Department of Sichuan Province of China under Grant No. 18ZA319.

Author information

Authors and Affiliations

  1. College of Engineering and Technology, Sichuan Radio and Television University, Chengdu, China

    Geqiao Liu & Mingjie Tan

  2. Research Center for Educational Information Management and Information Systems, Open University of China, Beijing, China

    Geqiao Liu & Mingjie Tan

Authors
  1. Geqiao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingjie Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Geqiao Liu .

Editor information

Editors and Affiliations

  1. Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

    Houbing Song

  2. School of Astronautics and Aeronautic, UESTC, Chengdu, China

    Dingde Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, G., Tan, M. (2021). Algorithm for Double-Layer Structure Multi-label Classification with Optimal Sequence Based on Attention Mechanism. In: Song, H., Jiang, D. (eds) Simulation Tools and Techniques. SIMUtools 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 369. Springer, Cham. https://doi.org/10.1007/978-3-030-72792-5_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72792-5_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72791-8

  • Online ISBN: 978-3-030-72792-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation