Skip to main content
SpringerLink
Log in

Multimodal fuzzy granular representation and classification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In a complex classification task, samples are represented by various types of multimodal features, including structured data, text, images, video, audio, etc. These data are usually high dimensionally, large-sized, structurally complex, and semantically inconsistent. The representation, translation, alignment, fusion and co-learning of multimodal data are core technical challenges to traditional classification tasks. Kernel functions are applied in dealing with multimodal data for extracting some nonlinear information. However, they cannot consider the aspects of complex structures and uncertain semantics in a multimodal classification task. Fuzzy granular computing emerges as a powerful vehicle to handle the structured and uncertain multimodal data. In this paper, we propose a framework of multimodal classification based on kernel functions and fuzzy granular computing. First, a fuzzy granulation based on kernel functions is introduced to extract nonlinear features for the multimodal classification. Then, a model of multimodal fuzzy classification including fuzzy granular representation, fusion and learning for multimodal data is constructed. Finally, we design an efficient fuzzy granular classification algorithm for big multimodal data based on the proposed model. Experimental results demonstrate the effectiveness of our proposed model and its corresponding algorithm.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the currentstudy are available in the [UCI] repository, [https://archive.ics.uci.edu/ml/index.php].

References

  1. Ramachandram D, Taylor GW (2017) Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Mag 34(6):96–108

    Google Scholar 

  2. Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimed Syst 16:345–379

    Google Scholar 

  3. Shi Y, Suk HI, Gao Y, Lee SW, Shen D (2019) Leveraging coupled interaction for multimodal alzheimer’s disease diagnosis. IEEE Trans. Neural Netw Learn Syst 31(1):186–200

  4. Baltrušaitis T, Ahuja C, Morency LP (2018) Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41(2):423–443

    Google Scholar 

  5. Ullah H, Ullah B, Wu L, Abdalla FY, Ren G, Zhao Y (2020) Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-laplacian in non-subsampled shearlet transform domain. Biomed Signal Process Control 57:1–14

    Google Scholar 

  6. Zhou T, Ruan S, Canu S (2019) A review: Deep learning for medical image segmentation using multi-modality fusion. Array 3:1–11

    Google Scholar 

  7. Wang P, Cai D (2020) Multiple kernel learning by empirical target kernel. International Journal of Wavelets, Multiresolution and Information Processing 18(02):1950058

    MathSciNet  MATH  Google Scholar 

  8. Zhang X, Mahoor MH (2016) Task-dependent multi-task multiple kernel learning for facial action unit detection. Pattern Recognit 51:187–196

    Google Scholar 

  9. Sultan R, Tamimi H, Ashhab Y (2019) Improving classification performance using genetic programming to evolve string kernels. Int Arab J Inf Technol 16(3):454–459

    Google Scholar 

  10. Maji S, Berg AC, Malik J (2008) Classification using intersection kernel support vector machines is efficient. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8

  11. Jiang H, Shen D, Ching WK, Qiu Y (2022) A high-order norm-product regularized multiple kernel learning framework for kernel optimization. Inf Sci 606:72–91

    Google Scholar 

  12. Xue H, Song Y, Xu HM (2019) Multiple indefinite kernel learning for feature selection. Knowl Based Syst 191

  13. Xue H, Song Y (2020) Non-convex approximation based l 0-norm multiple indefinite kernel feature selection. Appl Intell 50:192–202

    Google Scholar 

  14. Liao Z, Gao L, Zhou T, Fan X, Zhang Y, Wu J (2019) An oil painters recognition method based on cluster multiple kernel learning algorithm. IEEE Access 7:26842–26854

    Google Scholar 

  15. Yang M, Wang Z, Li Y, Zhou Y, Li D, Du W (2022) Gravitation balanced multiple kernel learning for imbalanced classification. Neural Comput Appl 34(16):13807–13823

    Google Scholar 

  16. Chi Z, Wang Z, Wang B, Fang Z, Zhu Z, Li D, Du W (2022) Multiple kernel subspace learning for clustering and classification. IEEE Transactions on Knowledge and Data Engineering

  17. Zadeh LA (1979) Fuzzy sets and information granular. In: Fuzzy Set Theory and Applications, pp. 3–18

  18. Kerk YW, Tay KM, Lim CP (2019) Monotone interval fuzzy inference systems. IEEE Trans Fuzzy Syst 27(11):2255–2264

    Google Scholar 

  19. Tian H, Wang X, Lu Z, Huang Y, Tian G (2017) Adaptive fuzzy logic energy management strategy based on reasonable soc reference curve for online control of plug-in hybrid electric city bus. IEEE Trans Intell Transp Syst 19(5):1607–1617

    Google Scholar 

  20. Xiao Z, Gong Z, Liu K (2020) Fuzzy linear systems with the two-dimension fuzzy data. J Intell Fuzzy Syst 38(4):5295–5315

    Google Scholar 

  21. Ngo LT, Dang TH, Pedrycz W (2018) Towards interval-valued fuzzy set-based collaborative fuzzy clustering algorithms. Pattern Recognit 81:404–416

    Google Scholar 

  22. Vatankhah Ghadim H, Tarafdar Hagh M, Ghassem Zadeh S (2022) Fermat-curve based fuzzy inference system for the fuzzy logic controller performance optimization in load frequency control application. Fuzzy Optim Decis Mak1–32

  23. Salimi-Badr A, Ebadzadeh MM (2020) A novel self-organizing fuzzy neural network to learn and mimic habitual sequential tasks. IEEE Trans Cybern 52(1):323–332

    Google Scholar 

  24. Lang G, Miao D, Fujita H (2019) Three-way group conflict analysis based on pythagorean fuzzy set theory. IEEE Transactions on Fuzzy Systems 28(3):447–461

    Google Scholar 

  25. Singh PK (2019) Bipolar fuzzy concept learning using next neighbor and euclidean distance. Soft Computing 23(12):4503–4520

    MATH  Google Scholar 

  26. Zadeh LA (1996) Fuzzy logic= computing with words. IEEE Trans Fuzzy Syst 4(2):103–111

    Google Scholar 

  27. Zadeh LA (1997) Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic. Fuzzy Sets Syst 90(2):111–127

    MathSciNet  MATH  Google Scholar 

  28. Zadeh LA (2001) A new direction in ai: Toward a computational theory of perceptions. AI magazine 22(1):73–73

    MATH  Google Scholar 

  29. Pedrycz W, Izakian H (2014) Cluster-centric fuzzy modeling. IEEE Trans Fuzzy Syst 22(6):1585–1597

    Google Scholar 

  30. Al-Hmouz R, Pedrycz W, Balamash A, Morfeq A (2015) Description and classification of granular time series. Soft Computing 19:1003–1017

    MATH  Google Scholar 

  31. Pedrycz W, Park BJ, Oh SK (2008) The design of granular classifiers: a study in the synergy of interval calculus and fuzzy sets in pattern recognition. Pattern Recognit 41(12):3720–3735

    MATH  Google Scholar 

  32. Roh SB, Pedrycz W, Ahn TC (2014) A design of granular fuzzy classifier. Expert Syst Appl 41(15):6786–6795

    Google Scholar 

  33. Yang T, Zhong X, Lang G, Qian Y, Dai J (2020) Granular matrix: A new approach for granular structure reduction and redundancy evaluation. IEEE Trans Fuzzy Syst 28(12):3133–3144

    Google Scholar 

  34. Singh PK (2019) Vague concept lattice reduction using granular computing and vague entropy. Math Comput Simul 165:56–73

    MathSciNet  MATH  Google Scholar 

  35. Tan A, Wu WZ, Qian Y, Liang J, Chen J, Li J (2018) Intuitionistic fuzzy rough set-based granular structures and attribute subset selection. IEEE Trans Fuzzy Syst 27(3):527–539

    Google Scholar 

  36. Yu P, Li J, Wang H, Lin G (2019) Matrix-based approaches for updating approximations in multigranulation rough set while adding and deleting attributes. Int J Comput Intell Syst 12(2):855–872

    Google Scholar 

  37. Sun L, Wang L, Ding W, Qian Y, Xu J (2020) Feature selection using fuzzy neighborhood entropy-based uncertainty measures for fuzzy neighborhood multigranulation rough sets. IEEE Trans Fuzzy Syst 29(1):19–33

    Google Scholar 

  38. Fujita H, Gaeta A, Loia V, Orciuoli F (2018) Resilience analysis of critical infrastructures: a cognitive approach based on granular computing. IEEE Trans Cybern 49(5):1835–1848

    Google Scholar 

  39. Fujita H, Gaeta A, Loia V, Orciuoli F (2019) Hypotheses analysis and assessment in counterterrorism activities: a method based on owa and fuzzy probabilistic rough sets. IEEE Trans Fuzzy Syst 28(5):831–845

  40. Qian J, Liu C, Yue X (2019) Multigranulation sequential three-way decisions based on multiple thresholds. Int J Approx Reason 105:396–416

    MathSciNet  MATH  Google Scholar 

  41. Chen Y, Miao D (2020) Granular regression with a gradient descent method. Inf Sci 537:246–260

    MathSciNet  MATH  Google Scholar 

  42. Qian J, Hong C, Yu Y, Liu C, Miao D (2022) Generalized multigranulation sequential three-way decision models for hierarchical classification. Inf Sci 616:66–87

    Google Scholar 

  43. Niu J, Chen D, Li J, Wang H (2022) A dynamic rule-based classification model via granular computing. Inf Sci 584:325–341

    Google Scholar 

  44. Shen Y, Pedrycz W, Wang X (2018) Clustering homogeneous granular data: Formation and evaluation. IEEE Trans Cybern 49(4):1391–1402

    Google Scholar 

  45. Chen Y, Qin N, Li W, Xu F (2019) Granule structures, distances and measures in neighborhood systems. Knowl Based Syst 165:268–281

    Google Scholar 

  46. Yue X, Zhou J, Yao Y, Miao D (2020) Shadowed neighborhoods based on fuzzy rough transformation for three-way classification. IEEE Trans Fuzzy Syst 28(5):978–991

    Google Scholar 

  47. Zadeh LA (1965) Fuzzy sets. Inf. Control 8(3):338–353

    MathSciNet  MATH  Google Scholar 

  48. Pal SK, Dutta-Majumder DK (1986) Fuzzy Mathematical Approach to Pattern Recognition

  49. Bede B, Bede B (2013) Fuzzy Analysis. Springer

    MATH  Google Scholar 

  50. Smola AJ, Schölkopf B (1998) Learning with Kernels vol. 4, 639–646. Springer Berlin Heidelberg

  51. Tata S, Patel JM (2007) Estimating the selectivity of tf-idf based cosine similarity predicates. ACM Sigmod Record 36(2):7-12

    Google Scholar 

  52. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. Computer Ence, 1–12

  53. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543

  54. Spiegel MR (1975) Theory and problems of probability and statistics. ESAIM - Probab Stat 114–115

  55. Lu L, Huang B, Zhang Q, Ke D, Xu Y (2012) Research on algorithm of combing lda-based discriminant classifier and mfcc feature extraction for pure acoustic listening similarity. Int J Adv Comput Technol 4(5):106–113

    Google Scholar 

  56. Jana DK, Bhunia P, Adhikary SD, Mishra A (2023) Analyzing of salient features and classification of wine type based on quality through various neural network and support vector machine classifiers. Results Control Optim 11:100219

    Google Scholar 

  57. Chen Y, Qin N, Li W, Xu F (2019) Granule structures, distances and measures in neighborhood systems. Knowl Based Syst 165:268–281

    Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No. 61976183) and the Research Cooperation Project of Industry and University in Fujian Province (No. 2020H6101).

Author information

Authors and Affiliations

  1. School of Mechanical and Automotive Engineering, Xiamen University of Technology, Xiamen, 361024, China

    Fenggang Han

  2. School of Computer & Information Engineering, Xiamen University of Technology, Xiamen, 361024, China

    Xiao Zhang, Linjie He, Liru Kong & Yumin Chen

Authors
  1. Fenggang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linjie He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liru Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yumin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liru Kong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, F., Zhang, X., He, L. et al. Multimodal fuzzy granular representation and classification. Appl Intell 53, 29433–29447 (2023). https://doi.org/10.1007/s10489-023-05080-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05080-8

Keywords

Search

Navigation