Skip to main content
SpringerLink
Log in

Location emotion recognition for travel recommendation based on social network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

With the rapid extension of social network, a huge amount of text and multimedia data concerning venues have constantly been generated. More and more people now try to obtain others’ reviews on one venue from the different social network places. Thus, it is essential to analyze both text and multimedia content in an integral manner to get a better venue semantic evaluation, which can effectively be applied to several applications such as travel recommendation, venue summarization, emergency monitoring. In this paper, we propose a novel multimedia location emotion recognition model to handle this problem. First, we utilize traditional and classic emotion multimedia datasets to train several recognition models according to different modals. Then, we propose a novel method to fuse the recognition results provided by these different pre-train recognition models and output the emotion label of the given venue. Finally, we recommend the related venues to users with respect to the emotion label. In order to demonstrate the performance of our approach, we collect the related location multimedia data. The related experiments also demonstrate the superiority of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Nie, W.-Z., Peng, W.-J., Wang, X., Zhao, Y., Yu-Ting, S.: Multimedia venue semantic modeling based on multimodal data. J. Vis. Commun. Image Represent. 48, 375–385 (2017)

    Article  Google Scholar 

  2. Chen, K.-Y., Luesukprasert, L., Seng-cho, T.C.: Hot topic extraction based on timeline analysis and multidimensional sentence modeling. IEEE Trans. Knowl. Data Eng. 19(8), 1016–1025 (2007)

    Article  Google Scholar 

  3. Wang, X., Zhao, Y.-L., Nie, L., Gao, Y., Nie, W., Zha, Z.-J., Chua, T.-S.: Semantic-based location recommendation with multimodal venue semantics. IEEE Trans. Multimed. 17(3), 409–419 (2015)

    Article  Google Scholar 

  4. Jin, L., Li, K., Hao, H., Qi, G.-J., Tang, J.: Semantic neighbor graph hashing for multimodal retrieval. IEEE Trans. Image Process. 27(3), 1405–1417 (2018)

    Article  MathSciNet  Google Scholar 

  5. Zhao, S., Ding, G., Gao, Y., Han, J.: Learning visual emotion distributions via multi-modal features fusion. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 369–377. ACM, New York (2017)

  6. Zhao, S., Yao, H., Gao, Y., Ding, G., Chua, T.-S.: Predicting personalized image emotion perceptions in social networks. IEEE Trans. Affect. Comput. 9(4), 526–540 (2016)

    Article  Google Scholar 

  7. Chung-Hsien, W., Chuang, Z.-J., Lin, Y.-C.: Emotion recognition from text using semantic labels and separable mixture models. ACM Trans. Asian Lang. Inf. Process. (TALIP) 5(2), 165–183 (2006)

    Article  Google Scholar 

  8. Zhao, S., Zhao, X., Ding, G., Keutzer, K.: Emotiongan: unsupervised domain adaptation for learning discrete probability distributions of image emotions. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 1319–1327. ACM, New York (2018)

  9. Cheng, Z., Chang, X., Zhu, L., Kanjirathinkal, R.C., Kankanhalli, M.: Mmalfm: explainable recommendation by leveraging reviews and images. ACM Trans. Inf. Syst. (TOIS) 37(2), 16 (2019)

    Article  Google Scholar 

  10. Chung-Hsien, W., Liang, W.-B.: Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels. IEEE Trans. Affect. Comput. 2(1), 10–21 (2011)

    Article  Google Scholar 

  11. Ebrahimi Kahou, S., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., et al.: Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 543–550. ACM, New York (2013)

  12. Zhao, S., Yao, H., Gao, Y., Ji, R., Ding, G.: Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans. Multimed. 19(3), 632–645 (2017)

    Article  Google Scholar 

  13. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. Preprint (2013). arXiv:1301.3781

  14. Zhao, S., Ding, G., Gao, Y., Han, J.: Approximating discrete probability distribution of image emotions by multi-modal features fusion. Transfer 1000(1), 4669–4675 (2017)

    Google Scholar 

  15. Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 467–474. ACM, New York (2015)

  16. Cheng, X., Yan, X., Lan, Y., Guo, J.: Btm: topic modeling over short texts. IEEE Trans. Knowl. Data Eng. 1, 1–1 (2014)

    Google Scholar 

  17. Mei, Q., Cai, D., Zhang, D., Zhai, C.: Topic modeling with network regularization. In: Proceedings of the 17th International Conference on World Wide Web, pp. 101–110. ACM, New York (2008)

  18. Lu, Y., Zhai, C.: Opinion integration through semi-supervised topic modeling. In: Proceedings of the 17th International Conference on World Wide Web, pp. 121–130. ACM, New York (2008)

  19. Tang, J., Jin, R., Zhang, J.: A topic modeling approach and its integration into the random walk framework for academic search. In: 8th IEEE International Conference on Data Mining, 2008. ICDM’08, pp. 1055–1060. IEEE (2008)

  20. Larochelle, H., Lauly, S.: A neural autoregressive topic model. In: Advances in Neural Information Processing Systems, vol. 2, pp. 2708–2716 (2012)

  21. Zheng, Y., Zhang, Y.-J., Larochelle, H.: Topic modeling of multimodal data: an autoregressive approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1370–1377 (2014)

  22. Zhao, X., Wang, N., Zhang, Y., Shaoyi, D., Gao, Y., Sun, J.: Beyond pairwise matching: person reidentification via high-order relevance learning. IEEE Trans. Neural Netw. Learn. Syst. 29(8), 3701–3714 (2018)

    Article  Google Scholar 

  23. Zhao, S., Gao, Y., Ding, G., Chua, T.-S.: Real-time multimedia social event detection in microblog. IEEE Trans. Cybern. 99, 1–14 (2017)

    Google Scholar 

  24. Nguyen, C.-T., Zhan, D.-C., Zhou, Z.-H.: Multi-modal image annotation with multi-instance multi-label LDA. In: 23rd International Joint Conference on Artificial Intelligence (2013)

  25. Zhang, Z., Lin, H., Zhao, X., Ji, R., Gao, Y.: Inductive multi-hypergraph learning and its application on view-based 3D object classification. IEEE Trans. Image Process. 27(12), 5957–5968 (2018)

    Article  MathSciNet  Google Scholar 

  26. Min, W., Bao, B.-K., Xu, C., Hossain, M.S., et al.: Cross-platform multi-modal topic modeling for personalized inter-platform recommendation. IEEE Trans. Multimed. 17(10), 1787–1801 (2015)

    Article  Google Scholar 

  27. Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 83–92. ACM, New York (2010)

  28. Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., Morency, L.-P.: Context-dependent sentiment analysis in user-generated videos. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 873–883 (2017)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (61872267, 61502337, 61772359, 61472275).

Author information

Authors and Affiliations

  1. The School of Electrical and Information Engineering, Tianjin University, Tianjin, China

    Weizhi Nie, Hai Ding, Dan Song & Xingjian Long

Authors
  1. Weizhi Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hai Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xingjian Long
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dan Song or Xingjian Long.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nie, W., Ding, H., Song, D. et al. Location emotion recognition for travel recommendation based on social network. SIViP 13, 1259–1266 (2019). https://doi.org/10.1007/s11760-019-01457-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-019-01457-w

Keywords

Search

Navigation