Skip to main content
SpringerLink
Log in

Overview of the NLPCC 2022 Shared Task: Multi-modal Dialogue Understanding and Generation

  • Conference paper
  • First Online:
Book cover Natural Language Processing and Chinese Computing (NLPCC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13552))

Abstract

In this paper, we give an overview of multi-modal dialogue understanding and generation at NLPCC 2022 shared task, which includes three sub-tasks: dialogue scene identification, dialogue session identification, and dialogue response generation. A bilingual multi-modal dialogue dataset consisting of 100M utterances was made public for the shared task. The dataset contains 119K dialogue scene boundaries and 62K dialogue session boundaries which are both annotated manually. Details of the shared task, dataset, evaluation metric and evaluation results will be presented in order.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agrawal, A., et al.: VQA: visual question answering. Int. J. Comput. Vis. 123, 4–31 (2015)

    Article  MathSciNet  Google Scholar 

  2. AlAmri, H., et al.: Audio visual scene-aware dialog. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7550–7559 (2019)

    Google Scholar 

  3. Baraldi, L., Grana, C., Cucchiara, R.: A deep siamese network for scene detection in broadcast videos. In: Proceedings of the 23rd ACM International Conference on Multimedia (2015)

    Google Scholar 

  4. Chasanis, V., Likas, A.C., Galatsanos, N.P.: Scene detection in videos using shot clustering and sequence alignment. IEEE Trans. Multimedia 11, 89–100 (2009)

    Article  Google Scholar 

  5. Colby, K.M., Weber, S., Hilf, F.D.: Artificial paranoia. Artif. Intell. 2, 1–25 (1971)

    Article  Google Scholar 

  6. Das, A., et al.: Visual dialog. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 326–335 (2017)

    Google Scholar 

  7. Galley, M., McKeown, K., Fosler-Lussier, E., Jing, H.: Discourse segmentation of multi-party conversation. In: ACL (2003)

    Google Scholar 

  8. Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., Xu, W.: Are you talking to a machine? Dataset and methods for multilingual image question. In: NIPS (2015)

    Google Scholar 

  9. Gao, J., Galley, M., Li, L.: Neural approaches to conversational AI. arXiv arXiv:1809.08267 (2019)

  10. Han, B., Wu, W.: Video scene segmentation using a novel boundary evaluation criterion and dynamic programming. In: 2011 IEEE International Conference on Multimedia and Expo, pp. 1–6 (2011)

    Google Scholar 

  11. Hearst, M.A.: Text tiling: segmenting text into multi-paragraph subtopic passages. Comput. Linguist. 23, 33–64 (1997)

    Google Scholar 

  12. Hori, C., et al.: End-to-end audio visual scene-aware dialog using multimodal attention-based video features. In: 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2019, pp. 2352–2356. IEEE (2019)

    Google Scholar 

  13. Huang, Q., Xiong, Y., Rao, A., Wang, J., Lin, D.: MovieNet: a holistic dataset for movie understanding. arXiv arXiv:2007.10937 (2020)

  14. Khan, O.Z., Robichaud, J.P., Crook, P.A., Sarikaya, R.: Hypotheses ranking and state tracking for a multi-domain dialog system using multiple ASR alternates. In: INTERSPEECH (2015)

    Google Scholar 

  15. Lavie, A., Agarwal, A.: METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the 2nd Workshop on Statistical Machine Translation, pp. 228–231 (2007)

    Google Scholar 

  16. Le, H., Chen, N.F., Hoi, S.: Learning reasoning paths over semantic graphs for video-grounded dialogues. In: International Conference on Learning Representations (2021)

    Google Scholar 

  17. Le, H., Sahoo, D., Chen, N., Hoi, S.C.: BiST: bi-directional spatio-temporal reasoning for video-grounded dialogues. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1846–1859 (2020)

    Google Scholar 

  18. Le, H., Sahoo, D., Chen, N.F., Hoi, S.C.H.: Multimodal transformer networks for end-to-end video-grounded dialogue systems. In: ACL (2019)

    Google Scholar 

  19. Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., Jurafsky, D.: Adversarial learning for neural dialogue generation. In: EMNLP (2017)

    Google Scholar 

  20. Li, Z., Li, Z., Zhang, J., Feng, Y., Niu, C., Zhou, J.: Bridging text and video: a universal multimodal transformer for video-audio scene-aware dialog. arXiv preprint arXiv:2002.00163 (2020)

  21. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)

    Google Scholar 

  22. Malinowski, M., Fritz, M.: A multi-world approach to question answering about real-world scenes based on uncertain input. In: NIPS (2014)

    Google Scholar 

  23. Meng, Y., et al.: OpenViDial: a large-scale, open-domain dialogue dataset with visual contexts. arXiv arXiv:2012.15015 (2020)

  24. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics (2002)

    Google Scholar 

  25. Protasov, S., Khan, A., Sozykin, K., Ahmad, M.: Using deep features for video scene detection and annotation. Sig. Image Video Process. 12, 991–999 (2018)

    Article  Google Scholar 

  26. Rao, A., et al.: A local-to-global approach to multi-modal movie scene segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10143–10152 (2020)

    Google Scholar 

  27. Rasheed, Z., Shah, M.: Scene detection in hollywood movies and tv shows. In: 2003 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, p. II-343 (2003)

    Google Scholar 

  28. Rotman, D., Porat, D., Ashour, G.: Optimal sequential grouping for robust video scene detection using multiple modalities. Int. J. Semant. Comput. 11, 193–208 (2017)

    Article  Google Scholar 

  29. Rui, Y., Huang, T.S., Mehrotra, S.: Exploring video structure beyond the shots. In: Proceedings of the IEEE International Conference on Multimedia Computing and Systems (Cat. No.98TB100241), pp. 237–240 (1998)

    Google Scholar 

  30. Shum, H., He, X., Li, D.: From Eliza to Xiaoice: challenges and opportunities with social chatbots. Front. Inf. Technol. Electron. Eng. 19, 10–26 (2018)

    Article  Google Scholar 

  31. Sidiropoulos, P., Mezaris, V., Kompatsiaris, Y., Meinedo, H., Bugalho, M.M.F., Trancoso, I.: Temporal video segmentation to scenes using high-level audiovisual features. IEEE Trans. Circ. Syst. Video Technol. 21, 1163–1177 (2011)

    Article  Google Scholar 

  32. Song, Y., Mou, L., Yan, R., Yi, L., Zhu, Z., Hu, X., Zhang, M.: Dialogue session segmentation by embedding-enhanced texttiling. arXiv arXiv:1610.03955 (2016)

  33. Sordoni, A., et al.: A neural network approach to context-sensitive generation of conversational responses. In: NAACL (2015)

    Google Scholar 

  34. Takanobu, R., et al.: A weakly supervised method for topic segmentation and labeling in goal-oriented dialogues via reinforcement learning. In: IJCAI (2018)

    Google Scholar 

  35. Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CIDEr: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)

    Google Scholar 

  36. Wang, S., Meng, Y., Li, X., Sun, X., Ouyang, R., Li, J.: OpenViDial 2.0: a larger-scale, open-domain dialogue generation dataset with visual contexts. arXiv arXiv:2109.12761 (2021)

  37. Wang, S., et al.: Modeling text-visual mutual dependency for multi-modal dialog generation. arXiv arXiv:2105.14445 (2021)

  38. Weizenbaum, J.: Eliza—a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 36–45 (1966)

    Article  Google Scholar 

  39. Xing, L., Carenini, G.: Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring. In: SIGDIAL (2021)

    Google Scholar 

  40. Xu, Y., Zhao, H., Zhang, Z.: Topic-aware multi-turn dialogue modeling. In: AAAI (2021)

    Google Scholar 

  41. Zhao, T., Zhao, R., Eskénazi, M.: Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In: ACL (2017)

    Google Scholar 

  42. Zhou, L., Gao, J., Li, D., Shum, H.: The design and implementation of XiaoIce, an empathetic social chatbot. Comput. Linguist. 46, 53–93 (2020)

    Article  Google Scholar 

  43. Zhu, Y., Groth, O., Bernstein, M.S., Fei-Fei, L.: Visual7W: grounded question answering in images. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4995–5004 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Wangxuan Institute of Computer Technology, Peking University, Beijing, China

    Yuxuan Wang, Xueliang Zhao & Dongyan Zhao

  2. Center for Data Science, AAIS, Peking University, Beijing, China

    Yuxuan Wang, Xueliang Zhao & Dongyan Zhao

  3. Artificial Intelligence Institute, Peking University, Beijing, China

    Dongyan Zhao

Authors
  1. Yuxuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xueliang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dongyan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongyan Zhao .

Editor information

Editors and Affiliations

  1. Singapore University of Technology and Design, Singapore, Singapore

    Wei Lu

  2. Nanjing University, Nanjing, China

    Shujian Huang

  3. Soochow University, Suzhou, China

    Yu Hong

  4. Soochow University, Soochow, China

    Xiabing Zhou

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Y., Zhao, X., Zhao, D. (2022). Overview of the NLPCC 2022 Shared Task: Multi-modal Dialogue Understanding and Generation. In: Lu, W., Huang, S., Hong, Y., Zhou, X. (eds) Natural Language Processing and Chinese Computing. NLPCC 2022. Lecture Notes in Computer Science(), vol 13552. Springer, Cham. https://doi.org/10.1007/978-3-031-17189-5_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17189-5_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17188-8

  • Online ISBN: 978-3-031-17189-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation