Abstract
Chinese classical poetry generation from images is an overwhelmingly challenging work in the field of artificial intelligence. Inspired by recent advances in automatically generating description of an image and Chinese poem generation, in this paper, we present a generative model based on deep recurrent framework that describes images in the form of poems. Our model consists of two parts, one is to extract information according to the semantics presented in images, and the other is to generate each line of the poem incrementally according to the extracted semantic information from the images by a recurrent neural network. Experimental results thoroughly demonstrate the effectiveness of our approach by manual evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Bernstein, M., Khosla, A., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. IJCV 115, 211–252 (2015)
Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: CVPR, pp. 1602–1605 (2009)
Kulkarni, G., Premraj, V., Dhar, S., Li, S., Berg, A., Choi, Y., Berg, T.: Baby talk: understanding and generating simple image descriptions. In: CVPR (2011)
Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: Every picture tells a story: generating sentences for images. In: ECCV (2010)
Yao, B.Z., Yang, X., Lin, L., Lee, M.W., Zhu, S.C.: I2T: image parsing to text description. Proc. IEEE 98(8), 1485–1508 (2010)
Elliott, D., Keller, F.: Image description using visual dependency representations. In: EMNLP, pp. 1292–1302 (2013)
Li, S., Kulkarni, G., Berg, T.L., Berg, A.C., Choi, Y.: Composing simple image descriptions using web-scale n-grams. In: CoNLL (2011)
Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Intell. Res. 47(1), 853–899 (2013)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. arXiv preprint arXiv:1411.4555 (2014)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015)
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044 (2015)
Zhang, X., Lapata, M.: Chinese poetry generation with recurrent neural networks. In: EMNLP, pp. 670–680 (2014)
Wang, Q., Luo, T., Wang, D., Xing, C.: Chinese song iambics generation with neural attention-based model. CoRR, abs/1604.06274 (2016)
Yi, X., Li, R., Sun, M.: Generating Chinese classical poems with RNN encoder-decoder. CoRR, abs/1604.01537 (2016)
Colton, S., Goodwin, J., Veale, T.: Full FACE poetry generation. In: ICCC, pp. 95–102 (2012)
Oliveira, H.: Automatic generation of poetry: an overview. Universidade de Coimbra (2009)
Oliveira, H.: Poetryme: a versatile platform for poetry generation. Comput. Creat. Concept Inven. Gen. Intell. 1, 21 (2012)
Jiang, L., Zhou, M.: Generating Chinese couplets using a statistical MT approach. In: Proceedings of the 22nd International Conference on Computational Linguistics, pp. 377–384 (2008)
He, J., Zhou, M., Jiang, L.: Generating Chinese classical poems with statistical machine translation models. In: AAAI, pp. 1650–1656 (2012)
Zhou, C.L., You, W., Ding, X.: Genetic algorithm and its implementation of automatic generation of Chinese songci. J. Softw. 21(3), 427–437 (2010)
Wang, L.: A Summary of Rhyming Constraints of Chinese Poems (in Chinese). Beijing Press, Beijing (2002)
Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR (2009)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016)
Wu, Q., Shen, C., Liu, L., Dick, A., van den Hengel, A.: What value do explicit high level concepts have in vision to language problems? In: CVPR (2016)
Hong, R., Yang, Y., Wang, M., Hua, X.-S.: Learning visual semantic relationships for efficient visual retrieval. IEEE Trans. Big Data 1(4), 152–161 (2015)
Zhang, H., Shang, X., Luan, H.-B., Wang, M., Chua, T.-S.: Learning from collective intelligence: feature learning using social images and tags. TOMCCAP 13(1), 1:1–1:23 (2016)
Zhang, H., Kyaw, Z., Chang, S.-F., Chua, T.-S.: Visual translation embedding network for visual relation detection. In: CVPR (2017)
Acknowledgments
This work was supported by the National Natural Science Foundation of China (NSFC) under grants 61472116 and 61502139.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Xing, S., Liu, X., Hong, R., Zhao, Y. (2018). Generating Chinese Poems from Images Based on Neural Network. In: Zeng, B., Huang, Q., El Saddik, A., Li, H., Jiang, S., Fan, X. (eds) Advances in Multimedia Information Processing – PCM 2017. PCM 2017. Lecture Notes in Computer Science(), vol 10735. Springer, Cham. https://doi.org/10.1007/978-3-319-77380-3_52
Download citation
DOI: https://doi.org/10.1007/978-3-319-77380-3_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77379-7
Online ISBN: 978-3-319-77380-3
eBook Packages: Computer ScienceComputer Science (R0)