Skip to main content
SpringerLink
Log in

AACR: Feature Fusion Effects of Algebraic Amalgamation Composed Representation on (De)Compositional Network for Caption Generation for Images

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Progress in image captioning is gradually getting complex as researchers try to generalize the model and define the representation between visual features and natural language processing. In the absence of any established relationship, every time a new dividend is added, it produced very little improvement, not considerable enough to make it general. This work tried to define such kind of relationship in the form of representation called Algebraic Amalgamation-based Composed Representation (AACR) which generalized the scheme of language modeling and structuring the linguistic attributes (related to grammar and parts of speech of language) which will provide a much better structure and grammatically correct sentence. AACR enables better and more unique representation and structuring of the feature space and enables transfer learning like infrastructure for all machines to interact with the external world (both human and machine) with these representations. A large part of the different ways of defining and improving these AACR are discussed and their performance concerning the traditional procedures and feature representations are evaluated for image captioning application. The new models achieved considerable improvement than the corresponding previous architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Anderson P, He X, Buehler C, Teney D, Johnson M, Gould S, Zhang L. Bottom-up and top-down attention for image captioning and visual question answering. CVPR. 2018;3(5):6.

    Google Scholar 

  2. Anne HL, et al. Deep compositional captioning: describing novel object categories without paired training data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1–10.

  3. Chen H, Ding G, Lin Z, Zhao S, Ha J. Show, observe and tell: attribute-driven attention model for image captioning. In: IJCAI, 2018, pp. 606–12.

  4. Chen M, Ding G, Zhao S, Chen H, Liu Q, Han J. Reference based LSTM for image captioning. In: AAAI, 2017, pp. 3981–87.

  5. Chen H, Zhang H, Chen PY, Yi J, Hsieh CJ Show-and-fool: crafting adversarial examples for neural image captioning. arXiv preprint. 2017; arXiv:1712.02051.

  6. Chen T, Zhang Z, You Q, Fang C, Wang Z, Jin H, Luo J. Factual or emotional: stylized image captioning with adaptive learning and attention. arXiv preprint. 2018; arXiv:1807.03871.

  7. Chen F, Ji R, Sun X, Wu Y, Su J. GroupCap: group-based image captioning with structured relevance and diversity constraints. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1345–53.

  8. Chen X, Lawrence Zitnick C. Mind’s eye: a recurrent visual representation for image caption generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2422–31.

  9. Chen F, Ji R, Su J, Wu Y, Wu Y. Structcap: structured semantic embedding for image captioning. In: Proceedings of the 2017 ACM on multimedia conference, ACM, 2017, pp. 46–54.

  10. Chunseong Park C, Kim B, Kim G. Attend to you: personalized image captioning with context sequence memory networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 895–903.

  11. Cohn-Gordon R, Goodman N, Potts C. Pragmatically informative image captioning with character-level reference. arXiv preprint. 2018; arXiv:1804.05417.

  12. Cornia M, Baraldi L, Serra G, Cucchiara R. Paying more attention to saliency: image captioning with saliency and context attention. ACM Trans Multimed Comput Commun Appl. 2018;14(2):48.

    Article  Google Scholar 

  13. Devlin J, et al. Language models for image captioning: the quirks and what works. arXiv preprint. 2015; arXiv:1505.01809.

  14. Devlin J, Gupta S, Girshick R, Mitchell M, Zitnick CL. Exploring nearest neighbor approaches for image captioning. arXiv preprint. 2015; arXiv:1505.04467.

  15. Donahue J, et al. Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2625–34.

  16. Fang H, et al. From captions to visual concepts and back. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1473–82.

  17. Farhadi A, et al. Every picture tells a story: generating sentences from images. In: European conference on computer vision, Springer, Berlin, Heidelberg, 2010.

  18. Fu K, Jin J, Cui R, Sha F, Zhang C. Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts. IEEE Trans Pattern Anal Mach Intell. 2017;39(12):2321–34.

    Article  Google Scholar 

  19. Fu K, Li J, Jin J, Zhang C. Image-text surgery: efficient concept learning in image captioning by generating pseudopairs. IEEE Trans Neural Netw Learn Syst. 2018;99:1–12.

    Google Scholar 

  20. Gan C, et al. Stylenet: generating attractive visual captions with styles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3137–46.

  21. Gan Z, et al. Semantic compositional networks for visual captioning. arXiv preprint. 2016; arXiv:1611.08002.

  22. Girshick R, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014. pp. 580–87.

  23. Harzig P, Brehm S, Lienhart R, Kaiser C, Schallner R. Multimodal image captioning for marketing analysis. arXiv preprint. 2018; arXiv:1802.01958.

  24. Hodosh M, Young P, Hockenmaier J. Framing image description as a ranking task: data, models and evaluation metrics. J Artif Intell Res. 2013;47:853–99.

    Article  MathSciNet  MATH  Google Scholar 

  25. Jia X, et al. Guiding the long-short term memory model for image caption generation. In: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2407–15.

  26. Jiang W, Ma L, Chen X, Zhang H, Liu W. Learning to guide decoding for image captioning. arXiv preprint. 2018; arXiv:1804.00887.

  27. Jin J, et al. Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv preprint. 2015; arXiv:1506.06272.

  28. Karpathy A, Armand J, Fei Fei FL. Deep fragment embeddings for bidirectional image sentence mapping. In: Advances in neural information processing systems, 2014, pp. 1889–97.

  29. Karpathy A, Fei-Fei L. Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp. 3128–37.

  30. Kilickaya M, Akkus BK, Cakici R, Erdem A, Erdem E, Ikizler-Cinbis N. Data-driven image captioning via salient region discovery. IET Comput Vis. 2017;11(6):398–406.

    Article  Google Scholar 

  31. Kiros R, Ruslan S, Zemel RS. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint. 2014; arXiv:1411.2539.

  32. Kiros R, Zemel R, Salakhutdinov Ruslan R. A multiplicative model for learning distributed text-based attribute representations. Adv Neural Inf Process Syst. 2014.

  33. Kiros R, Salakhutdinov R, Zemel R. Multimodal neural language models. In: International conference on machine learning, 2014, pp. 595–603.

  34. Krishna R, et al. Visual genome: connecting language and vision using crowdsourced dense image annotations. Int J Comput Vis. 2017;123(1):32–73.

    Article  MathSciNet  Google Scholar 

  35. Kulkarni G, et al. Babytalk: understanding and generating simple image descriptions. IEEE Trans Pattern Anal Mach Intell. 2013;35(12):2891–903.

    Article  Google Scholar 

  36. Kuznetsova P, et al. TREETALK: composition and compression of trees for image descriptions. TACL. 2014;2(10):351–62.

    Google Scholar 

  37. LTran D, et al. Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision, 2015, pp. 4489–97.

  38. Li X, Wang X, Xu C, Lan W, Wei Q, Yang G, Xu J. COCO-CN for cross-lingual image tagging, captioning and retrieval. arXiv preprint. 2018; arXiv:1805.08661.

  39. Li S, et al. Composing simple image descriptions using web-scale n-grams. In: Proceedings of the fifteenth conference on computational natural language learning. Association for computational linguistics, 2011.

  40. Liu S, Zhu Z, Ye N, Guadarrama S, Murphy K. Improved image captioning via policy gradient optimization of spider. Proc IEEE Int Conf Comput Vis. 2017;3:3.

    Google Scholar 

  41. Liu C, Sun F, Wang C, Wang F, Yuille A. MAT: a multimodal attentive translator for image captioning. arXiv preprint. 2017; arXiv:1702.05658.

  42. Liu X, Li H, Shao J, Chen D, Wang X. Show, tell and discriminate: image captioning by self-retrieval with partially labeled data. arXiv preprint. 2018); arXiv:1803.08314.

  43. Liu C, Mao J, Sha F, Yuille AL. Attention correctness in neural image captioning. In: AAAI, 2017, pp. 4176–82.

  44. Lu J, Xiong C, Parikh D, Socher R. Knowing when to look: adaptive attention via a visual sentinel for image captioning. Proc IEEE Conf Comput Vis Pattern Recognit. 2017;6:2.

    Google Scholar 

  45. Lu D, Whitehead S, Huang L, Ji H, Chang SF. Entity-aware image caption generation. arXiv preprint. 2018; arXiv:1804.07889.

  46. Lu J, Yang J, Batra D, Parikh D. Neural baby talk. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7219–28.

  47. Mao J, et al. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint. 2014; arXiv:1412.6632.

  48. Mao J, et al. Learning like a child: fast novel visual concept learning from sentence descriptions of images. In: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2533–41.

  49. Mathews AP, Lexing X, Xuming H. SentiCap: generating image descriptions with sentiments. In: Thirtieth AAAI conference on artificial intelligence. 2016.

  50. Melnyk I, Sercu T, Dognin PL, Ross J, Mroueh Y. Improved image captioning with adversarial semantic alignment. arXiv preprint. 2018; arXiv:1805.00063.

  51. Memisevic R, Geoffrey H. Unsupervised learning of image transformations. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2007, pp. 1–8.

  52. Mitchell M, et al. Midge: generating image descriptions from computer vision detections. In: Proceedings of the 13th conference of the European chapter of the association for computational linguistics. Association for computational linguistics, 2012.

  53. Ordonez V, Girish K, Berg TL. Im2text: describing images using 1 million captioned photographs. In: Advances in neural information processing systems, 2011, pp. 1143–51.

  54. Palangi H, Smolensky P, He X, Deng L. Question-answering with grammatically-interpretable representations. 2017. arXiv:1705.08432

  55. Park CC, Kim B, Kim G. Towards personalized image captioning via multimodal memory networks. IEEE Trans Pattern Anal Mach. 2018;41(4):999–12.

    Article  Google Scholar 

  56. Pu Y, et al. Variational autoencoder for deep learning of images, labels and captions. Adv Neural Inf Process Syst. 2016.

  57. Ren Z, Wang X, Zhang N, Lv X, Li LJ. Deep reinforcement learning-based image captioning with embedding reward. arXiv preprint. 2017; arXiv:1704.03899.

  58. Rennie SJ, Marcheret E, Mroueh Y, Ross J, Goel V. Self-critical sequence training for image captioning. CVPR. 2017;1(2):3.

    Google Scholar 

  59. Sharma P, Ding N, Goodman S, Soricut R. Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th annual meeting of the association for computational linguistics, 2018, vol. 1, pp. 2556–65.

  60. Socher R, et al. Grounded compositional semantics for finding and describing images with sentences. Trans Assoc Comput Linguist. 2014;2:207–18.

    Article  Google Scholar 

  61. Sur C. UCRLF: unified constrained reinforcement learning framework for phase-aware architectures for autonomous vehicle signaling and trajectory optimization. Evol Intell. 2019;12(4):689–12.

    Article  Google Scholar 

  62. Sur C. Survey of deep learning and architectures for visual captioning-transitioning between media and natural languages. Multimed Tools Appl. 2019;78(22):32187–237.

    Article  Google Scholar 

  63. Sur C. Representation for language understanding. Gainesville: University of Florida; 2018. pp. 1–90. https://drive.google.com/file/d/15Fhmt5aM_b0J5jtE9mdWInQPfDS3TqVw.

  64. Sur C. SACT: Self-aware multi-space feature composition transformer for multinomial attention for video captioning. 2020; arXiv:2006.14262.

  65. Sur C. ReLGAN: generalization of consistency for GAN with disjoint constraints and relative learning of generative processes for multiple transformation learning. 2020; arXiv:2006.07809.

  66. Sur C. Self-segregating and coordinated-segregating transformer for focused deep multi-modular network for visual question answering. 202; arXiv:2006.14264.

  67. Sur C. Gaussian smoothen semantic features (GSSF)--exploring the linguistic aspects of visual captioning in Indian languages (Bengali) using MSCOCO framework. 2020; arXiv:2002.06701

  68. Sur C. MRRC: Multiple role representation crossover interpretation for image captioning with R-CNN feature distribution composition (FDC). 2020;arXiv:2002.06436.

  69. Sur C. aiTPR: Attribute Interaction-Tensor Product Representation for Image Caption. 2020;arXiv:2001.09545.

  70. Sur C. CRUR: Coupled-Recurrent Unit for Unification, Conceptualization and Context Capture for Language Representation--A Generalization of Bi Directional LSTM. 2019;arXiv:1911.10132 .

  71. Sur C. Tpsgtr: Neural-symbolic tensor product scene-graph-triplet representation for image captioning. 2019;arXiv:1911.10115.

  72. Sur C, Pei L, Yingjie Z, Dapeng W. Semantic tensor product for image captioning. In: 2019 5th international conference on big data computing and communications (BIGCOM), pp. 33–37. IEEE, 2019.

  73. Sur C. Feature Fusion Effects of Tensor Product Representation on (De) Compositional Network for Caption Generation for Images. 2018;arXiv:1812.06624.

  74. Sutskever I, James M, Hinton GE. Generating text with recurrent neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), 2011.

  75. Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, 2014, pp. 3104–12.

  76. Tavakoliy HR, Shetty R, Borji A, Laaksonen J. Paying attention to descriptions generated by image captioning models. In: Computer vision (ICCV), 2017 IEEE international conference, IEEE, 2017, pp. 2506–15.

  77. Tran K, et al. Rich image captioning in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 49–56, 2016.

  78. Vinyals O, Toshev A, Bengio S, Erhan D. Show and tell: lessons learned from the 2015 mscoco image captioning challenge. IEEE Trans Pattern Anal Mach Intell. 2017;39(4):652–63.

    Article  Google Scholar 

  79. Vinyals O, et al. Show and tell: a neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3156–3164.

  80. Wang C, Haojin Y, Christoph M. Image captioning with deep bidirectional LSTMs and multi-task learning. ACM Trans Multimed Comput Commun Appl. 2018;14(2s):40.

    Google Scholar 

  81. Wang Y, Lin Z, Shen X, Cohen S, Cottrell GW. Skeleton key: image captioning by skeleton-attribute decomposition. arXiv preprint 2017;arXiv:1704.06972.

  82. Wu Q, Shen C, Wang P, Dick A, van den Hengel A. Image captioning and visual question answering based on attributes and external knowledge. IEEE Trans Pattern Anal Mach. 2017;40(6):1367–81.

    Article  Google Scholar 

  83. Wu C, Wei Y, Chu X, Su F, Wang L. Modeling visual and word-conditional semantic attention for image captioning. Signal Process Image Commun. 2018;67:100–7.

    Article  Google Scholar 

  84. Wu Q, et al. What value do explicit high level concepts have in vision to language problems?. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 203–12.

  85. Wu J, Hu Z, Mooney RJ. Joint image captioning and question answering. arXiv preprint. 2018;arXiv:1805.08389.

  86. Xu K, et al. Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning, 2015, pp. 2048–57.

  87. Yang Z, et al. Review networks for caption generation. In: Advances in neural information processing systems, 2016, pp. 2361–69.

  88. Yang Y, et al. Corpus-guided sentence generation of natural images. In: Proceedings of the conference on empirical methods in natural language processing. Association for computational linguistics, 2011.

  89. Yang Z, Yuan Y, Wu Y, Salakhutdinov R, Cohen WW. Encode, review, and decode: reviewer module for caption generation. arXiv preprint. 2016;arXiv:1605.07912.

  90. Yao T, Pan Y, Li Y, Mei T. Incorporating copying mechanism in image captioning for learning novel objects. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, 2017, pp. 5263–71.

  91. Yao T, Pan Y, Li Y, Qiu Z, Mei T. Boosting image captioning with attributes. In: IEEE international conference on computer vision, ICCV, 2017, pp. 22–29.

  92. Ye S, Liu N, Han J. Attentive linear transformation for image captioning. IEEE Trans Image Process. 2018.

  93. You Q, Jin H, Luo J. Image captioning at Will: a versatile scheme for effectively injecting sentiments into image descriptions. arXiv preprint. 2018;arXiv:1801.10121.

  94. You Q, et al. Image captioning with semantic attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4651–59.

  95. Young P, et al. From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans Assoc Comput Linguist. 2014;2:67–78.

    Article  Google Scholar 

  96. Zhang M, Yang Y, Zhang H, Ji Y, Shen HT, Chua TS. More is better: precise and detailed image captioning using online positive recall and missing concepts mining. IEEE Trans Image Process. 2018;28(1):32–44.

    Article  MathSciNet  MATH  Google Scholar 

  97. Zhang L, Sung F, Liu F, Xiang T, Gong S, Yang Y, Hospedales TM. Actor-critic sequence training for image captioning. arXiv preprint. 2017; arXiv:1706.09601.

  98. Zhao W, Wang B, Ye J, Yang M, Zhao Z, Luo R, Qiao Y. A multi-task learning approach for image captioning. In: IJCAI, 2018, pp. 1205–11.

Download references

Acknowledgements

The author has used University of Florida HiperGator, equipped with NVIDIA Tesla K80 GPU, extensively for the experiments. The author acknowledges University of Florida Research Computing for providing computational resources and support that have contributed to the research results reported in this publication. URL: eduhttp://researchcomputing.ufl.

Author information

Authors and Affiliations

  1. Computer and Information Science and Engineering Department, University of Florida, Gainesville, USA

    Chiranjib Sur

Authors
  1. Chiranjib Sur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chiranjib Sur.

Ethics declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sur, C. AACR: Feature Fusion Effects of Algebraic Amalgamation Composed Representation on (De)Compositional Network for Caption Generation for Images. SN COMPUT. SCI. 1, 229 (2020). https://doi.org/10.1007/s42979-020-00238-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-020-00238-4

Keywords

Search

Navigation