Abstract
Semantic annotation has attracted a growing interest in the information retrieval and computer vision. Existing methods have typically focused on several visual cues and semantic context information with an image itself using different frameworks, neglecting the prior knowledge constraints about the real world. However, strong prior knowledge embedding should be considered to improve the performance of semantic annotation tasks. Note that semantic objects will interact each other during the semantic prediction stage, and the support visual relationships can affect the recall and accuracy of semantic annotations. In this paper, we exploit a novel method to semantic modeling with prior knowledge embedding to jointly find the semantic objects and the corresponding support relationships in the images. Inference in the model can be conducted exactly via graph modeling and knowledge embedding, and the parameters can be learned at the supervised learning stage. The extensive experiments on COCO15 and Stanford Visual Relationship data sets confirm the benefits of semantic annotation for the objects for the knowledge embedding.
Supported by Research Start-up Fundation for the Doctoral Program of Liaocheng University (318051654) and Shandong Province Higher Education Science and Technology Program (J18KA390) and Natural Science Foundation of Shandong Province (ZR2016AM24, ZR2018BF010).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017)
Grainger, T., AlJadda, K., Korayem, M., Smith, A.: The semantic knowledge graph: a compact, auto-generated model for real-time traversal and ranking of any relationship within a domain. In: Proceedings of the IEEE International Conference on Data Science and Advanced Analytics, Montreal, pp. 420–429. IEEE (2016)
Shalaby, W., Zadrozny, W., Jin, H.: Beyond word embeddings: learning entity and concept representations from large scale knowledge bases (2018). arXiv preprint: arXiv:1801.00388
Raposo, D., Santoro, A., Barrett, D., Pascanu, R., Lillicrap, T., Battaglia, P.: Discovering objects and their relations from entangled scene representations (2017). arXiv preprint: arXiv:1702.05068
Liang, X., Lee, L., Xing, E.P.: Deep variation-structured reinforcement learning for visual relationship and attribute detection. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, pp. 4408–4417. IEEE (2017)
Yu, L.F., Yeung, S.K., Tang, C.K., Terzopoulos, D., Chan, T.F., Osher, S.J.: Make it home: automatic optimization of furniture arrangement. ACM Trans. Graph. 30, 86:1–86:12 (2011)
Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., Hanrahan, P.: Example-based synthesis of 3D object arrangements. ACM Trans. Graph. 31(6), 135 (2012)
Yeh, Y.T., Yang, L., Watson, M., Goodman, N.D., Hanrahan, P.: Synthesizing open worlds with constraints using locally annealed reversible jump MCMC. ACM Trans. Graph. 31(4), 56 (2012)
Fan, Y.: Scene classification based on knowledge sharing and latent structural constraints. In: Proceedings of the International Conference on Computer Science and Network Technology, Harbin, pp. 1356–1360. IEEE (2015)
Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., Jaiswal, S.: Graph2vec: learning distributed representations of graphs (2017). arXiv preprint: arXiv:1707.05005
Duran, A.G., Niepert, M.: Learning graph representations with embedding propagation. In: Advances in Neural Information Processing Systems, pp. 5125–5136 (2017)
Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proc. IEEE 104, 11–13 (2016)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks (2016). arXiv preprint: arXiv:1609.02907
Edwards, M., Xie, X.: Graph based convolutional neural network (2016). arXiv preprint: arXiv:1609.08965
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Ohio, pp. 580–587. IEEE (2014)
Babaee, M., Wolf, T., Rigoll, G.: Toward semantic attributes in dictionary learning and non-negative matrix factorization. Pattern Recogn. Lett. 80, 172–178 (2016)
Hackel, T., Usvyatsov, M., Galliani, S., Wegner, J.D., Schindler, K.: Inference, learning and attention mechanisms that exploit and preserve sparsity in convolutional networks (2018). arXiv preprint: arXiv:1801.10585
Lin, T.-Y.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with language priors. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part I. LNCS, vol. 9905, pp. 852–869. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_51
Yuan, F., Kingsley, K., Jie, L., Cheston, T.: Object detection meets knowledge graphs. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne. IEEE (2017)
Zhuo, W., Salzmann, M., He, X., Miaomiao, L.: Indoor scene parsing with instance segmentation, semantic labeling and support relationship inference. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, Honolulu. IEEE (2017)
Baier, S., Ma, Y., Tresp, V.: Improving visual relationship detection using semantic modeling of scene descriptions. In: Proceedings of the International Conference on Semantic Web, Vienna, pp. 53–68. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Fan, Y., Fan, L., Yang, J. (2018). Improving Semantic Annotation Using Semantic Modeling of Knowledge Embedding. In: Sun, X., Pan, Z., Bertino, E. (eds) Cloud Computing and Security. ICCCS 2018. Lecture Notes in Computer Science(), vol 11068. Springer, Cham. https://doi.org/10.1007/978-3-030-00021-9_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-00021-9_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00020-2
Online ISBN: 978-3-030-00021-9
eBook Packages: Computer ScienceComputer Science (R0)