Abstract
Deep learning methods have made remarkable breakthroughs in machine learning in general and in automated driving (AD) in particular. However, there are still unsolved problems to guarantee reliability and safety of automated systems, especially to effectively incorporate all available information and knowledge in the driving task. Knowledge graphs (KG) have recently gained significant attention from both industry and academia for applications that benefit by exploiting structured, dynamic, and relational data. The complexity of graph-structured data with complex relationships and inter-dependencies between objects has posed significant challenges to existing machine learning algorithms. However, recent progress in knowledge graph embeddings and graph neural networks allows to applying machine learning to graph-structured data. Therefore, we motivate and discuss the benefit of KGs applied to AD. Then, we survey, analyze and categorize ontologies and KG-based approaches for AD. We discuss current research challenges and propose promising future research directions for KG-based solutions for AD.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arnold, E., Al-Jarrah, O.Y., et al.: A survey on 3d object detection methods for autonomous driving applications. T-ITS 20(10), 3782–3795 (2019)
ASAM: ASAM OpenSCENARIO V2.0 (2022)
ASAM: ASAM OpenX, proposal (2022)
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: DBpedia: a nucleus for a web of open data. In: Aberer, K., et al. (eds.) ASWC/ISWC -2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76298-0_52
Badue, C., Guidolini, R., Carneiro, R.V., et al.: Self-driving cars: a survey. Expert Syst. Appl. 165, 113816 (2021)
Bagschik, G., Menzel, T., Maurer, M.: Ontology based scene creation for the development of automated vehicles. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1813-1820. IEEE (2018)
Bordes, J.B., Davoine, F., Xu, P., Denoeux, T.: Evidential grammars: a compositional approach for scene understanding. application to multimodal street data. Appl. Soft Comput. 61, 1173–1185 (2017)
Buechel, M., Hinz, G., Ruehl, F., et al.: Ontology-based traffic scene modeling, traffic regulations dependent situational awareness and decision-making for automated vehicles. In: IEEE Intelligent Vehicles Symposium, IV (2017)
Burton, S., Gauerhof, L., Heinzemann, C.: Making the case for safety of machine learning in highly automated driving. In: SAFECOMP Workshops (2017)
Cai, H., Zheng, V., Chang, K.: A comprehensive survey of graph embedding: problems, techniques, and applications. In: IEEE TKDE (2018)
Chami, I., Abu-El-Haija, S., Perozzi, B., Ré, C., Murphy, K.: Machine learning on graphs: a model and comprehensive taxonomy. CoRR abs/2005.03675 (2020)
Chang, X., Ren, P., Xu, P., Li, Chen, X., Hauptmann, A.: Scene graphs: a survey of generations and applications. arXiv abs/2104.01111 (2021)
Chen, W., Kloul, L.: An ontology-based approach to generate the advanced driver assistance use cases of highway traffic. In: Proceedings of the 10th IC3K (2018)
Chowdhury, S.N., Wickramarachchi, R., Gad-Elrab, M.H., Stepanova, D., Henson, C.: Towards leveraging commonsense knowledge for autonomous driving. In: ISWC (2021)
Claussmann, L., Revilloud, M., Glaser, S., Gruyer, D.: A study on al-based approaches for high-level decision making in highway autonomous driving. In: SMC (2017)
Dianov, I., Ramírez-Amaro, K., Cheng, G.: Generating compact models for traffic scenarios to estimate driver behavior using semantic reasoning. In: IROS (2015)
Dickmanns, E., Graefe, V.: Dynamic monocular machine vision. Mach. Vis. Appl. (2005)
Dierkes, F., Raaijmakers, M., Schmidt, M., Bouzouraa, M.E., Hofmann, U., Maurer, M.: Towards a multi-hypothesis road representation for automated driving. In: IEEE ITSC (2015)
Elahi, M.F., Luo, X., Tian, R.: A framework for modeling knowledge graphs via processing natural descriptions of vehicle-pedestrian interactions. In: Stephanidis, C., Duffy, V.G., Streitz, N., Konomi, S., Krömker, H. (eds.) HCII 2020. LNCS, vol. 12429, pp. 40–50. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59987-4_4
Fang, F., Yamaguchi, S., Khiat, A.: Ontology-based reasoning approach for long-term behavior prediction of road users. In: IEEE ITSC (2019)
Feld, M., Müller, C.A.: The automotive ontology: managing knowledge inside the vehicle and sharing it between cars. In: AutomotiveUI (2011)
Fuchs, S., Rass, S., Lamprecht, B., Kyamakya, K.: A model for ontology-based scene description for context-aware driver assistance systems. In: ICST AMBI-SYS (2008)
de Gelder, E., et al.: Ontology for scenarios for the assessment of automated vehicles. CoRR abs/2001.11507 (2020)
Geyer, S., Baltzer, M., Franz, B., Hakuli, S., Kauer, M., Kienle, M., et al.: Concept and development of a unified ontology for generating test and use-case catalogues for assisted and automated vehicle guidance. In: IET ITS (2014)
González, D., Pérez, J., Montero, V.M., Nashashibi, F.: A review of motion planning techniques for automated vehicles. In: T-ITS (2016)
Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: IJCNN (2005)
Gouidis, F., Vassiliades, A., Patkos, T., et al.: A review on intelligent object perception methods combining knowledge-based reasoning and machine learning. In: Proceedings of the AAAI-MAKE Symposium (2020)
Goyal, P., Ferrara, E.: Graph embedding techniques, applications, and performance: a survey. Knowl. Based Syst. 151, 78–94 (2018)
Grigorescu, S.M., Trasnea, B., Cocias, T.T., Macesanu, G.: A survey of deep learning techniques for autonomous driving. J. Field Robot. 37(3), 362–386 (2020)
Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., Ling, M.: Scene graph generation with external knowledge and image reconstruction. In: CVPR (2019)
Guha, R.V., Brickley, D., Macbeth, S.: Schema.org: evolution of structured data on the web. Commun. ACM (2016)
Guo, Z., Huang, Y., Hu, X., Wei, H., Zhao, B.: A survey on deep learning based approaches for scene understanding in autonomous driving. Electronics 10(4), 471 (2021)
Gutiérrez, G., Iglesias, J.A., Ordóñez, F.J., Ledezma, A., Sanchis, A.: Agent-based framework for advanced driver assistance systems in urban environments. In: FUSION (2014)
Halilaj, L., Dindorkar, I., Lüttin, J., Rothermel, S.: A knowledge graph-based approach for situation comprehension in driving scenarios. In: Verborgh, R., et al. (eds.) ESWC 2021. LNCS, vol. 12731, pp. 699–716. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77385-4_42
Halilaj, L., Luettin, J., Henson, C., Monka, S.: Knowledge graphs for automated driving. In: IEEE AIKE-Artificial Intelligence and Knowledge Engineering (2022)
Halilaj, L., Luettin, J., Rothermel, S., Arumugam, S.K., Dindorkar, I.: Towards a knowledge graph-based approach for context-aware points-of-interest recommendations. In: ACM/SIGAPP SAC, pp. 1846–1854 (2021)
Henson, C., Schmid, S., Tran, A.T., Karatzoglou, A.: Using a knowledge graph of scenes to enable search of autonomous driving data. In: ISWC (2019)
Herrmann, M., Witt, C., Lake, L., Guneshka, S., Heinzemann, C., Bonarens, F., et al.: Using ontologies for dataset engineering in automotive AI applications. In: 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE) (2022)
Hina, M.D., Thierry, C., Soukane, A., Ramdane-Cherif, A.: Ontological and machine learning approaches for managing driving context in intelligent transportation. In: IC3K (2017)
Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Hogan, A., Blomqvist, E., Cochez, M., et al.: Knowledge graphs. In: ACM Computing Surveys (2021)
Homayounfar, N., Liang, J., Ma, W.C., Fan, J., Wu, X., Urtasun, R.: Dagmapper: learning to map by discovering lane topology. In: ICCV (2019)
Hovi, J., Ichise, R.: Feasibility study: rule generation for ontology-based decision-making systems. In: Wang, X., Lisi, F.A., Xiao, G., Botoeva, E. (eds.) JIST 2019. CCIS, vol. 1157, pp. 88–99. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-3412-6_9
Huang, L., Liang, H., Yu, B., Li, B., Zhu, H.: Ontology-based driving scene modeling, situation assessment and decision making for autonomous vehicles. In: (ACIRS) (2019)
Huang, Y., Chen, Y.: Survey of state-of-art autonomous driving technologies with deep learning. In: IEEE QRS-C (2020)
Hülsen, M., Zöllner, J.M., Weiss, C.: Traffic intersection situation description ontology for advanced driver assistance. IEEE IV Symposium (2011)
Hülsen, M., Zöllner, J.M., Haeberlen, N., Weiss, C.: Asynchronous real-time framework for knowledge-based intersection assistance. In: IEEE ITSC (2011)
ISO: ISO 26262–1:2018: Road vehicles - functional safety (2018)
Janai, J., Güney, F., Behl, A., Geiger, A.: Computer vision for autonomous vehicles: Problems, datasets and state of the art. Found. Trends Comput. Graph. Vis. 12(1–3), 1–308 (2020)
Janowicz, K., Haller, A., Cox, S.J.D., Phuoc, D.L., Lefrançois, M.: SOSA: a lightweight ontology for sensors, observations, samples, and actuators. J. Web Semant. 56, 1–10 (2019)
Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: representation, acquisition and applications. In: IEEE Transactions on Neural Networks and Learning Systems (2021)
Johnson, J., et al.: Image retrieval using scene graphs. In: CVPR (2015)
Kaleeswaran, A., Nordmann, A., Mehdi, A.: Towards integrating ontologies into verification for autonomous driving. In: ISWC Satellites (2019)
Kannan, S., Thangavelu, A., Kalivaradhan, R.: An intelligent driver assistance system (i-das) for vehicle safety modelling using ontology approach. Int. J. UbiComp 1(3), 15–29 (2010)
Kim, J.E., Henson, C., Huang, K., Tran, T.A., Lin, W.Y.: Accelerating road sign ground truth construction with knowledge graph and machine learning. arXiv abs/2012.02672 (2020)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)
Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. 23, 4909–4926 (2022)
Klotz, B., Troncy, R., Wilms, D., Bonnet, C.: Vsso: the vehicle signal and attribute ontology. In: SSN@ISWC (2018)
Kohlhaas, R., Bittner, T., Schamm, T., Zöllner, J.M.: Semantic state space for high-level maneuver planning in structured traffic scenes. In: ITSC (2014)
Kunze, L., Bruls, T., Suleymanov, T., Newman, P.: Reading between the lanes: road layout reconstruction from partially segmented scenes. In: ITSC (2018)
Lateef, F., Ruichek, Y.: Survey on semantic segmentation using deep learning techniques. Neurocomputing 338, 321–348 (2019)
Leroy, J., Gruyer, D., Orfila, O., Faouzi, N.E.E.: Five key components based risk indicators ontology for the modelling and identification of critical interaction between human driven and automated vehicles. In: IFAC (2020)
Li, X., Ying, X., Chuah, M.C.: Grip: graph-based interaction-aware trajectory prediction. In: 2019 IEEE ITSC, pp. 3960–3966 (2019)
Li, Y., Tao, J., Wotawa, F.: Ontology-based test generation for automated and autonomous driving functions. In: IST (2020)
Lilis, Y., Zidianakis, E., Partarakis, N., Antona, M., Stephanidis, C.: Personalizing HMI elements in ADAS using ontology meta-models and rule based reasoning. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2017. LNCS, vol. 10277, pp. 383–401. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58706-6_31
Liu, H., Singh, P.: Conceptnet - a practical commonsense reasoning tool-kit. BT Technol. J. 22, 211–226 (2004). https://doi.org/10.1023/B:BTTJ.0000047600.45421.6d
Liu, L., et al.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128(2), 261–318 (2019). https://doi.org/10.1007/s11263-019-01247-4
Lu, P., Xu, S., Peng, H.: Graph-embedded lane detection. In: IEEE Transactions on Image Processing (2021)
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: a survey. In: IEEE Transactions PAMI (2021)
Mohammad, M.A., Kaloskampis, I., Hicks, Y., Setchi, R.: Ontology-based framework for risk assessment in road scenes using videos. In: International Conference KES (2015)
Monka, S., Halilaj, L., Rettinger, A.: A survey on visual transfer learning using knowledge graphs. Semant. Web 13, 477–510 (2022)
Monka, S., Halilaj, L., Schmid, S., Rettinger, A.: Learning visual models using a knowledge graph as a trainer. In: Hotho, A., et al. (eds.) ISWC 2021. LNCS, vol. 12922, pp. 357–373. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88361-4_21
Morignot, P., Nashashibi, F.: An ontology-based approach to relax traffic regulation for autonomous vehicle assistance. arXiv abs/1212.0768 (2012)
Mozaffari, S., Al-Jarrah, O.Y., Dianati, M., Jennings, P., Mouzakitis, A.: Deep learning-based vehicle behaviour prediction for autonomous driving applications: a review. arXiv abs/1912.11676 (2019)
Narote, S.P., Bhujbal, P.N., Narote, A.S., Dhane, D.M.: A review of recent advances in lane detection and departure warning system. Pattern Recognit. 73, 216–234 (2018)
Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. In: Proceedings of the IEEE (2016)
Paardekooper, J.P., Comi, M., et al.: A hybrid-ai approach for competence assessment of automated driving functions. In: SafeAI@AAAI (2021)
Paden, B., Cáp, M., Yong, S.Z., Yershov, D.S., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. In: IEEE Transactions on Intelligent Vehicles (2016)
Pollard, E., Morignot, P., Nashashibi, F.: An ontology-based model to determine the automation level of an automated vehicle for co-driving. In: Proceedings of the FUSION (2013)
Qiu, H., Ayara, A., Glimm, B.: A knowledge architecture layer for map data in autonomous vehicles. In: ITSC (2020)
Qiu, H., Ayara, A., Glimm, B.: Ontology-based processing of dynamic maps in automated driving. In: KEOD (2020)
Qiu, H., Ayara, A., Glimm, B.: Ontology-based map data quality assurance. In: Verborgh, R., et al. (eds.) ESWC 2021. LNCS, vol. 12731, pp. 73–89. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77385-4_5
Regele, R.: Using ontology-based traffic models for more efficient decision making of autonomous vehicles. In: ICAS (2008)
Ryu, M.W., Cha, S.H.: Context-awareness based driving assistance system for autonomous vehicles. Int. J. Control Autom. 1(1), 153–162 (2018)
Sarwar, S., Zia, S., ul Qayyum, Z., Iqbal, M., Safyan, M., Mumtaz, S., et al.: Context aware ontology-based hybrid intelligent framework for vehicle driver categorization. In: Transactions on Emerging Telecommunications Technologies (2019)
Schafer, F., Kriesten, R., Chrenko, D., Gechter, F.: No need to learn from each other? - potentials of knowledge modeling in autonomous vehicle systems engineering towards new methods in multidisciplinary contexts. In: ICE/ITMC (2017)
Schlenoff, C., Balakirsky, S., Uschold, M., Provine, R., Smith, S.: Using ontologies to aid navigation planning in autonomous vehicles. Knowl. Eng. Rev. 18(3), 243–255 (2003)
Scholtes, M., Westhofen, L., Turner, L.R., Lotto, K., Schuldes, M., Weber, H., et al.: 6-layer model for a structured description and categorization of urban traffic and environment. IEEE Access 9, 59131–59147 (2021)
Schwarting, W., Pierson, A., et al.: Social behavior for autonomous vehicles. In: Proceedings of the National Academy of Sciences, USA (2019)
Singhal, A.: Introducing the knowledge graph: things, not strings. https://blog.google/products/search/introducing-knowledge-graph-things-not/ (2012). 07 May 2021
Spehr, J., Rosebrock, D., Mossau, D., Auer, R., Brosig, S., Wahl, F.: Hierarchical scene understanding for intelligent vehicles. In: 2011 IEEE Intelligent Vehicles Symposium (IV) (2011)
Studer, R., Benjamins, V.R., Fensel, D.: Knowledge engineering: Principles and methods. Data Knowl. Eng. 25(1–2), 161–197 (1998)
Suryawanshi, Y., Qiu, H., Ayara, A., Glimm, B.: An ontological model for map data in automotive systems. In: IEEE AIKE (2019)
Tang, J., Li, S., Liu, P.: A review of lane detection methods based on deep learning. Pattern Recognit. 11, 107623 (2021)
Töpfer, D., Spehr, J., Effertz, J., Stiller, C.: Efficient road scene understanding for intelligent vehicles using compositional hierarchical models. In: T-ITS (2015)
Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., Maurer, M.: Defining and substantiating the terms scene, situation, and scenario for automated driving. In: ITSC (2015)
Ulbrich, S., Nothdurft, T., Maurer, M., Hecker, P.: Graph-based context representation, environment modeling and information aggregation for automated driving. In: IEEE Intelligent Vehicles Symposium Proceedings (2014)
Urbieta, I.R., Nieto, M., García, M., Otaegui, O.: Design and implementation of an ontology for semantic labeling and testing: automotive global ontology (AGO). Appli. Sci. 11(17), 7782 (2021)
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: 6th International Conference on Learning Representations, ICLR (2018)
Venkateshkumar, S., Sridhar, M., Ott, P.: Latent hierarchical part based models for road scene understanding. In: ICCVW (2015)
Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge graph embedding: a survey of approaches and applications. In: IEEE Transactions on Knowledge and Data Engineering (2017)
Werner, S., Rettinger, A., Halilaj, L., Luettin, J.: Embedding Taxonomical, Situational or Sequential Knowledge Graph Context for Recommendation Tasks. In: Further with Knowledge Graphs (2021)
Werner, S., Rettinger, A., Halilaj, L., Lüttin, J.: RETRA: recurrent transformers for learning temporally contextualized knowledge graph embeddings. In: Verborgh, R., et al. (eds.) ESWC 2021. LNCS, vol. 12731, pp. 425–440. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77385-4_25
Westhofen, L., Neurohr, C., Butz, M., Scholtes, M., Schuldes, M.: Using ontologies for the formalization and recognition of criticality for automated driving. IEEE Open J. Intell. Transp. Syst. 3, 519–538 (2022)
Wickramarachchi, R., Henson, C., Sheth, A.: An evaluation of knowledge graph embeddings for autonomous driving data: experience and practice. In: AAAI-MAKE (2020)
Wickramarachchi, R., Henson, C., Sheth, A.: Knowledge-infused learning for entity prediction in driving scenes. Frontiers in Big Data (2021)
Woo, S., Kim, D., Cho, D., Kweon, I.S.: Linknet: relational embedding for scene graph. In: NeurIPS (2018)
Xiong, Z., Dixit, V., Waller, S.: The development of an ontology for driving context modelling and reasoning. In: ITSC (2016)
Zareian, A., Karaman, S., Chang, S.-F.: Bridging knowledge graphs to generate scene graphs. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 606–623. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_36
Zhao, L., Ichise, R., et al., T.Y.: Ontology-based decision making on uncontrolled intersections and narrow roads. In: 2015 IEEE Intelligent Vehicles Symposium (IV) (2015)
Zhao, L., Ichise, R., Liu, Z., Mita, S., Sasaki, Y.: Ontology-based driving decision making: a feasibility study at uncontrolled intersections. In: IEICE (2017)
Zhao, L., Ichise, R., Mita, S., Sasaki, Y.: Core ontologies for safe autonomous driving. In: ISWC (2015)
Zhu, H., Yuen, K., Mihaylova, L., Leung, H.: Overview of environment perception for intelligent vehicles. In: T-ITS (2017)
Zipfl, M., et al.: Relation-based motion prediction using traffic scene graphs. In: IEEE ITSC (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luettin, J., Monka, S., Henson, C., Halilaj, L. (2022). A Survey on Knowledge Graph-Based Methods for Automated Driving. In: Villazón-Terrazas, B., Ortiz-Rodriguez, F., Tiwari, S., Sicilia, MA., Martín-Moncunill, D. (eds) Knowledge Graphs and Semantic Web . KGSWC 2022. Communications in Computer and Information Science, vol 1686. Springer, Cham. https://doi.org/10.1007/978-3-031-21422-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-21422-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21421-9
Online ISBN: 978-3-031-21422-6
eBook Packages: Computer ScienceComputer Science (R0)