Abstract
The objective of this paper is to present a neuro-symbolic AI based technique to represent field-medicine knowledge, referred as to TON-ViT. TON-ViT integrates a Deep Learning Model with an explicit symbolic manipulation, a task graph. This task graph describes the steps of each trauma resuscitation as denoted by a verb and noun pair. Through this representation, symbolic processing and manipulation on task graphs, we can find stereotypical procedures, regardless of style of the performer. Furthermore, we can use this technique to find differences in styles, errors, shortcuts and generate procedures never seen before. When used in combination with a transformer, it can help recognize actions in egocentric vision datasets. Last, through symbolic manipulations on the graph, it is possible to generate medical knowledge which the model has not seen before. We present preliminary results after testing the TON-ViT with the Trauma Thompson Dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Fensel, D., et al.: Introduction: what is a knowledge graph? In: Knowledge Graphs, pp. 1–10. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37439-6_1
Zou, X.: A survey on application of knowledge graph. J. Phys. Conf. Ser. 1487(1), 012016 (2020). https://doi.org/10.1088/1742-6596/1487/1/012016
Yang, S., Zou, L., Wang, Z., Yan, J., Wen, J.-R.: Efficiently answering technical questions—a knowledge graph approach. Proc. AAAI Conf. Artif. Intell. 31(1),(2017). https://doi.org/10.1609/aaai.v31i1.10956
Liew, C.Y., Labadin, J., Kok, W.C., Eze, M.O.: A methodology framework for bipartite network modeling. Appl. Netw. Sci. 8(1), 6 (2023). https://doi.org/10.1007/s41109-023-00533-y
Li, Z., et al.: Temporal knowledge graph reasoning based on evolutional representation learning. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 408–417. ACM (2021). ISBN 978-1-4503-8037-9. https://doi.org/10.1145/3404835.3462963.
Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. IEEE Trans. Knowl. Data Eng. 34(8), 3549–3568 (2022). https://doi.org/10.1109/TKDE.2020.3028705
Wang, X., et al.: Learning intents behind interactions with knowledge graph for recommendation. In: Proceedings of the Web Conference 2021, pp. 878–887. ACM (2021). ISBN 978-1-4503-8312-7. https://doi.org/10.1145/3442381.3450133
Chen, H., Luo, X.: An automatic literature knowledge graph and reasoning network modeling framework based on ontology and natural language processing. Adv. Eng. Informatics 42, 100959 (2019). https://doi.org/10.1016/j.aei.2019.100959
Peña, J., Rochat, Y.: Bipartite graphs as models of population structures in evolutionary multiplayer games. PLoS ONE 7(9), e44514 (2012). https://doi.org/10.1371/journal.pone.0044514
Paulius, D., Huang, Y., Milton, R., Buchanan, W.D., Sam, J., Sun, Y.: Functional object-oriented network for manipulation learning. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2655–2662. https://doi.org/10.1109/IROS.2016.7759413.
Tiddi, I., Schlobach, S.: Knowledge graphs as tools for explainable machine learning: a survey. Artif. Intell. 302, 103627 (2022). https://doi.org/10.1016/j.artint.2021.103627
Marino, K., Salakhutdinov, R., Gupta, A.: The more you know: using knowledge graphs for image classification. arXiv preprint arXiv:1612.04844 (2017)
Paulheim, H.: Knowledge graph refinement: a survey of approaches and evaluation methods. Semantic Web 8(3), 489–508 (2016). https://doi.org/10.3233/SW-160218
Chaudhri, V.K., et al.: Knowledge graphs: introduction, history, and perspectives. AI Magazine 43(1), 17–29 (2022). https://doi.org/10.1002/aaai.12033
Lecue, F.: On the role of knowledge graphs in explainable AI. Semantic Web 11(1), 41–51 (2020). https://doi.org/10.3233/SW-190374
Zhang, W., Paudel, B., Zhang, W., Bernstein, A., Chen, H.: Interaction embeddings for prediction and explanation in knowledge graphs. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 96–104. ACM (2019). ISBN 978-1-4503-5940-5. https://doi.org/10.1145/3289600.3291014
Manghani, K.: Quality assurance: importance of systems and standard operating procedures. Perspect. Clin. Res. 2(1), 34 (2011). https://doi.org/10.4103/2229-3485.76288
Hitzler, P., Eberhart, A., Ebrahimi, M., Sarker, M.K., Zhou, L.: Neuro-symbolic approaches in artificial intelligence. Natl. Sci. Rev. 9(6), nwac035 (2022). https://doi.org/10.1093/nsr/nwac035
Hitzler, P.: Some advances regarding ontologies and neuro-symbolic artificial intelligence. In: Brazdil, P., van Rijn, J.N., Gouk, H., Mohr, F. (eds.) ECMLPKDD Workshop on Meta-Knowledge Transfer, volume 191 of Proceedings of Machine Learning Research, pp. 8–10. PMLR (2022). www.proceedings.mlr.press/v191/hitzler22a.html
Xie, X., Kersting, K., Neider, D.: Neuro-symbolic verification of deep neural networks. arXiv preprint arXiv:2203.00938 (2022)
Hamilton, K., Nayak, A., Božić, B., Longo, L.: Is Neuro-symbolic AI Meeting Its Promises in Natural Language Processing? A Structured Review, pp. 1–42 (2022). https://doi.org/10.3233/SW-223228. www.medra.org/servlet/aliasResolver?alias=iospress &doi=10.3233/SW-223228
Oltramari, A., Francis, J., Henson, C., Ma, K., Wickramarachchi, R.: Neuro-symbolic architectures for context understanding. arXiv preprint arxiv.org/abs/2003.04707 (2020). https://doi.org/10.48550/ARXIV.2003.04707.Publisher: arXiv Version Number: 1
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2021)
Pan, X., Ye, T., Han, D., Song, S., Huang, G.: Contrastive language-image pre-training with knowledge graphs. arXiv preprint arXiv:2210.08901 (2022)
Zhao, H., Torralba, A., Torresani, L., Yan, Z.: HACS: human action clips and segments dataset for recognition and temporal localization. arXiv preprint arXiv:1712.09374 (2019)
Rao, T.S., Radhakrishnan, R., Andrade, C.: Standard operating procedures for clinical practice. Ind. J. Psychiatry 53(1), 1–3 (2011). https://doi.org/10.4103/0019-5545.75542
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Additional information
Disclaimers: The views expressed are those of the author(s) and do not reflect the official policy of the Department of the Army, the Department of Defense, or the U.S. Government. The investigators have adhered to the policies for the protection of human subjects as prescribed in 45 CFR 46.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhuo, Y. et al. (2024). TON-ViT: A Neuro-Symbolic AI Based on Task Oriented Network with a Vision Transformer. In: Waiter, G., Lambrou, T., Leontidis, G., Oren, N., Morris, T., Gordon, S. (eds) Medical Image Understanding and Analysis. MIUA 2023. Lecture Notes in Computer Science, vol 14122. Springer, Cham. https://doi.org/10.1007/978-3-031-48593-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-48593-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48592-3
Online ISBN: 978-3-031-48593-0
eBook Packages: Computer ScienceComputer Science (R0)