skip to main content
10.1145/3678717.3691226acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
short-paper
Open access

Metric Reasoning in Large Language Models

Published: 22 November 2024 Publication History

Abstract

Spatial reasoning is a particularly challenging task that requires inferring implicit information about objects based on their relative positions in space. In an effort to develop general purpose geo-foundation models that can perform a variety of spatial reasoning tasks, preliminary work has explored what kinds of world knowledge and spatial reasoning capabilities Large Language Models (LLMs) naturally inherit from their training data. Recent work suggests that LLMs contain geospatial knowledge in the form of understanding geo-coordinates and associating spatial meaning to the key terms "near" and "far." In this paper, we show that LLMs lack the ability to adapt the meaning of the words "near" and "far" to the appropriate scale when provided contextual reference points. By uncovering biases in how LLMs answer distance-related spatial questions, we set the groundwork for developing new techniques that may enable LLMs to perform accurate spatial reasoning.

References

[1]
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023).
[2]
Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser. 2023. Are Large Language Models Geospatially Knowledgeable?. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems (Hamburg, Germany) (SIGSPATIAL '23). Association for Computing Machinery, New York, NY, USA, Article 75, 4 pages. https://doi.org/10.1145/3589132.3625625
[3]
Anthony G Cohn and Jose Hernandez-Orallo. 2023. Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs. arXiv preprint arXiv:2304.11164 (2023).
[4]
Michael D Lieberman, Hanan Samet, and Jagan Sankaranayananan. 2010. Geotagging: Using proximity, sibling, and prominence clues to understand comma groups. In In Proceedings of the 6th workshop on geographic information retrieval.
[5]
Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. 2023. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798 (2023).
[6]
Peter Mooney, Wencong Cui, Boyuan Guan, and Levente Juhász. 2023. Towards Understanding the Geospatial Skills of ChatGPT: Taking a Geographic Information Systems (GIS) Exam. In Proceedings of the 6th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (Hamburg, Germany) (GeoAI '23). Association for Computing Machinery, New York, NY, USA, 85--94. https://doi.org/10.1145/3615886.3627745
[7]
Kent O'Sullivan, Nicole R. Schneider, Aleeza Rasheed, and Hanan Samet. 2023. GESTALT: Geospatially Enhanced Search with Terrain Augmented Location Targeting. In Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data (Hamburg, Germany) (GeoSearch'23). Association for Computing Machinery.
[8]
Kent O'Sullivan, Nicole R. Schneider, and Hanan Samet. 2023. COMPASS: Cardinal Orientation Manipulation and Pattern-Aware Spatial Search. In Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data (Hamburg, Germany) (GeoSearch'23). Association for Computing Machinery.
[9]
Jianzhong Qi, Zuqing Li, and Egemen Tanin. 2023. MaaSDB: Spatial Databases in the Era of Large Language Models (Vision Paper). In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems (Hamburg, Germany) (SIGSPATIAL '23). Association for Computing Machinery, New York, NY, USA, Article 54, 4 pages. https://doi.org/10.1145/3589132.3625597
[10]
Nicole R. Schneider, Kent O'Sullivan, and Hanan Samet. 2024. Graph-based Spatial Pattern Matching: A Theoretical Comparison. In Proceedings of the 32nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (Atlanta, USA). Association for Computing Machinery.
[11]
Nicole R. Schneider, Kent O'Sullivan, and Hanan Samet. 2024. The Future of Graph-based Spatial Pattern Matching (Vision Paper). In 40th IEEE International Conference on Data Engineering, ICDE 2024 - SEAGraph Workshop. Utrecht, Netherlands.
[12]
Achille C. Varzi. 2007. Spatial Reasoning and Ontology: Parts, Wholes, and Locations. Springer Netherlands, Dordrecht, 945--1038. https://doi.org/10.1007/978-1-4020-5587-4_15
[13]
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128 (2023).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGSPATIAL '24: Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems
October 2024
743 pages
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 November 2024

Check for updates

Author Tags

  1. Large language models
  2. geo-foundation models
  3. metric relations

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Funding Sources

Conference

SIGSPATIAL '24
Sponsor:

Acceptance Rates

SIGSPATIAL '24 Paper Acceptance Rate 37 of 122 submissions, 30%;
Overall Acceptance Rate 257 of 1,238 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 57
    Total Downloads
  • Downloads (Last 12 months)57
  • Downloads (Last 6 weeks)40
Reflects downloads up to 14 Jan 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media