Connecting Foundation Models with the Physical World using Reconfigurable Drone Agents
Pages 1745 - 1747
Abstract
Foundation models excel in tasks such as content generation, zero-shot classifications, and reasoning. However, they struggle with sensing, interacting, and actuating in the physical world due to their dependence on limited sensors and actuators in providing timely contextual information or physical interactions. This reliance restricts the system's adaptability and coverage. To address these issues and create an embodied AI with foundation models (FMs), we introduce Embodied Reconfigurable Drone Agent (EmbodiedRDA). EmbodiedRDA features a custom drone platform that can autonomously swap payloads to reconfigure itself with a diverse list of sensors and actuators. We designed FM agents to instruct the drone to equip itself with appropriate physical modules, analyze sensor data, make decisions, and control the drone's actions. This enables the system to perform a variety of tasks in dynamic physical environments, bridging the gap between the digital and physical worlds.
References
[1]
2023. Creality3D Ender-3, a fully Open Source 3D printer perfect for new users on a budget. https://github.com/Creality3DPrinting/Ender-3.
[2]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023).
[3]
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023).
[4]
Minghui Zhao, Junxi Xia, Kaiyuan Hou, Yanchen Liu, Stephen Xia, and Xiaofan Jiang. 2024. RASP: A Drone-based Reconfigurable Actuation and Sensing Platform for Engaging Physical Environments with Foundation Models. arXiv:2403.12853 [cs.RO] https://arxiv.org/abs/2403.12853
[5]
Minghui Zhao, Stephen Xia, Jingping Nie, Kaiyuan Hou, Avik Dhupar, and Xiaofan Jiang. 2023. LegoSENSE: An Open and Modular Sensing Platform for Rapidly-Deployable IoT Applications. In Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation. 367--380.
Index Terms
- Connecting Foundation Models with the Physical World using Reconfigurable Drone Agents
Recommendations
Predictive precision of enhanced drone landings
AbstractPrecise drone landing remains a persistent challenge due to the high level of accuracy needed. We propose a novel technique that collects actual drone landing data and employs machine learning algorithms to predict errors in autonomous landing. ...
Highlights- A novel framework for predicting drone landing errors precisely.
- A comprehensive dataset of real drone landings to ascertain influential factors impacting landing accuracy.
- Application of advanced machine learning approaches to ...
Ethical issues with use of drone aircraft
ETHICS '14: Proceedings of the IEEE 2014 International Symposium on Ethics in Engineering, Science, and TechnologyThis analysis is aimed at identifying the ethical issues related to a variety of types of Drone aircraft as they are currently deployed in recreational, civilian/commercial and military arenas. A series of examples will be examined in order to attempt ...
Reference architecture specification for drone systems
Highlights- Reference architecture specification for future drone systems.
- Requirements/...
AbstractUnmanned Aerial Systems (UASs) are useful for performing missions ranging from simple to tedious/dangerous ones. In recent years, UAS architectures have been introduced by different groups with different viewpoints. However, these ...
Comments
Information & Contributors
Information
Published In
December 2024
2476 pages
ISBN:9798400704895
DOI:10.1145/3636534
- Chair:
- Weisong Shi,
- Program Chairs:
- Deepak Ganesan,
- Nicholas Lane
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 December 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
ACM MobiCom '24
Sponsor:
ACM MobiCom '24: 30th Annual International Conference on Mobile Computing and Networking
November 18 - 22, 2024
DC, Washington D.C., USA
Acceptance Rates
Overall Acceptance Rate 440 of 2,972 submissions, 15%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 52Total Downloads
- Downloads (Last 12 months)52
- Downloads (Last 6 weeks)28
Reflects downloads up to 27 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in