skip to main content
10.1145/3610419.3610494acmotherconferencesArticle/Chapter ViewAbstractPublication PagesairConference Proceedingsconference-collections
research-article

A Study on Reinforcement Learning based Control of Quadcopter with a Cable-suspended Payload

Published: 02 November 2023 Publication History

Abstract

Flying a drone is as simple as playing a video game. However, the suspension of the payload underneath complicates its dynamic behavior and makes control challenging. The default onboard control algorithms are not designed to cope with the unknown interaction introduced by the suspended payload, i.e., the payload’s oscillations. Attempts have been made previously using model-based adaptive control techniques to solve this problem. Another way of addressing this problem is using data-driven control techniques such as Reinforcement Learning (RL). RL techniques have been proven to perform well in modeling complex, coupled, and unknown dynamics. This work discusses a study of implementing the RL based controller for manual flying of the quadcopter with a cable-suspended payload system. The simulations are carried out in a specially designed physics environment that simulates the dynamical behavior of a quadcopter-payload system. The RL agent is trained using the proximal policy optimization approach, and numerous simulations are run to ensure that performance is as expected. Finally, the process of putting the provided controller into actual hardware is covered along with any potential difficulties.

References

[1]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. https://doi.org/10.48550/ARXIV.1606.01540
[2]
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. 2022. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems 5 (2022), 411–444.
[3]
Francesco Bullo and Andrew D Lewis. 2019. Geometric control of mechanical systems: modeling, analysis, and design for simple mechanical control systems. Vol. 49. Springer.
[4]
Peter I Corke and Oussama Khatib. 2011. Robotics, vision and control: fundamental algorithms in MATLAB. Vol. 73. Springer.
[5]
S. Dai, T. Lee, and D. S. Bernstein. 2014. Adaptive control of a quadrotor UAV transporting a cable-suspended load with unknown mass. In 53rd IEEE Conference on Decision and Control. IEEE, 6149–6154.
[6]
Aleksandra Faust, Ivana Palunko, Patricio Cruz, Rafael Fierro, and Lydia Tapia. 2013. Learning swing-free trajectories for UAVs with a suspended load. In 2013 IEEE International Conference on Robotics and Automation. 4902–4909. https://doi.org/10.1109/ICRA.2013.6631277
[7]
Aleksandra Faust, Ivana Palunko, Patricio Cruz, Rafael Fierro, and Lydia Tapia. 2017. Automated aerial suspended cargo delivery through reinforcement learning. Artificial Intelligence 247 (2017), 381–398.
[8]
Philipp Foehn, Davide Falanga, Naveen Kuppuswamy, Russ Tedrake, and Davide Scaramuzza. 2017. Fast Trajectory Optimization for Agile Quadrotor Maneuvers with a Cable-Suspended Payload. In Robotics: Science and Systems. 1–10.
[9]
Michael Gassner, Titus Cieslewski, and Davide Scaramuzza. 2017. Dynamic collaboration without communication: Vision-based cable-suspended load transport with two quadrotors. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, IEEE, 5196–5202.
[10]
Farhad A Goodarzi and Taeyoung Lee. 2016. Stabilization of a rigid body payload with multiple cooperative quadrotors. Journal of Dynamic Systems, Measurement, and Control 138, 12 (2016), 121001.
[11]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. PMLR, 1861–1870.
[12]
Jemin Hwangbo, Inkyu Sa, Roland Siegwart, and Marco Hutter. 2017. Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters 2, 4 (2017), 2096–2103.
[13]
Taeyoung Lee, Melvin Leok, and N. Harris McClamroch. 2010. Geometric tracking control of a quadrotor UAV on SE(3). In 49th IEEE Conference on Decision and Control (CDC). IEEE, 5420–5425. https://doi.org/10.1109/CDC.2010.5717652
[14]
Taeyoung Lee, Koushil Sreenath, and Vijay Kumar. 2013. Geometric control of cooperating multiple quadrotor UAVs with a suspended payload. In 52nd IEEE conference on decision and control. IEEE, 5510–5515.
[15]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
[16]
N. Michael, D. Mellinger, Q. Lindsey, and V. Kumar. 2010. The GRASP Multiple Micro-UAV Testbed. IEEE Robotics Automation Magazine 17, 3 (2010), 56–65.
[17]
Pratik Prajapati, Sagar Parekh, and Vineet Vashista. 2021. On-board cable attitude measurement and controller for outdoor aerial transportation. Robotica (2021), 1–15. https://doi.org/10.1017/S0263574721001302
[18]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
[19]
K. Sreenath, T. Lee, and V. Kumar. 2013. Geometric control and differential flatness of a quadrotor UAV with a cable-suspended load. In 52nd IEEE Conference on Decision and Control. 2269–2274.
[20]
Koushil Sreenath, Nathan Michael, and Vijay Kumar. 2013. Trajectory generation and control of a quadrotor with a cable-suspended load-a differentially-flat hybrid system. In 2013 IEEE International Conference on Robotics and Automation. IEEE, 4888–4895.
[21]
Sarah Tang and Vijay Kumar. 2015. Mixed integer quadratic program trajectory generation for a quadrotor with a cable-suspended payload. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, IEEE, 2216–2222.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
AIR '23: Proceedings of the 2023 6th International Conference on Advances in Robotics
July 2023
583 pages
ISBN:9781450399807
DOI:10.1145/3610419
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Cable-suspended payload
  2. Quadcopters
  3. Reinforcement Learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

AIR 2023

Acceptance Rates

Overall Acceptance Rate 69 of 140 submissions, 49%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 68
    Total Downloads
  • Downloads (Last 12 months)49
  • Downloads (Last 6 weeks)5
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media