Quadrocopter control using an on-board video system with off-board processing

https://doi.org/10.1016/j.robot.2011.10.009Get rights and content

Abstract

In recent years, Unmanned Aerial Vehicles (UAVs) have gained increasing popularity. These vehicles are employed in many applications, from military operations to civilian tasks. One of the main fields of UAV research is the vehicle positioning problem. Fully autonomous vehicles are required to be as self-sustained as possible in terms of external sensors. To achieve this in situations where the global positioning system (GPS) does not function, computer vision can be used. This paper presents an implementation of computer vision to hold a quadrotor aircraft in a stable hovering position using a low-cost, consumer-grade, video system. The successful implementation of this system required the development of a data-fusion algorithm that uses both inertial sensors and visual system measurements for the purpose of positioning. The system design is unique in its ability to successfully handle missing and considerably delayed video system data. Finally, a control algorithm was implemented and the whole system was tested experimentally. The results suggest the successful continuation of research in this field.

Highlights

► Successful handling of missing and considerably delayed video system data with the application in robust visual servo control. ► On-board Kalman filtering with reduced computational requirements. ► Sensor fusion of the variably delayed sensor measurements. ► Experiment-proven implementation on a X-3D quadrocopter.

Introduction

A quadrotor is a four-rotor helicopter. The idea of using four rotors is not new, as a full-scale, four-rotor helicopter was built by De Bothezat as early as 1921. However, quadrocopters are dynamically unstable and therefore suitable control methods are required to make them stable [1]. These control approaches normally use two control loops—a high-speed, inner loop that controls the helicopter’s attitude based on the outputs from the IMU (Inertial Measurement Unit) in a strap-down configuration [2] and a slower, outer loop that controls the helicopter’s position. While the attitude can be easily determined by measuring the acceleration due to the Earth’s gravitational field and the rotational velocities, there are no universal positioning systems available. Outdoors, the GPS (Global Positioning System) that relies on receiving its signal from satellites can be used.

However, indoors, controlled objects must rely either on beacon-based systems that use RF (Radio Frequency) waves (or a combination of RF and Ultrasonic waves [3]), a combination of sensors and SLAM (simultaneous localization and mapping) algorithms [4], [5], [6] or a visual system that usually incorporates one or more video cameras. Two different video-camera arrangements are possible—a video camera fixed to an object [7], [8], [9], [10], [11], [12]; a single video camera fixed in the object’s environment [13] or a constellation of multiple video cameras, such as Vicon visual tracking system, which is often used in the research on the quadrocopter control [14], [15], [16], [17].

However, real autonomous systems are required to be independent of those external sensors, but are limited by the weight of the sensors and the processing circuitry. In [18], the authors fixed a mobile phone to a quadrocopter and used its processor for localization during the flight, but there was no closed-loop control. In 2010, the company Parrot presented the AR.Drone [19], a self-stabilizing quadrocopter, controlled remotely via a phone or a computer. It combines a fast, down-facing camera, an IMU unit, an ultrasonic distance sensor and a fast signal processor to achieve its stable flight. The self-stabilizing nature makes it an attractive platform for experiments dealing with visual-based control [20]. A similar approach uses wireless transmitters to transmit the video data to an off-board computer, where image-analysis is executed and the results are passed back to the vehicle [21], [22], [23], [24]. Our approach uses the latter configuration, but a great deal of care is taken with the inherent delays and the outages of the video-camera data with such a system.

To solve the issues with the delay and lost data, sensor-fusion methods in combination with a Kalman-filtering technique can be used [25]. In [26] different methods to solve the problem of delayed measurements are described. One such method involves extrapolating delayed measurements to the present time and still obtaining the optimality of the filter, while the other suggests updating the covariance matrix and the state at a different time. Similarly, our method uses past information about the system states, such that the states are directly compared to the measurements, but the Kalman innovation is then used in the present time to correct the states. The system uses the on-board, 6-axis, IMU unit and the image-based position and velocity data received via the wireless link. This unit measures the rotational velocities around the principal axes, x,y and z, and the accelerations in the inertial reference frame. The accelerations are translated into the base reference frame (fixed to the target on the ground) and used for the prediction stage of the Kalman filter. In order to boost the performance of the filter implementation on a small microcontroller, a linear steady-state variant of Kalman filter was designed with minimal number of operations over matrices. Additionally, the Kalman filter correction stage with the delay-cancelling technique is executed only when the new image-based position data is received, while the prediction stage of the Kalman filter is calculated every 10 ms on the on-board computer, making this system resilient to short (in the range of a few seconds) communication interruptions.

This paper presents a slightly modified approach to visual servo-control that provides promising results. The initial analysis of the quadrocopter’s dynamics is provided; this serves as a basis for the Kalman-filtering technique described later on. The innovative system for delay approximation is then presented, which complements the initial Kalman filter. Finally, the visual servo-control was evaluated during the stationary flight of the quadrocopter and the results of the experiment are presented.

Section snippets

The description of the quadrotor system

The proposed approach was implemented on a quadrotor helicopter X-3D-BL (quadrocopter), originally presented in [14]. The X-3D-BL is a commercial product and serves as a testbed for many experiments ([15], [7], [18], [27], [16] and others). The quadrocopter (shown in Fig. 1) has a classical four rotor design with two counter rotating pairs of propellers arranged in a square and connected to the cross of the diagonals. The controller board, including sensors, is mounted in the middle of the

Computer-vision system

Instead of relying on external positioning systems, our study is aimed at using the video camera as the primary sensor for positioning purposes. To avoid putting a powerful and heavy computer on-board the aircraft, video is wirelessly transferred to a standard personal computer, running our custom video-recognition software. The software, developed in Visual C# using the combination of AForge.NET and Emgu CV frameworks for image processing and glyph Recognition library GRATF [28] for glyph

Position estimation

Position information, produced by the image recognition, is subjected to delays and signal outages before it reaches the control input of the quadrocopter. One major drawback of direct visual servoing is the need for the target to stay inside the field of view of the camera. In the case of the quadrocopter under direct visual servo control, one must be aware of the unstable nature of this aircraft. This nature requires a functioning visual servoing loop, which is directly dependent on the

Controller design

Because of its simple structure and robustness, a state-space controller was selected for the x- and y-axis, and because of the strong integrating part, required for the z-axis controller, a PI-D controller was selected for the z-axis. A problem we encountered during the controller design phase, i.e., that the manufacturer of the quadrocopter does not provide any information on the internal, primary, closed-loop controller that is used to follow the tilt and thrust commands. Parametric

The experimental results

The complete system (illustrated in Fig. 2) was evaluated by two-part experiment. The first part of the experiment was focused on the quadrocopter responses during the reference-point changes. From the initial position at (0 cm, 0 cm, 70 cm) in the T coordinate system, the quadrocopter was instructed to move to the position (0 cm, −30 cm, 80 cm) and then return to the starting position. The move was made twice and the results are shown in Fig. 10. The results indicate that the quadrocopter

Conclusion

In this paper, the research results using Kalman filtering to fuse the highly-delayed, low-frequency, sensor measurements with no-delayed high-frequency measurements are presented. To accomplish this task some novel approaches were taken both in the visual recognition system and in the Kalman filtering. The solution was proven in multiple experiments where the quadrocopter autonomous hover mode was tested. The task was especially challenging as the solution is composed of software running on a

Matevž Bošnak graduated in 2009 at the Faculty of Electrical Engineering, University of Ljubljana. He now works as a Ph.D. student in the Laboratory of Autonomous Mobile Systems at the same faculty.

References (38)

  • M. Blösch, S. Weiss, D. Scaramuzza, R. Siegwart, Vision based MAV navigation in unknown and unstructured environments,...
  • K.E. Wenzel et al.

    Low-cost visual tracking of a landing place and hovering flight control with a microcontroller

    Journal of Intelligent and Robotic Systems

    (2009)
  • N. Guenard et al.

    A practical visual servo control for an unmanned aerial vehicle

    IEEE Transactions on Robotics

    (2008)
  • D.-G. Sim et al.

    Integrated position estimation using aerial image sequences

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2002)
  • S. Hutchinson et al.

    A tutorial on visual servo control

    IEEE Transactions on Robotics and Automation

    (1996)
  • G. Chatterji et al.

    GPS/machine vision navigation system for aircraft

    IEEE Transactions on Aerospace and Electronic Systems

    (1997)
  • M. Tarhan et al.

    EKF based attitude estimation and stabilization of a quadrotor UAV using vanishing points in catadioptric images

    Journal of Intelligent and Robotic Systems

    (2011)
  • J. Kim et al.

    Accurate modeling and robust hovering control for a quadrotor VTOL aircraft

    Journal of Intelligent and Robotic Systems

    (2010)
  • D. Gurdan, J. Stumpf, M. Achtelik, K.-M. Doth, G. Hirzinger, D. Rus, Energy-efficient autonomous four-rotor flying...
  • Cited by (52)

    • Exergoenvironmental, environmental impact and damage cost analyses of a micro turbojet engine (m-TJE)

      2022, Energy Reports
      Citation Excerpt :

      The low cost and more efficient UAV design have become more critical with the increase in its performance, flight characteristics, and load carries. Recent growth in the demand for this type of UAV has led to the development of many different kinds of UAVs (Bošnak et al., 2012; Yali et al., 2012; Hoffmann et al., 2011; Romero et al., 2009). Besides, the propulsion technology of UAVs is one of the most critical developments in aviation.

    • Discrete time control based in neural networks for pendulums

      2018, Applied Soft Computing Journal
      Citation Excerpt :

      There is some research about the online control for the trajectory tracking of nonlinear processes. In [2,4,14], PID controllers are addressed. Fuzzy PID controllers are designed in [1,12,37].

    • On a translated frame-based approach to geometric modeling of robots

      2017, Robotics and Autonomous Systems
      Citation Excerpt :

      The reference frame definition, i.e. the choice of the position of its origin and the orientation of its axes, is a subjective decision. For a robotic structure, viewed as a chain of rigid bodies (elements) linked by revolute or prismatic joints, where reference frames are attached to each element, the mentioned definitions are influenced by mathematical reasons [8–13]. The geometric model of the robot is based on the methodology of the frames definitions, which imply mathematical transformations.

    • Adaptive RBFNNs/integral sliding mode control for a quadrotor aircraft

      2016, Neurocomputing
      Citation Excerpt :

      Quadrotor UAVs have received increasing attentions in the past decade, because of their specific characteristics such as autonomous flight, low cost, vertical takeoff/landing ability and onboard vision system [1,2], and their wide applications like surveillance, building exploration and information collection [3,4].

    View all citing articles on Scopus

    Matevž Bošnak graduated in 2009 at the Faculty of Electrical Engineering, University of Ljubljana. He now works as a Ph.D. student in the Laboratory of Autonomous Mobile Systems at the same faculty.

    Drago Matko received the B.Sc., M.Sc. and Ph.D. degrees in electrical engineering in 1971, 1973 and 1977 respectively, from the University of Ljubljana, Slovenia for work in the field of adaptive control systems. He visited the Institute of Space and Astronautical Science in Sagamihara Kanagawa, Japan, as a Foreign Research Fellow for 9 months in 1995–96 and for 6 months in 2003–04. He received an award from the Slovenian ministry for research and technology for his work in the field of computer-aided design of control systems in 1989 and the Zois award for achievements in science in 2003.

    Sašo Blažič received the B.Sc., M.Sc., and Ph.D. degrees in 1996, 1999, and 2002, respectively, from the Faculty of Electrical Engineering, University of Ljubljana. Currently, he is an Associate Professor at the same faculty. The research interests of Sašo Blažič include the adaptive, fuzzy and predictive control of dynamical systems and the modeling of nonlinear systems. Lately, the focus of his research is in the area of autonomous mobile systems.

    View full text