Abstract
This paper introduces the path planning of a 1 cm3 mobile microrobot that is designed for microassembly in a microfactory. Since the conventional path planning method can not achieve high microassembly positioning accuracy, a supervised learning assisted reinforcement learning (SL-RL) method has been developed. In this mixed learning method, the reinforcement learning (RL) is used to search a movement path in the normal learning area. But when the microrobot moves into the buffer area, the supervised learning (SL) is employed to prevent it from moving out of the boundary. The SL-RL uses a gradient descent algorithm based on uniform grid tile coding under SARSA(λ) to handle the large learning state space. In addition to the uniform grid tile model, two irregular tile models called an uneven grid tile model and a cobweb tile model are designed to partition the microrobot state space. The main conclusions demonstrated by simulations are as follows: First, the SL-RL method achieves higher positioning accuracy than the conventional path planning method; second, the SL-RL method achieves higher positioning accuracy and learning efficiency than the single RL method; and third, the irregular tile models show higher learning efficiency than the uniform tile model. The cobweb tile model performs especially well.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
McLurkin JD (1996) Using cooperative robots for explosive ordnance disposal. Technical Document. Massachusetts Institute of Technology, Artificial Intelligence Laboratory. Cambridge, MA, USA
Byrne RH, Adkins DR, Eskridge SE, Harrington JJ, Heller EJ, Hurtado JE (2001) Miniature mobile robots for plume tracking and source localization research. J Micromechatron 1(3):253–261
Caprari G, Estier T, Siegwart R (2002) Fascination of down scaling: Alice the sugar cube robot. J Micromechatron 1(3):177–189
Dario P, Carrozza MC, Stefanini C, Attanasio SD (1998) A mobile microrobot actuated by a new electromagnetic wobble micromotor. IEEE/ASME Trans Mech 3(1):9–16
Estana R, Seyfried J, Schmoeckel F, Thiel M, Buerkle A, Woern H (2004) Exploring the micro- and nanoworld with cubic centimetre-sized autonomous microrobots. Ind Robot: Int J 31(2):159–178
Driesen W, Varidel T, Mazerolle S, Bergander A, Breguet JM (2005) Flexible micromanipulation platform based on tethered cm3-sized mobile micro robots. In: IEEE international conference on robotics and biomimetics, Hong Kong, China, pp 145–150
Martel S (2005) Fundamental principles and issues of highspeed piezoactuated three-legged motion for miniature robots designed for nanometer-scale operations. Int J Robot Res 24(7):575–588
Kortschack A, Shirinov A, Trüper T, Fatikow S (2005) Development of mobile versatile nanohandling microrobots: design, driving principles, haptic control. Robotica 23(4):419–434
Breguet JM, Pernette E, Clavel R (1996) Stick and slip actuators and parallel architectures dedicated to microrobotics. In: Sulzmann A (ed) Proceedings of the society of photo-optical instrumentation engineers (SPIE), vol 2906. SPIE, pp 13–24
Li JH, Li ZB, Chen JP (2007) An omni-directional mobile millimeter-sized microrobot with 3-mm electromagnetic micromotors for a micro-factory. Adv Robot 21(12):1369–1391
Li JH, Li ZB, Chen JP (2008) A microstep control approach for a millimeter-sized omni-directional mobile microrobot actuated by 3-mm-electromagnetic micromotors. Robotica. doi:10.1017/S0263574708005158
Sutton RS (1996) Generalization in reinforcement learning: successful examples using sparse coarse coding. In: Touretzky DS, Mozer MC, Hasselmo ME (eds) Advances in neural information processing systems, vol 8. MIT Press, Cambridge, pp 1038–1044
Maravall D, de Lope J, Martín HJA (2009) Hybridizing evolutionary computation and reinforcement learning for the design of almost universal controllers for autonomous robots. Neurocomputing 72(4–6):887–894
Kondo T, Ito K (2004) A reinforcement learning with evolutionary state recruitment strategy for autonomous mobile robots control. Robot Auton Syst 46(2):111–124
Duan Y, Liu Q, Xu X (2007) Application of reinforcement learning in robot soccer. Eng Appl Artif Intell 20(7):936–950
Mabu S, Hatakeyamay H, Thu MT, Hirasawa K, Hu J (2006) Genetic network programming with reinforcement learning and its application to making mobile robot behavior. IEEJ Trans Electron, Inf Syst 126(8):1009–1015
Erus G, Polat F (2007) A layered approach to learning coordination knowledge in multiagent environments. Appl Intell 27(3):249–267
Baldassarre G (2003) Forward and bidirectional planning based on reinforcement learning and neural networks in a simulated robot. In: Lecture notes in artificial intelligence, vol 2684. Springer, Berlin, pp 179–200
Chohra A, Benmehrez C, Farah A (1998) Neural navigation approach for intelligent autonomous vehicles (IAV) in partially structured environments. Appl Intell 8(3):219–233
Ye C, Yung NHC, Wang DW (2003) A fuzzy controller with supervised learning assisted reinforcement learning algorithm for obstacle avoidance. IEEE Trans Syst Man Cybern Part B—Cybernetics 33(1):17–27
Tan KC, Chen YJ, Wang LF, Liu DK (2005) Intelligent sensor fusion and learning for autonomous robot navigation. Appl Artif Intell 19(5):433–456
Cicirelli G, D’Orazio T, Distante A (2005) Different learning methodologies for vision-based navigation behaviors. Int J Pattern Recognit Artif Intell 19(8):949–975
Kurozumi R, Fujisawa S, Yamamoto T, Suita Y (2002) Path planning for mobile robots using an improved reinforcement learning scheme. In: Proceedings of the 41st SICE annual conference, vol 4. IEEE Press, New York, pp 2178–2183
Wang ZM, Yue H (2005) Mobile robot local path planning based on Q reinforcement learning and CMAC. In: Proceedings of the 24th Chinese control conference, vols 1–2. South China Univ Technology Press, China, pp 1494–1496 (In Chinese)
Albus JS (1981) Brains, behavior, and robotics. McGraw-Hill, New York
Li JH, Li ZB, Chen JP (2008) Reinforcement learning based precise positioning method for a millimeters-sized omnidirectional mobile microrobot. In: Xiong CH, Liu HH, Huang YG, Xiong YL (eds) Lecture notes in artificial intelligence, vol 5314. Springer, Berlin, pp 943–952
Siegwart R, Nourbakhsh IR (2004) Introduction to autonomous mobile robots. MIT Press, Cambridge
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
Doya K (2000) Reinforcement learning in continuous time and space. Neural Comput 12(1):219–245
Stone P, Sutton RS, Kuhlmann G (2005) Reinforcement learning for RoboCup-soccer keepaway. Adapt Behav 13(3):165–188
Stone P, Sutton RS (2001) Scaling reinforcement learning toward RoboCup soccer. In: Brodley CE, Danyluk AP (eds) Proceedings of the eighteenth international conference on machine learning. Morgan Kaufmann, San Francisco, pp 537–544
Tokarchuk L, Bigham J, Cuthbert L (2006) Fuzzy and tile coding function approximation in agent coevolution. In: Deved V (ed) Proceedings of the 24th IASTED international conference on artificial intelligence and applications. ACTA Press, Anaheim, pp 353–358
Bucak IO, Zohdy MA (2001) Reinforcement learning control of nonlinear multi-link system. Eng Appl Artif Intell 14(5):563–575
Lee ISK, Lau HYK (2004) Adaptive state space partitioning for reinforcement learning. Eng Appl Artif Intell 17(6):577–588
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Li, J., Li, Z. & Chen, J. Microassembly path planning using reinforcement learning for improving positioning accuracy of a 1 cm3 omni-directional mobile microrobot. Appl Intell 34, 211–225 (2011). https://doi.org/10.1007/s10489-009-0191-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-009-0191-x