Abstract
Recently, mobile robots are attracting attention in various industries. Outdoor unmanned security mobile robot is a key issue for surveillance. The main task of these security robots is to protect people and property. For this purpose, the robot should be able to autonomously navigation and interaction with people is essential. Especially, it is required to perform autonomous driving by avoiding collision with humans and obstacles, tracking a certain human for intruder surveillance or safety of people. For the outdoor security task, we propose the novel localization and control methods that is not only overcome the weather conditions for positioning, but also build a safety route for avoiding and following relative to human position. The robust localization method is based on detecting the salient features by the information filters for multi-layered knowledge (e.g. sensory, episodic, semantic and cloud big data), and then estimate an accurately position of the moved robot. Next, the safety route is defined a rollover model of a security robot on slope and suggests a path generation using DWA (Dynamic Window Approach) method with safety ratio. The method of evaluating rollover is the ZMP (Zero Moment Point) concept. If there is a ZMP between the wheels of the steering mobile robot, it can be safe. The results show that the autonomous navigation is possible with robust localization method, and then it can follow a specific human by the safety path generation. The proposed method is expected to be usable in various applications requiring outdoor surveillance.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Multi-layered knowledge
- Most weather conditions
- Outdoor autonomous navigation
- Rollover model
- Security robot
- Human following robot
1 Introduction
Interaction technology between human and robot has been continuously developed. Among the applications for surveillance, outdoor unmanned security robots have attracted attention in recent years. The robots have to perform autonomous navigation and human interaction in various outdoor environments. First, autonomous driving has been carried out steadily, and many researchers are conducting it. First, autonomous driving has been continuously studied by many researchers. Especially, the navigation techniques for the outdoor environment have been performed mostly for good weather, and it is difficult to cope with most weather conditions. This is because effective feature extraction is difficult in complex weather environments. In order to overcome this problem, robust features are extracted by utilizing all of the multi-modal sensor data available to the robot. Then, there is a need for a method of locating the most useful features using prior knowledge of the current weather, location, and topography.
On the other hand, the security robot is required to interact with humans to perform missions to track or guide interested humans. For this purpose, it is necessary to consider both the control method of measuring and maintaining the distance from the human, the method of driving safely by recognizing the speed and the humility. Therefore, a safe control method is required for rollover that does not overturn while measuring distance.
In this paper, we propose a robust localization method for outdoor environment and a control method for interaction with human. The proposed method can drive outdoors for most weather conditions, and can track and guide humans on the unpaved road or slope way. Experimental results show that robust position estimation is feasible based on knowledge-based features, and a stability model can be defined to enable safe travel with less risk of rollover.
The remainder of paper is organized as follows. In Sect. 2, some related works are summarized. Section 3 describes localization for outdoor autonomous navigation, including multi-layered knowledge based salient map, information filter, and some results. The safety path generation method with ZMP model is described in Sect. 4. Section 5 presents the human robot interaction by following in surveillance application. Finally, we summarize the proposed method.
2 Related Works
Research on the position estimation of robots has been carried out steadily. The Researches based on various sensors estimate a location with a fixed map using vision data [1] or sensor data [2] in indoor environment. However, in order to carry out unmanned security, localization method should be possible in outdoor environment.
There is a method [3] of recognizing the road environment by mixing LiDAR based on GPS, a method [4, 5] using a laser sensor, a method using a single camera [6] or a stereo vision [7] for outdoor localization. These methods determine the location by mixing the global GPS and the local pose estimation using the sensor data. In the same way, there is also research on location recognition using only global and local vision data [8]. However, for practical security, robust driving methods are needed in most weather conditions, including seasonal changes.
In order to overcome the environmental changes, a method of estimating the location using robust features in the image for seasonal changes [9], a method that is useful not only for seasons but also for night and daytime conditions [10, 11], and techniques have been proposed for rain or snow weather conditions [12]. These methods are solving some of the changes in the outdoor environment depending on the feature quantity using a specific image only. Therefore, in this paper, we propose a localization method considering most of season, night/day and weather conditions.
On the other hand, there were various efforts for stable control of robots. However, it seems that most of them are not suitable for roads that have roughness for active steering [13], steering and braking [14, 15], which are mostly controls for flat roads. Therefore, we propose a stable model-based control algorithm by the centrifugal force generated in the turning process in order to prevent the rollover on the road with the slope.
Most of the interaction methods between human and robot were performed based on sensor data in indoor environments [16, 17]. However, in this paper, we propose a human-following interaction method using the above-mentioned location recognition and control method for outdoor unmanned security robot. It performs interactions that follow specific humans for outdoor security applications. Consequently, the proposed interaction method is expected to improve the service of the outdoor unmanned security robot.
3 Localization for Outdoor Autonomous Navigation
3.1 Multi-layered Knowledge Augmented Based Strategy Map
A multi-layered knowledge augmented map database for extracting valid data from multi-modal sensor is shown in Fig. 1. This map is based on a multi-layer including semantic knowledge (semantic data, episodic data, semantic and cloud big data), climate, time, geographical features and driving strategies. The information is used to enhance a dynamic navigation map. Figure 2 shows an example of fusion of valid salient data based strategy map at the current location by query from the multi-layered knowledge augmented DB map. This is a way to represent the most useful data at the current position using a high-level knowledge database. Moreover, each data has a level of robustness and reliability can be measured using it. Therefore, we can choice the salient data which has strong and weak data in the given knowledge data (weather condition, current time, season, temp, RH, etc.).
3.2 Reliable Localization Method
Reliable localization estimation is performed by extracting salient pointers and features with the multi-layer prior knowledge-based information filters. This method is estimated by fusion of non-model based Monte Carlo Localizer [18] and model-based EKF Localizer [19] in a reliability pose. Each localizer considers the knowledge-based robust level to determine its uncertainty. The final pose estimation using this is a reliable result, as shown in Fig. 3.
The proposed method employs an information filter for multi-modal sensor data, the salient points and features can be extracted as,
where Xi is observed data, and Vk denotes the noise vectors. The \( {\text{I}}\left( \cdot \right) \) derived from the information state vector Yk
Therefore, the information filter can be extracted a reliable position from the multi-modal sensor data. Finally, we can estimate the accurate position for security robots.
3.3 Experimental Results for the Proposed Localization Method
In order to show the usefulness of the proposed method, a salient dominant vertical structure was extracted from the heavy rain images. The Fig. 4 shows the results of extracting valid salient features by using information filter. In Fig. 4(a), the image shows a result in the good weather condition. Figure 4(b) is the start image in the heavy rainy day. The third image is a result without filter, as shown in Fig. 4(c). Finally, Fig. 4(d) shows the result image with the proposed information filter (e.g. blurring, lens flare effect repression). Table 1 shows that the salient feature extraction rate improves.
Figure 5 shows the results of applying the proposed method to 3D LiDAR data in a factitious rainy situation. Figure 5(a) and (d) show the start image and final image in a sunny day, and there is no position error. In Fig. 5(b) and (e), the images show that localization error is caused by the raindrops. Finally, the results of our method can reduce the error and obtain a valid location, as shown in Fig. 5(c) and (f). Therefore, the precise location in clear weather loses its accuracy in rainy conditions, but it can be seen that the proposed method is capable of reliable localization by elimination feature error, as shown in Table 2.
4 Safety Path Generation for Human Computer Interaction
4.1 Robot Modeling for Overcoming Rollover
The rollover of the mobile robot is determined by the gravitational force in the ramp and the centrifugal force generated in the turning process of the mobile robot [20]. Figure 1 shows a graphical representation of a turn at a ramp and the resulting force. At this time, the case where the robot is rolled over is expressed by Eq. (3),
where v is velocity of robot, w is angular velocity of robot, θ presents the angle of slope. This equation represents that the rollover is dependent on v, w, and θ. Moreover, in this equation, only the occurrence of rollover is known, and the risk of rollover is unknown. Therefore, it is necessary to quantify the risk of rollover using ZMP (zero moment point). This model is modified to introduce the ZMP concept, as shown in Fig. 7.
The risk of rollover of a robot can be expressed numerically by the concept of ZMP, and if ZMP model exists between the outer and inner wheels of the robot, it can be said to be stable against rollover. Furthermore, this ZMP model was represented, as Eq. (4),
where p is ZMP, even if it is the same size, which the risk of rollover depends on the robots should be normalized, as Eq. (5),
where stable factor s must be between 0 and 1 to be stable, and if it is negative, it can be determined that rollover has been occurred. If the risk of rollover can be determined numerically, a stable path can be created.
The algorithm used in this paper is DWA (dynamic window approach). DWA calculates the cost according to each cost function in the area consisting of robot velocity and angular velocity set, and determines the most suitable robot velocity and angular velocity. This can be expressed by Eq. (6),
Figure 8 shows the experimental result through the algorithm applying the rollover model. The robot is turning in a slope, and the result is the same as the graph. In the graph, Rollover cost (sky blue color) indicates the risk of rollover. The results show that the value was 0.4 or higher when the risk of overturning was not taken into consideration, and the risk of overturning was lowered to 0.25 when the risk of overturning was considered.
5 Human-Robot Interaction by Following
In the security mission, the unmanned robot must perform the role of tracking or guiding the human. In particular, a mission to track a specific human is initiated by a tracking command received from an administrator, extracting a human region from an image obtained from a camera mounted on the robot. To do this, we employ a method for extracting human regions from the camera image and an interaction method for performing mission based on tracking method [21].
5.1 Human-Robot Distance Measurement
In order to perform various missions in the outdoor environment, the robot is equipped with a multimodal sensor module. The mission for extracting and tracking human regions in various camera modules conducts by RGBD cameras. When a tracking command is received from the control tower, the robot measures the distance of the tracking area from the depth image. The area from the center (xi, yi) of the region to the width and height of the region is divided as shown in Fig. 9(a) (green rectangle), and the depth of the region is accumulated in units of 1 m, and then the final distance D is calculated, as shown in Fig. 9(b) and Eq. (7).
5.2 Human-Following Interaction
The interaction of tracking and following the robot human was performed based on the following scenarios. First, when the abnormal situation (extraction of the crowd density based on the threshold) occurs in the fixed agent, the robot receives the abnormality confirmation command from the control tower. Second, the robot moves to the fixed agent where the abnormal situation occurs, identifies the abnormal situation, and transmits the image to the control tower by distinguishing the person who is falling from the person standing. Next, the robot receives the specific human tracking command, and then starts tracking the specified human, as shown in Fig. 10. Finally, the robot carries out the below sequence to perform the human following interactions.
Human Following Interaction Sequence
-
1.
Calculate the straight line distance between the robot and the center coordinates of the human from (xi, yi) and D.
-
2.
Calculate the relative angle between the robot and the human.
-
3.
Tracking Flag is executed when straight line distance is over 1.5 m.
-
4.
Left/Right Flag is executed when relative angle is more than 15°.
-
5.
When the Tracking Flag is executed, a linear velocity tv* is generated in proportion to the straight line distance to the object.
*tv = (tv_gain)*(z_distance) The tv_gain is experimentally determined to be 0.2.
-
6.
When the Left/Right Flag is executed, the rotation speed rv** is generated in proportion to the relative angle with respect to the object.
**rv = (rv_gain)*(d_angle) The rv_gain is experimentally determined to be 0.35.
Figure 11 shows a result image of human following interaction.
6 Conclusion and Future Work
Interactions between robots and humans are very important in outdoor unmanned security missions. Among the interactions, the task of following the human is one of the most active that the robot can perform on the control tower orders. In this paper, we propose a localization method robust to outdoor weather environment and a control method for safe driving in rough load for human following interaction. The results of the localization showed the robustness to the weather, and the control method proved to be safe path generation. Human robot interaction based on these methods showed usefulness to perform a given task, and it was found that robot could actively respond. Therefore, it is expected that the proposed methods can be applied to the security robot more practical.
In the future, we will add a snow removal filter to secure application to most robots by using autonomous navigation based on semantic information and driving techniques for various locomotion. Moreover, we are also developing a human following interaction method for collaborative tracking of multiple robots.
References
Ullah, M.M., Pronobis, A., Caputo, B., Luo, J., Jensfelt, P., Christensen, H.I.: Towards robust place recognition for robot localization, In: International Conference on Robotics and Automation, pp. 530–537. IEEE, USA (2008)
Emidio, D., Francesco, M.: Mobile robot localization using the phase of passive UHF RFID signals. IEEE Trans. Ind. Electro. 61(1), 365–376 (2014)
Jesse, L., Michael, M., Sebastian, T.: Robotics: Science and System, 3rd edn. MIT, Cambridge (2008)
Jose, G., Eduardo, N., Stephan, B.: Localization and map building using laser range sensors in outdoor applications. J. Robot. Syst. 17(10), 565–583 (2000)
Martin, A., Sen, Z., Lhua, X.: Particle filter based outdoor robot localization using natural features extracted from laser scanners. In: Proceedings of the International Conference on Robotics & Automation, pp. 1493–1498. IEEE (2004)
Eric, R., Maxime, L., Michel, D., Jean-Marc, L.: Monocular vision for mobile robot localization and autonomous navigation. Int. J. Comput. Vision 74(3), 237–260 (2007)
Motilal, A., Kurt, K.: Real-time localization in outdoor environments using stereo vision and inexpensive GPS. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR 2006). IEEE (2006)
Christian, W., Hashem, T., Andreas, M., Andreas, Z.: A hybrid approach for vision-based outdoor robot localization using global and local image features. In: Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, pp. 1047–1052 (2007)
Tayyab, N., Luciano, S., Wolfram, B., Cyrill, S.: Robust visual robot localization across seasons using network flows. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Canada, pp. 2564–2570 (2014)
Christoffer, V., Achim, J.L.: SIFT, SURF & seasons: appearance-based Long-term localization in outdoor environments. Robot. Auton. Syst. 58(2), 149–156 (2010)
Michael, M., Eleonora, V., Walter, S., David, C.: Vision-based simultaneous localization and mapping in changing outdoor environments. J. Field Robot. 31(5), 780–802 (2014)
Naoki, A., Yasunari, K., Shogo, Y., Koichi, O.: Development of autonomous mobile robot that can navigate in rainy situations. J. Robot. Mechatron. 28(4), 441–450 (2016)
Robert, B.: Safety, Comfort and Convenience Systems, 3rd edn. Wiley, Hoboken (2006)
Ackermann, J., Odenthal, D.: Advantages of active steering for vehicle dynamics control. In: International Conference on Advances in Vehicle Control and Safety, France (1998)
Brad, S., Tore, H., Anders, R.: Vehicle dynamics control and controller allocation for rollover prevention. In: Proceedings of International conference on control applications, Munich, Germany (2006)
Donato, D.P., Annalisa, M., Grazia, C., Arcangelo, D.: An autonomous mobile robotic system for surveillance of indoor environments. Int. J. Advan. Robot. Syst. 7(1), 19–26 (2010)
Jiale, G., Hong, W., Zhiguo, L., Naishi, F., Fo, H.: Research on human-robot interaction security strategy of movement authorization for service robot based on people’s attention monitoring. In: Proceedings of the International Conference on Intelligence and Safety for Robotics, pp. 521–526. IEEE (2018)
Sebastian, T., Dieter, F., Wolfram, B., Frank, D.: Robust Monte Carlo localization for mobile robots. J. Artif. Intell. 128(1–2), 99–141 (2001)
Ling, C., Huosheng, H., Klaus, M.: EKF based mobile robot localization. In: Proceeding of International Conference on Emerging Security Technologies, Italy, pp. 149–154 (2012)
Ji-Hyun, P., Tae-Young, U., Gi-Deok, B., Young-Ho, C.: Stability evaluation of outdoor unmanned security robot in terrain information. In: Proceedings of 18th International Conference on Control, Automation and Systems, Korea, pp. 955–957 (2018)
Jongwon, C., et al.: Context-aware deep feature compression for high-speed visual tracking. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 478–488. IEEE (2018)
Acknowledgements
This work was supported by the ICT R&D program of IITP, 2017-0-00306. Development of multimodal sensor-based intelligent systems for outdoor surveillance robots.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Uhm, T., Park, JH., Bae, GD., Choi, YH. (2019). Multi-modal Sensor Based Localization and Control Method for Human-Following Outdoor Security Mobile Robot. In: Streitz, N., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. HCII 2019. Lecture Notes in Computer Science(), vol 11587. Springer, Cham. https://doi.org/10.1007/978-3-030-21935-2_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-21935-2_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-21934-5
Online ISBN: 978-3-030-21935-2
eBook Packages: Computer ScienceComputer Science (R0)