Skip to main content
SpringerLink
Log in

Improving multi-modal data fusion by anomaly detection

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

If we aim for autonomous navigation of a mobile robot, it is crucial and essential to have proper state estimation of its position and orientation. We already designed a multi-modal data fusion algorithm that combines visual, laser-based, inertial, and odometric modalities in order to achieve robust solution to a general localization problem in challenging Urban Search and Rescue environment. Since different sensory modalities are prone to different nature of errors, and their reliability varies vastly as the environment changes dynamically, we investigated further means of improving the localization. The common practice related to the EKF-based solutions such as ours is a standard statistical test of the observations—or of its corresponding filter residuals—performed to reject anomalous data that deteriorate the filter performance. In this paper we show how important it is to treat well visual and laser anomalous residuals, especially in multi-modal data fusion systems where the frequency of incoming observations varies significantly across the modalities. In practice, the most complicated part is to correctly identify the actual anomalies, which are to be rejected, and therefore here lies our major contribution. We go beyond the standard statistical tests by exploring different state-of-the-art machine learning approaches and exploiting our rich dataset that we share with the robotics community. We demonstrate the implications of our research both indoor (with precise reference from a Vicon system) as well as in challenging outdoor environment. In the final, we prove that monitoring the health of the observations in Kalman filtering is something, that is often overlooked, however, it definitively should not be.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Open-source computer vision library http://opencv.org/.

  2. Open-source ICP library https://github.com/ethz-asl/libpointmatcher.

  3. We exploit this difference between predicted and actual measurements in the anomaly detection.

  4. We used the gmdistribution class from Matlab Statistics toolbox and fitted the GMM for number of components equal to \(k = 1,2,3\) with the standard parameters. Maximum of 3 mixtures was selected, because models with more than 3 mixtures usually resulted in negligible weights for some redundant mixtures.

  5. We took linear, polynomial and radial basis function (RBF) kernels in consideration.

  6. We used the LIBSVM tool (version 3.17) (Chang and Lin 2011).

  7. We used the fminunc from Matlab Optimization toolbox for the minimization.

  8. Data collected in a room monitored with twelve cameras covering more than 20 m\(^2\) and giving a few millimeter accuracy at 100 Hz.

  9. The datasets are publicly available at https://sites.google.com/site/kubelvla/public-datasets.

  10. True Positive (TP)—anomaly correctly classified as anomaly; False Negative (FN)—anomaly incorrectly classified as normal; False Positive (FP)—normal data incorrectly classified as anomaly; True Negative (TN)—normal data correctly classified as normal.

  11. These experiments are publicly available as well at https://sites.google.com/site/kubelvla/public-datasets.

  12. The distance measurements are taken in continuous mode at 7.5 Hz with measurement accuracy about 3 mm.

References

  • Agamennoni, G., Nieto, J. I., & Nebot, E. M. (2011). An outlier-robust kalman filter. In IEEE International Conference on Robotics and Automation (ICRA), 2011 (pp. 1551–1558). IEEE.

  • Ali, J., & Ushaq, M. (2009). A consistent and robust Kalman filter design for in-motion alignment of inertial navigation system. Measurement, 42(4), 577–582.

    Article  Google Scholar 

  • BarShalom, Y., Li, X. R., & Kirubarajan, T. (2001). Estimation with applications to tracking and navigation. New York: Wiley.

    Book  Google Scholar 

  • Besl, P., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Borges, G. A., & Aldon, M. J. (2003). Robustified estimation algorithms for mobile robot localization based on geometrical environment maps. Robotics and Autonomous Systems, 45(3), 131–159.

    Article  Google Scholar 

  • Brunner, C., Peynot, T., Vidal-Calleja, T., & Underwood, J. (2013). Selective combination of visual and thermal imaging for resilient localization in adverse conditions: Day and night, smoke and fire. Journal of Field Robotics, 30(4), 641–666.

    Article  Google Scholar 

  • Caron, F., Duflos, E., Pomorski, D., & Vanheeghe, P. (2006). Gps/imu data fusion using multisensor Kalman filtering: Introduction of contextual aspects. Information Fusion, 7(2), 221–230.

    Article  Google Scholar 

  • Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys (CSUR), 41(3), 15.

    Article  Google Scholar 

  • Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2, 1–27. http://www.csie.ntu.edu.tw/cjlin/libsvm.

  • Chen, Z. (2003). Bayesian filtering: From Kalman filters to particle filters, and beyond. Statistics, 182(1), 1–69.

    Article  Google Scholar 

  • Christensen, A. L., OGrady, R., Birattari, M., & Dorigo, M. (2008). Fault detection in autonomous robots based on fault injection and learning. Autonomous Robots, 24(1), 49–67.

    Article  Google Scholar 

  • Dissanayake, G., Sukkarieh, S., Nebot, E., & Durrant-Whyte, H. (2001). The aiding of a low-cost strapdown inertial measurement unit using vehicle model constraints for land vehicle applications. IEEE Transactions on Robotics and Automation, 17(5), 731–747.

    Article  Google Scholar 

  • Divis, J. (2013). Visual odometry from omnidirectional camera. Master thesism Charles University, Prague.

  • Dua, S., & Du, X. (2011). Data mining and machine learning in cybersecurity. Boca Raton: Taylor & Francis.

    Book  MATH  Google Scholar 

  • Endo, D., Okada, Y., Nagatani, K., & Yoshida, K. (2007). Path following control for tracked vehicles based on slip-compensating odometry. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems IROS 2007 (pp. 2871–2876).

  • Farrell, J. A. (2008). Aided navigation: GPS with high rate sensors. New York: McGraw-Hill.

    Google Scholar 

  • Fraundorfer, F., & Scaramuzza, D. (2012). Visual odometry: Part II: Matching, robustness, optimization, and applications. IEEE Robotics & Automation Magazine, 19(2), 78–90.

    Article  Google Scholar 

  • Gertler, J. (1998). Fault detection and diagnosis in engineering systems. Boca Raton: CRC Press.

    Google Scholar 

  • Goel, P., Dedeoglu, G., Roumeliotis, S. I., & Sukhatme, G. (2000). Fault detection and identification in a mobile robot using multiple model estimation and neural network. In Proceedings of the IEEE International Conference on Robotics and Automation, 2000, ICRA’00 (Vol. 3, pp. 2302–2309). IEEE.

  • Görner, M., & Stelzer, A. (2013). A leg proprioception based 6 DOF odometry for statically stable walking robots. Autonomous Robots, 34(4), 311–326.

    Article  Google Scholar 

  • Grewal, M., & Andrews, A. (2008). Practical considerations. In Kalman filtering: Theory and practice using MATLAB (pp. 355–426). Wiley-IEEE Press.

  • Gustafsson, F., Gunnarsson, F., Bergman, N., Forssell, U., Jansson, J., Karlsson, R., et al. (2002). Particle filters for positioning, navigation, and tracking. IEEE Transactions on Signal Processing, 50(2), 425–437.

    Article  Google Scholar 

  • Howard, A. (2008). Real-time stereo visual odometry for autonomous ground vehicles. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008, IROS 2008 (pp. 3946–3952). IEEE.

  • Hsu, C. W., Chang, C. C., Lin, C. J., et al. (2003). A practical guide to support vector classification. Technical report.

  • Hwang, I., Kim, S., Kim, Y., & Seah, C. E. (2010). A survey of fault detection, isolation, and reconfiguration methods. IEEE Transactions on Control Systems Technology, 18(3), 636–653.

    Article  Google Scholar 

  • Knorn, F., & Leith, D. J. (2008). Adaptive Kalman filtering for anomaly detection in software appliances. INFOCOM Workshops 2008 (pp. 1–6). IEEE.

  • Konolige, K., Agrawal, M., & Sola, J. (2011). Large-scale visual odometry for rough terrain. In M. Kaneko & Y. Nakamura (Eds.), Robotics research (Vol. 66, pp. 201–212)., Springer tracts in advanced robotics Berlin: Springer.

    Chapter  Google Scholar 

  • Kruijff, G. J., Janicek, M., Keshavdas, S., Larochelle, B., Zender, H., Smets, N., et al. (2012). Experience in system design for human-robot teaming in urban search & rescue. In Proceedings of 8th International Conference on Field and Service Robotics (pp. 111–125). Springer, STAR.

  • Kubat, M., Holte, R., & Matwin, S. (1997). Learning when negative examples abound. In Machine learning: ECML-97 (pp. 146–153). Springer.

  • Kubelka, V., & Reinstein, M. (2012). Complementary filtering approach to orientation estimation using inertial sensors only. In IEEE International Conference on Robotics and Automation (ICRA), 2012 (pp 599–605). IEEE.

  • Kubelka, V., Oswald, L., Pomerlau, F., Colas, F., Svoboda, T., & Reinstein, M. (2014). Robust data fusion of multi-modal sensory information for mobile robots. Journal of Field Robotics. doi:10.1002/rob.21535.

  • Kümmerle, R., Grisetti, G., Strasdat, H., Konolige, K., & Burgard, W. (2011). G2o: A general framework for graph optimization. In IEEE International Conference on Robotics and Automation (ICRA), 2011 (pp. 3607–3613).

  • Lau, T. K., & Lin, K. (2011). Evolutionary tuning of sigma-point Kalman filters. In IEEE International Conference on Robotics and Automation (ICRA), 2011 (pp. 771–776). IEEE.

  • Laxhammar, R., Falkman, G., & Sviestins, E. (2009). Anomaly detection in sea traffic—A comparison of the Gaussian mixture model and the kernel density estimator. In 12th International Conference on Information Fusion, 2009, FUSION’09 (pp. 756–763). IEEE.

  • Ma, J., Susca, S., Bajracharya, M., Matthies, L., Malchano, M., & Wooden, D. (2012). Robust multi-sensor, day/night 6-dof pose estimation for a dynamic legged vehicle in GPS-denied environments. In Proceedings of the IEEE International Robotics and Automation (ICRA) Conference (pp. 619–626).

  • Mitchell, T. M. (1997). Machine learning (1st ed.). New York: McGraw-Hill Inc.

    MATH  Google Scholar 

  • Morales, Y., Takeuchi, E., & Tsubouchi, T. (2008). Vehicle localization in outdoor woodland environments with sensor fault detection. In IEEE International Conference on Robotics and Automation, 2008, ICRA 2008 (pp. 449–454). IEEE.

  • Murphy, K. P. (2012). Machine learning: A probabilistic perspective., Adaptive computation and machine learning series Cambridge, MA: The MIT Press.

    Google Scholar 

  • Ndong, J., & Salamatian, K. (2011). A robust anomaly detection technique using combined statistical methods. In Ninth Annual Communication Networks and Services Research Conference (CNSR), 2011 (pp. 101–108). IEEE.

  • Pettersson, O. (2005). Execution monitoring in robotics: A survey. Robotics and Autonomous Systems, 53(2), 73–88.

    Article  MathSciNet  Google Scholar 

  • Pomerleau, F., Colas, F., Siegwart, R., & Magnenat, S. (2013). Comparing ICP variants on real-world data sets. Autonomous Robots, 34(3), 133–148.

    Article  Google Scholar 

  • Reinstein, M., & Hoffmann, M. (2013). Dead reckoning in a dynamic quadruped robot based on multimodal proprioceptive sensory information. IEEE Transactions on Robotics, 29(2), 563–571.

    Article  Google Scholar 

  • Reinstein, M., Kubelka, V., & Zimmermann, K. (2013). Terrain adaptive odometry for mobile skid-steer robots. In IEEE International Conference on Robotics and Automation (ICRA), 2013 (pp. 4706–4711).

  • Rodriguez, F. S. A., Fremont, V., & Bonnifait, P. (2009). An experiment of a 3d real-time robust visual odometry for intelligent vehicles. In Proceedings of the 12th International IEEE Conference Intelligent Transportation Systems ITSC ’09 (pp. 1–6).

  • Sagha, H., Bayati, H., Millan, Jd R, & Chavarriaga, R. (2013). On-line anomaly detection and resilience in classifier ensembles. Pattern Recognition Letters, 34(15), 1916–1927.

    Article  Google Scholar 

  • Sarkka, S., & Nummenmaa, A. (2009). Recursive noise adaptive Kalman filtering by variational Bayesian approximations. IEEE Transactions on Automatic Control, 54(3), 596–600.

    Article  MathSciNet  Google Scholar 

  • Scaramuzza, D., & Fraundorfer, F. (2011). Visual odometry. IEEE Robotics & Automation Magazine, 18(4), 80–92.

    Article  Google Scholar 

  • Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe-Taylor, J., & Platt, J. C. (1999). Support vector method for novelty detection. NIPS, 12, 582–588.

    Google Scholar 

  • Shen, J., Tick, D., & Gans, N. (2011). Localization through fusion of discrete and continuous epipolar geometry with wheel and imu odometry. In Proceedings of the American Control Conference (ACC) (pp. 1292–1298).

  • Simanek, J., Reinstein, M., & Kubelka, V. (2015). Evaluation of the EKF-based estimation architectures for data fusion in mobile robots. IEEE/ASME Transactions on Mechatronics, 20(2), 985–990.

  • Simon, D. (2006). Optimal state estimation: Kalman, H infinity, and nonlinear approaches. www.wiley.com.

  • Soule, A., Salamatian, K., & Taft, N. (2005). Combining filtering and statistical methods for anomaly detection. In Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement (pp. 31–31). USENIX Association.

  • Sukkarieh, S., Nebot, E., & Durrant-Whyte, H. (1999). A high integrity IMU/GPS navigation loop for autonomous land vehicle applications. IEEE Transactions on Robotics and Automation, 15(3), 572–578.

    Article  Google Scholar 

  • Sundvall, P., & Jensfelt, P. (2006). Fault detection for mobile robots using redundant positioning systems. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006, ICRA 2006 (pp. 3781–3786). IEEE.

  • Suzuki, T., Kitamura, M., Amano, Y., & Hashizume, T. (2010). 6-DOF localization for a mobile robot using outdoor 3d voxel maps. In Proceedings of the IEEE/RSJ International Intelligent Robots and Systems (IROS) Conference (pp. 5737–5743).

  • Svoboda, T., Pajdla, T., & Hlaváč, V. (1998). Motion estimation using central panoramic cameras. In S. Hahn (Ed.) IEEE International Conference on Intelligent Vehicles (pp. 335–340). Causal Productions.

  • Ting, J. A., Theodorou, E., & Schaal, S. (2007). A Kalman filter for robust outlier detection. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2007, IROS 2007 (pp. 1514–1519).

  • Tsai, C. F., Hsu, Y. F., Lin, C. Y., & Lin, W. Y. (2009). Intrusion detection by machine learning: A review. Expert Systems with Applications, 36(10), 11,994–12,000.

    Article  Google Scholar 

  • Weiss, S. M. (2012). Vision based navigation for micro helicopters. PhD thesis, Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 20305, 2012.

  • Yi, J., Zhang, J., Song, D., & Jayasuriya, S. (2007). Imu-based localization and slip estimation for skid-steered mobile robots. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems IROS 2007 (pp. 2845–2850).

  • Yoshida, T., Irie, K., Koyanagi, E., & Tomono, M. (2010). A sensor platform for outdoor navigation using gyro-assisted odometry and roundly-swinging 3d laser scanner. In Proceedings of the IEEE/RSJ Int Intelligent Robots and Systems (IROS) Conference (pp. 1414–1420).

Download references

Acknowledgments

The research presented here was supported as follows: Jakub Simanek was supported by the Grant Agency of the CTU in Prague (SGS13/144/OHK3/2T/13); Vladimir Kubelka was supported by the Czech Science Foundation (14-13876S); and Michal Reinstein was supported by the European Union FP7 under the TRADR Project No. FP7-ICT-609763. We would like to thank Francis Colas and Francois Pomerleau for their valuable advices concerning the optimal estimation, and Lorenz Oswald for the assistance during the experiments. The Vicon motion capture system was kindly provided by Autonomous Systems Lab, ETH Zurich, Switzerland.

Author information

Authors and Affiliations

  1. Department of Measurement, Faculty of Electrical Engineering, Czech Technical University in Prague, Technicka 2, 16627, Prague, Czech Republic

    Jakub Simanek

  2. Department of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague, Karlovo namesti 13, 12135, Prague, Czech Republic

    Vladimir Kubelka & Michal Reinstein

Authors
  1. Jakub Simanek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir Kubelka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michal Reinstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michal Reinstein.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mpg 102648 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Simanek, J., Kubelka, V. & Reinstein, M. Improving multi-modal data fusion by anomaly detection. Auton Robot 39, 139–154 (2015). https://doi.org/10.1007/s10514-015-9431-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-015-9431-6

Keywords

Search

Navigation