Skip to main content
SpringerLink
Log in

Smart Fusion of Multi-sensor Ubiquitous Signals of Mobile Device for Localization in GNSS-Denied Scenarios

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

In order to support indoor and outdoor seamless location-based services (LBS), this paper proposes a smart fusion architecture for combing the ubiquitous signals of the mobile device integrated multi-modal sensors based on deep learning, which can fuse the vision/wireless/inertial information. The core of the fusion architecture is an improved four-layers deep neural network that integrating a convolutional neural network (CNN) and an improved particle filter. In the first place, inspired by creating the RGB-D image, we change the image gray by using a normalized magnetic strength and scale the image intensity by using a normalized WiFi signal strength, which results in a new image named RGB-WM image. Then, homogeneous features are extracted from the RGB-WM image based on the improved CNN for achieving context-awareness. Based on combing the context information, we introduce a new particle filter for fusing different information from multi-modal sensors. In order to evaluate our proposed positioning architecture, we have conducted extensive experiments in four different scenarios including our laboratory, and the campus of our university. Experimental results demonstrate the precision and recall of the RGB-WM image feature is 95.6 and 4.1% respectively. Furthermore, the proposed infrastructure-free fusion architecture reduced the root mean square error (RMSE) of locations in the range of 13.3–55.2% in walking experiments with two smartphones, under two motion conditions, which indicates a superior performance of our proposed image/WiFi/magnetic/inertial fusion architecture over the state-of-the-art with these four localization scenarios. The ubiquitous positioning accuracy of our proposed algorithm is less than 1.23 m, which can meet the requirement of the complex GNSS-denied regions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Deng, Z. A., Wang, G., Qin, D., Na, Z., Cui, Y., & Chen, J. (2016). Continuous indoor positioning fusing WiFi, smartphone sensors and landmarks. Sensors, 16, 1427.

    Article  Google Scholar 

  2. Tehrani, M., Uysal, M., & Yanikomeroglu, H. (2014). Device-to-device communication in 5G cellular networks: Challenges, solutions, and future directions. IEEE Communications Magazine, 52, 86–92.

    Article  Google Scholar 

  3. Arain, Q. A., Memon, H., Memon, I., Memon, M. H., Shaikh, R. A., & Mangi, F. A. (2017). Intelligent travel information platform based on location base services to predict user travel behavior from user-generated GPS traces. International Journal of Computers and Applications, 39, 1–14.

    Article  Google Scholar 

  4. Memon, I., Ali, Q., Zubedi, A., & Mangi, F. A. (2017). DPMM: Dynamic pseudonym-based multiple mix-zones generation for mobile traveler. Multimedia Tools and Applications, 76, 24359–24388.

    Article  Google Scholar 

  5. Makki, A., Siddig, A., Saad, M., & Bleakley, C. (2015). Survey of WiFi positioning using time-based techniques. Computer Networks, 88, 218–233.

    Article  Google Scholar 

  6. Jiao, J., Deng, Z., Xu, L., & Li, F. (2016). A hybrid of smartphone camera and basestation wide-area indoor positioning method. KSII Transactions on Internet & Information Systems, 10, 723–743.

    Google Scholar 

  7. Chen, L., Pei, L., Kuusniemi, H., Chen, Y. W., Kroger, T., & Chen, R. Z. (2013). Bayesian fusion for indoor positioning using bluetooth fingerprints. Wireless Personal Communications, 70, 1735–1745.

    Article  Google Scholar 

  8. Ahmed, H. I., Wei, P., Memon, I., Du, Y., & Xie, W. (2013). Estimation of time difference of arrival (TDoA) for the source radiates BPSK signal. IJCSI International Journal of Computer Science Issues, 10, 1694–0784.

    Google Scholar 

  9. De Angelis, G., Pasku, V., De Angelis, A., Dionigi, M., Mongiardo, M., Moschitta, A., et al. (2015). An indoor AC magnetic positioning system. IEEE Transactions on Instrumentation and Measurement, 64, 1275–1283.

    Google Scholar 

  10. Li, Y., Zhuang, Y., Lan, H., Zhang, P., Niu, X., & El-Sheimy, N. (2015). WiFi-aided magnetic matching for indoor navigation with consumer portable devices. Micromachines, 6, 747–764.

    Article  Google Scholar 

  11. Wang, F., Cui, J., Phang, S. K., Chen, B. M., & Lee, T. H. (2013). A mono-camera and scanning laser range finder based UAV indoor navigation system. In Unmanned Aircraft Systems (ICUAS), 2013 International Conference on, 2013 (pp. 694–701). IEEE.

  12. Kuo, Y.-S., Pannuto, P., Hsiao, K.-J., Dutta, P., & Arbor, A. (2014). Luxapose: Indoor positioning with mobile phones and visible light. In Mobicom’14 (pp. 299–301).

  13. Liu, Z., Zhang, L., Liu, Q., Yin, Y., Cheng, L., & Zimmermann, R. (2016). Fusion of magnetic and visual sensors for indoor localization: Infrastructure-free and more effective. IEEE Transactions on Multimedia, 9210, 1–15.

    Google Scholar 

  14. Santoso, F., Garratt, M. A., & Anavatti, S. G. (2016). Visual-inertial navigation systems for aerial robotics: Sensor fusion and technology. IEEE Transactions on Automation Science and Engineering, 14, 260–275.

    Article  Google Scholar 

  15. Li, Y., Zhuang, Y., Zhang, P., Lan, H., Niu, X., & El-Sheimy, N. (2017). An improved inertial/wifi/magnetic fusion structure for indoor navigation. Information Fusion, 34, 101–119.

    Article  Google Scholar 

  16. Wu, Z., Jedari, E., Muscedere, R., & Rashidzadeh, R. (2015). Improved particle filter based on WLAN RSSI fingerprinting and smart sensors for indoor localization. Computer Communications, 83, 64–71.

    Article  Google Scholar 

  17. Wen, F., Zhang, Z., Wang, K., Sheng, G., & Zhang, G. (2018). Angle estimation and mutual coupling self-calibration for ULA-based bistatic MIMO radar. Signal Processing, 144, 61–67.

    Article  Google Scholar 

  18. Wen, F., Zhang, Z., Zhang, G., Zhang, Y., Wang, X., & Zhang, X. (2017). A tensor-based covariance differencing method for direction estimation in bistatic MIMO radar with unknown spatial colored noise. IEEE Access, 5, 18451–18458.

    Article  Google Scholar 

  19. Wen, F., Xiong, X., Su, J., & Zhang, Z. (2017). Angle estimation for bistatic MIMO radar in the presence of spatial colored noise. Signal Processing, 134, 261–267.

    Article  Google Scholar 

  20. Wen, F., Xiong, X., & Zhang, Z. (2017). Angle and mutual coupling estimation in bistatic MIMO radar based on PARAFAC decomposition. Digital Signal Processing, 65, 1–10.

    Article  MathSciNet  Google Scholar 

  21. Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A., Banino, A., et al. (2017). Learning to navigate. In Iclr (pp. 1–11).

  22. Sharif Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: An astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2014 (pp. 806–813).

  23. Jiang, W. (2016). CNN-RNN: A unified framework for multi-label image classification. Cvpr, 2016, 2285–2294.

    Google Scholar 

  24. Greene, M. R., Baldassano, C., Esteva, A., Beck, D. M., & Fei-Fei, L. (2016). Visual scenes are categorized by function. Journal of Experimental Psychology: General, 145, 82–94.

    Article  Google Scholar 

  25. Rusdinar, A., Kim, J., Lee, J., & Kim, S. (2012). Implementation of real-time positioning system using extended Kalman filter and artificial landmark on ceiling. Journal of Mechanical Science and Technology, 26, 949–958.

    Article  Google Scholar 

  26. Bae, H., Golparvar-Fard, M., & White, J. (2015). Image-based localization and content authoring in structure-from-motion point cloud models for real-time field reporting applications. Journal of Computing in Civil Engineering, 29, B4014008.

    Article  Google Scholar 

  27. Jiao, J., Deng, Z., Mo, J., & Li, C. (2016). Turbo fusion of LPQ and HOG feature sets for indoor positioning using smartphone camera. Electronic Imaging, 2016, 1–7.

    Article  Google Scholar 

  28. Papaioannou, S., Wen, H., Markham, A., & Trigoni, N. (2015). Fusion of radio and camera sensor data for accurate indoor positioning. In Proceedings—11th IEEE international conference on mobile ad hoc and sensor systems, MASS 2014 (pp. 109–117).

  29. Vemprala, S., & Saripalli, S. (2016). Vision based collaborative localization for multirotor vehicles. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, 2016 (pp. 1653–1658). IEEE.

  30. Pei, L., Zhang, M., Zou, D., Chen, R., & Chen, Y. (2016). A survey of crowd sensing opportunistic signals for indoor localization. In Mobile Information Systems (vol. 2016).

  31. Chai, W. N., Chen, C., Edwan, E., Zhang, J. Y., Loffeld, O., & IEEE (2012). 2D/3D indoor navigation based on multi-sensor assisted pedestrian navigation in Wi-Fi environments. In 2012 Ubiquitous positioning, indoor navigation, and location based service (Upinlbs).

  32. Chen, Z., Zou, H., Jiang, H., Zhu, Q., Soh, Y. C., & Xie, L. (2015). Fusion of WiFi, smartphone sensors and landmarks using the kalman filter for indoor localization. Sensors (Switzerland), 15, 715–732.

    Article  Google Scholar 

  33. Lee, S., Cho, B., Koo, B., Ryu, S., Choi, J., & Kim, S. (2015). Kalman filter-based indoor position tracking with self-calibration for RSS variation mitigation. International Journal of Distributed Sensor Networks, 11(8), 674635.

    Article  Google Scholar 

  34. Kleinert, M., Stilla, U., & IEEE. (2013). A sensor-centric EKF for inertial-aided visual odometry. In 2013 International conference on indoor positioning and indoor navigation (Ipin).

  35. Deng, Z.-A., Hu, Y., Yu, J., & Na, Z. (2015). Extended Kalman filter for real time indoor localization by fusing WiFi and smartphone inertial sensors. Micromachines, 6, 523–543.

    Article  Google Scholar 

  36. Chen, X., Wang, X., & Xu, Y. (2014). Performance enhancement for a GPS vector-tracking loop utilizing an adaptive iterated extended kalman filter. Sensors, 14, 23630–23649.

    Article  Google Scholar 

  37. Xu, Y., Chen, X., & Li, Q. (2014). Adaptive iterated extended kalman filter and its application to autonomous integrated navigation for indoor robot. The Scientific World Journal, 2014, 2356–6140.

    Google Scholar 

  38. Khaleghi, B., Khamis, A., Karray, F. O., & Razavi, S. N. (2013). Multisensor data fusion: A review of the state-of-the-art. Information Fusion, 14, 28–44.

    Article  Google Scholar 

  39. Doucet, A., & Johansen, A. M. (2009). A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of Nonlinear Filtering, 12, 3.

    MATH  Google Scholar 

  40. Levchev, P., Krishnan, M. N., Yu, C., Menke, J., & Zakhor, A. (2014). Simultaneous fingerprinting and mapping for multimodal image and WiFi indoor positioning. In IPIN 2014–2014 international conference on indoor positioning and indoor navigation (pp. 442–450).

  41. Guerrero, L. A., Vasquez, F., & Ochoa, S. F. (2012). An indoor navigation system for the visually impaired. Sensors, 12, 8236–8258.

    Article  Google Scholar 

  42. Pak, J. M., Ahn, C. K., Shmaliy, Y. S., & Lim, M. T. (2015). Improving reliability of particle filter-based localization in wireless sensor networks via hybrid particle/FIR filtering. IEEE Transactions on Industrial Informatics, 11, 1089–1098.

    Article  Google Scholar 

  43. Xie, H., Gu, T., Tao, X., Ye, H., & Lu, J. (2016). A reliability-augmented particle filter for magnetic fingerprinting based indoor localization on smartphone. IEEE Transactions on Mobile Computing, 15, 1877–1892.

    Article  Google Scholar 

  44. Perez, I., Pinchin, J., Brown, M., Blum, J., & Sharples, S. (2016). Unsupervised labelling of sequential data for location identification in indoor environments. Expert Systems with Applications, 61, 386–393.

    Article  Google Scholar 

  45. He, S., & Chan, S. H. G. (2016). Wi-Fi fingerprint-based indoor positioning: Recent advances and comparisons. IEEE Communications Surveys and Tutorials, 18, 466–490.

    Article  Google Scholar 

  46. Memon, M. H., Li, J.-P., Memon, I., & Arain, Q. A. (2017). GEO matching regions: Multiple regions of interests using content based image retrieval based on relative locations. Multimedia Tools and Applications, 76, 15377–15411.

    Article  Google Scholar 

  47. Zhang, W., Liu, K., Zhang, W., Zhang, Y., & Gu, J. (2016). Deep Neural Networks for wireless localization in indoor and outdoor environments. Neurocomputing, 194, 279–287.

    Article  Google Scholar 

  48. Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., et al. (2016). Past, present, and future of simultaneous localization and mapping: Towards the robust-perception age. IEEE Transactions on Robotics, 32(6), 1309–1332.

    Article  Google Scholar 

  49. De Silva, O., Mann, G. K. I., & Gosine, R. G. (2015). An ultrasonic and vision-based relative positioning sensor for multirobot localization. Sensors Journal, IEEE, 15, 1716–1726.

    Article  Google Scholar 

  50. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).

  51. Girshick, R. (2015). Fast r-cnn.

  52. Ma, R., Guo, Q., Hu, C., & Xue, J. (2015). An improved WiFi indoor positioning algorithm by weighted fusion. Sensors (Basel, Switzerland), 15, 21824–21843.

    Article  Google Scholar 

Download references

Acknowledgements

The project sponsored by the the National Key Research and Development Program (No. 2016YFB0502002).

Author information

Authors and Affiliations

  1. Beijing University of Posts and Telecommunications, No 10, Xitucheng Road, Haidian District, Beijing, 100876, People’s Republic of China

    Jichao Jiao, Zhongliang Deng & Fei Li

  2. Department of Software Engineering, Mehran University of Engineering and Technology, Jamshoro, Pakistan

    Qasim Ali Arain

Authors
  1. Jichao Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongliang Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qasim Ali Arain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jichao Jiao.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiao, J., Deng, Z., Arain, Q.A. et al. Smart Fusion of Multi-sensor Ubiquitous Signals of Mobile Device for Localization in GNSS-Denied Scenarios. Wireless Pers Commun 116, 1507–1523 (2021). https://doi.org/10.1007/s11277-018-5725-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-018-5725-2

Keywords

Search

Navigation