Skip to main content
Springer Nature Link
Log in

Night-to-day thermal image translation for deep thermal place recognition

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Place recognition PR is a fundamental task in autonomous robot systems that is still actively being researched. In recent years, CNN-based place recognition methods have surpassed classical methods. However, place recognition in the thermal infrared (TIR) image domain has shown poor performance when applied to both traditional and CNN-based methods due to the appearance variation of the same place throughout the day caused by varying temperature differences. In this paper, we propose a GAN-based nighttime to daytime thermal image translation model that translates thermal images captured at different times of the day into contrast-consistent and detail-preserving images, thus achieving time-agnostic thermal image representations. By applying our GAN-based models to input thermal images for place recognition tasks, we achieved a top-1 accuracy of 80.69% on the STHeReO dataset, outperforming other baseline methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Kim G, Park B, Kim A (2019) 1-day learning, 1-year localization: long-term lidar localization using scan context image. IEEE Robot Autom Lett 4(2):1948–1955

    Article  Google Scholar 

  2. Lowry S, Sünderhauf N, Newman P, Leonard JJ, Cox D, Corke P, Milford MJ (2015) Visual place recognition: a survey. IEEE Trans Robot 32(1):1–19

    Article  Google Scholar 

  3. Barros T, Pereira R, Garrote L, Premebida C, Nunes UJ (2021) Place recognition survey: an update on deep learning approaches. arXiv preprint arXiv:2106.10458,

  4. Kim G, Choi S, Kim A (2021) Scan context++: structural place recognition robust to rotation and lateral variations in urban environments. IEEE Trans Rob 38(3):1856–1874

    Article  Google Scholar 

  5. Cait K, Wang B, Lu CX (2022) Autoplace: Robust place recognition with single-chip automotive radar. In: 2022 International conference on robotics and automation (ICRA) pp 2222–2228. IEEE

  6. Muhamad Risqi U, Saputra PPB, de Gusmao C, Xiaoxuan L, Almalioglu Y, Rosa S, Chen C, Wahlström J, Wang W, Markham A, Trigoni N (2020) Deeptio: a deep thermal-inertial odometry with visual hallucination. IEEE Rob Autom Lett 5(2):1672–1679

    Article  Google Scholar 

  7. Wang Yu, Haoyao C, Yufeng L, Shiwu Z (2023) Edge-based monocular thermal-inertial odometry in visually degraded environments. IEEE Rob Autom Lett 8(4):2078–2085

    Article  Google Scholar 

  8. Khattak S, Papachristos C, Alexis K (2019) Keyframe-based direct thermal–inertial odometry. In: 2019 International conference on robotics and automation (ICRA), pp 3563–3569. IEEE

  9. Saputra MRU, Lu CX, de Gusmao PPB, Wang B, Markham A, Trigoni N (2021) Graph-based thermal–inertial SLAM with probabilistic neural networks. IEEE Trans Rob 38(3):1875–1893

    Article  Google Scholar 

  10. Jiang J, Chen X, Dai W, Gao Z, Zhang Y (2022) Thermal-inertial SLAM for the environments with challenging illumination. IEEE Rob Autom Lett 7(4):8767–8774

    Article  Google Scholar 

  11. Luo F, Li Y, Zeng G, Peng P, Wang G, Li Y (2022) Thermal infrared image colorization for nighttime driving scenes with top-down guided attention. IEEE Trans Intell Transp Syst 23(9):15808–15823

    Article  Google Scholar 

  12. Young-Sik S, Ayoung K (2019) Sparse depth enhanced direct thermal-infrared slam beyond the visible spectrum. IEEE Rob Autom Lett 4(3):2918–2925. https://doi.org/10.1109/LRA.2019.2923381

    Article  Google Scholar 

  13. Ian G, Jean P-A, Mehdi M, Bing X, David W-F, Sherjil O, Aaron C, Yoshua B (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  14. Lin C-T, Huang S-W, Yen-Yi W, Lai S-H (2020) Gan-based day-to-night image style transfer for nighttime vehicle detection. IEEE Trans Intell Transp Syst 22(2):951–963

    Article  Google Scholar 

  15. Li X, Guo X, Zhang (2022) N2d-gan: a night-to-day image-to-image translator. In 2022 IEEE international conference on multimedia and expo (ICME), pp 1–6. IEEE

  16. Cho YSY, Jeong J, Dejavugan AK (2018) Multi-temporal image translation toward long-term robot autonomy. In: ICRA workshop on long-term autonomy and deployment of intelligent robots in the real-world, Brisbane

  17. Anoosheh A, Sattler T, Timofte R, Pollefeys M, Van Gool L (2019) Night-to-day image translation for retrieval-based localization. In 2019 International conference on robotics and automation (ICRA), pp 5958–5964. IEEE

  18. Zhu J-Y, Park T, Isola P, Efros AA (2017a) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  19. Park T, Efros AA, Zhang R, Zhu J-Y (2020) Contrastive learning for unpaired image-to-image translation. In European conference on computer vision, pp 319–345. Springer

  20. Gálvez-López D, Tardos JD (2012) Bags of binary words for fast place recognition in image sequences. IEEE Trans Rob 28(5):1188–1197

    Article  Google Scholar 

  21. Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. In: European conference on computer vision, pp 430–443. Springer

  22. Michael C, Vincent L, Mustafa O, Tomasz T, Christoph S, Pascal F (2011) Brief: Computing a local binary descriptor very fast. IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298

    Google Scholar 

  23. Arandjelovic R, Gronat P, Torii A, Pajdla T, Sivic J (2016) Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5297–5307

  24. Zhu J-Y, Park T, Isola P, Efros AA (2017b) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  25. Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE international conference on computer vision, pp 1501–1510

  26. Younggun C, Hyesu J, Ramavtar M, Gaurav P, Ayoung K (2020) Underwater image dehazing via unpaired image-to-image translation. Int J Control Autom Syst 18(3):605–614

    Article  Google Scholar 

  27. Huang X, Liu M-Y, Belongie S, Kautz J (2018) Multimodal unsupervised image-to-image translation. In: Proceedings of the European conference on computer vision (ECCV), pp 172–189

  28. Hwang S, Park J, Kim N, Choi Y, Kweon IS (2015) Multispectral pedestrian detection: Benchmark dataset and baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1037–1045

  29. Yun S, Jung M, Kim J, Jung S, Cho Y, Jeon M-H, Kim G, Kim A (2022) Sthereo: Stereo thermal dataset for research in odometry and mapping. In: 2022 IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 3857–3864. IEEE

  30. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  31. Demir U, Unal G (2018) Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422

Download references

Acknowledgements

This research was conducted with the support of the “National R &D Project for Smart Construction Technology (23SMIP-A158708-04) funded by the Korea Agency for Infrastructure Technology Advancement under the Ministry of Land, Infrastructure and Transport, and managed by the Korea Expressway Corporation.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Seoul National University, 1 Gwanak-ro Gwanak-Gu, Seoul, 08826, Republic of Korea

    Dong-Guw Lee, Hyeonjae Gil, Seungsang Yun, Jeongyun Kim & Ayoung Kim

Authors
  1. Dong-Guw Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyeonjae Gil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seungsang Yun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeongyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ayoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayoung Kim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, DG., Gil, H., Yun, S. et al. Night-to-day thermal image translation for deep thermal place recognition. Intel Serv Robotics 16, 403–413 (2023). https://doi.org/10.1007/s11370-023-00473-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-023-00473-7

Keywords