research-article

ROECS: A Robust Semi-direct Pipeline Towards Online Extrinsics Correction of the Surround-view System

Authors:

Yicong ZhouAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 3153 - 3161

https://doi.org/10.1145/3474085.3475461

Published: 17 October 2021 Publication History

Abstract

Generally, a surround-view system (SVS), which is an indispensable component of advanced driving assistant systems (ADAS), consists of four to six wide-angle fisheye cameras. As long as both intrinsics and extrinsics of all cameras have been calibrated, a top-down surround-view with the real scale can be synthesized at runtime from fisheye images captured by these cameras. However, when the vehicle is driving on the road, relative poses between cameras in the SVS may change from the initial calibrated states due to bumps or collisions. In case that extrinsics' representations are not adjusted accordingly, on the surround-view, obvious geometric misalignment will appear. Currently, the researches on correcting the extrinsics of the SVS in an online manner are quite sporadic, and a mature and robust pipeline is still lacking. As an attempt to fill this research gap to some extent, in this work, we present a novel extrinsics correction pipeline designed specially for the SVS, namely ROECS (Robust Online Extrinsics Correction of the Surround-view system). Specifically, a "refined bi-camera error" model is firstly designed. Then, by minimizing the overall "bi-camera error" within a sparse and semi-direct framework, the SVS's extrinsics can be iteratively optimized and become accurate eventually. Besides, an innovative three-step pixel selection strategy is also proposed. The superior robustness and the generalization capability of ROECS are validated by both quantitative and qualitative experimental results. To make the results reproducible, the collected data and the source code have been released at https://cslinzhang.github.io/ROECS/.

References

[1]

Roberto Battiti. 1992. First- and Second-order Methods for Learning: Between Steepest Descent and Newton's Method. Neural Computation, Vol. 4, 2 (1992), 141--166. https://doi.org/10.1162/neco.1992.4.2.141

Digital Library

[2]

Kyoungtaek Choi, Ho Gi Jung, and Jae Kyu Suhr. 2018. Automatic Calibration of an Around View Monitor System Exploiting Lane Markings. Sensors, Vol. 18, 9 (2018), 2956:1--26. https://doi.org/10.3390/s18092956

[3]

Javier Civera, Andrew J. Davison, and J. M. MartÍnez Montiel. 2008. Inverse Depth Parametrization for Monocular SLAM. IEEE Trans. Robotics, Vol. 24, 5 (2008), 932--945. https://doi.org/10.1109/TRO.2008.2003276

Digital Library

[4]

Juan M. Collado, Cristina Hilario, Arturo de la Escalera, and Jose M. Armingol. 2006. Self-calibration of an On-board Stereo-vision System for Driver Assistance Systems. In IEEE Intelligent Vehicles Symposium (IVS'06). IEEE, Meguro--Ku, Japan, 156--162. https://doi.org/10.1109/IVS.2006.1689621

[5]

Thao Dang and Christian Hoffmann. 2006. Tracking Camera Parameters of an Active Stereo Rig. In Joint DAGM Symposium (DAGM'06). Springer, Berlin, Germany, 627--636. https://doi.org/10.1007/11861898_63

Digital Library

[6]

Jakob Engel, Vladlen Koltun, and Daniel Cremers. 2018. Direct Sparse Odometry. IEEE Trans. Pattern Analysis and Machine Intell., Vol. 40, 3 (2018), 611--625. https://doi.org/10.1109/TPAMI.2017.2658577

[7]

Jakob Engel, Vsenko Usenko, and Daniel Cremers. 2016. A Photometrically Calibrated Benchmark For Monocular Visual Odometry. CoRR, Vol. abs/1607.02555 (2016). arxiv: 1607.02555 http://arxiv.org/abs/1607.02555

[8]

Markus Gressmann, Günther Palm, and Otto Löhlein. 2011. Surround View Pedestrian Detection Using Heterogeneous Classifier Cascades. In International IEEE Conference on Intelligent Transportation Systems (ITSC'11). IEEE, Washington, DC, USA, 1317--1324. https://doi.org/10.1109/ITSC.2011.6082895

[9]

Kazukuni Hamada, Zhencheng Hu, Mengyang Fan, and Hui Chen. 2015. Surround View based Parking Lot Detection and Tracking. In IEEE Intelligent Vehicles Symposium (IVS'2015). IEEE, Seoul, Korea (South), 1106--1111. https://doi.org/10.1109/IVS.2015.7225832

[10]

Peter Hansen, Hatem Alismail, Peter Rander, and Brett Browning. 2012. Online Continuous Stereo Extrinsic Parameter Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). IEEE, Providence, RI, USA, 1059--1066. https://doi.org/10.1109/CVPR.2012.6247784

Digital Library

[11]

Simon Hecker, Dengxin Dai, and Luc Van Gool. 2018. End-to-end Learning of Driving Models with Surround-view Cameras and Route Planners. In European Conference on Computer Vision (ECCV'18). Springer, Munich, Germany, 435--453. https://doi.org/10.1007/978--3-030-01234--2_27

[12]

William C. Hoffman. 1966. The Lie Algebra of Visual Perception. Journal of Mathematical Psychology, Vol. 3, 1 (1966), 65--98. https://doi.org/10.1016/0022--2496(66)90005--8

[13]

Stephanie Hold, Steffen Görmer, Anton Kummert, Mirko Meuter, and Stefan Muller-Schneiders. 2009. A Novel Approach for the Online Initial Calibration of Extrinsic Parameters for a Car-mounted Camera. In International IEEE Conference on Intelligent Transportation Systems (ITSC'09). IEEE, St. Louis, MO, USA, 420--425. https://doi.org/10.1109/ITSC.2009.5309853

[14]

Cong Hou, Haizhou Ai, and Shihong Lao. 2007. Multiview Pedestrian Detection based on Vector Boosting. In Asian Conference on Computer Vision (ACCV'07). Springer, Berlin, Heidelberg, 210--219. https://doi.org/10.1007/978--3--540--76386--4_19

Digital Library

[15]

Moré Jorge J. 1978. The Levenberg-Marquardt algorithm: Implementation and theory. In Numerical Analysis (Lecture Notes in Mathematics). Springer, Berlin, Heidelberg, 105--116. https://doi.org/10.1007/BFb0067700

[16]

Moritz Knorr, Wolfgang Niehsen, and Christoph Stiller. 2013. Online Extrinsic Multi-camera Calibration Using Ground Plane Induced Homographies. In IEEE Intelligent Vehicles Symposium (IVS'13). IEEE, Gold Coast, QLD, Australia, 236--241. https://doi.org/10.1109/IVS.2013.6629476

[17]

Linshen Li, Lin Zhang, Xiyuan Li, Xiao Liu, Ying Shen, and Lu Xiong. 2017. Vision-based Parking-slot Detection: A Benchmark and a Learning-based Approach. In IEEE International Conference on Multimedia and Expo (ICME'17). IEEE, Hong Kong, China, 649--654. https://doi.org/10.1109/ICME.2017.8019419

[18]

Chien-Chuan Lin and Ming-Shi Wang. 2012. A Vision based Top-view Transformation Model for a Vehicle Parking Assistant. Sensors, Vol. 12, 4 (2012), 4431--4446. https://doi.org/10.3390/s120404431

[19]

Yonggen Ling and Shaojie Shen. 2016. High-precision Online Markerless Stereo Extrinsic Calibration. In International Conference on Intelligent Robots and Systems (IROS'16). IEEE/RSJ, Daejeon, Korea (South), 1771--1778. https://doi.org/10.1109/IROS.2016.7759283

[20]

Xiao Liu, Lin Zhang, Ying Shen, Shaoming Zhang, and Shengjie Zhao. 2019. Online Camera Pose Optimization for the Surround-view System. In ACM International Conference on Multimedia (MM '19). ACM, New York, the United States, 383--391. https://doi.org/10.1145/3343031.3350885

Digital Library

[21]

Sergiu Nedevschi, Cristian Vancea, Tiberiu Marita, and Thorsten Graf. 2007. Online Extrinsic Parameters Calibration for Stereovision Systems Used in Far-range Detection Vehicle Applications. IEEE Trans. Intell. Transportation Systems, Vol. 8, 4 (2007). https://doi.org/10.1109/TITS.2007.908576

Digital Library

[22]

Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An Efficient Alternative to SIFT or SURF. International Conference on Computer Vision (ICCV'11) (2011), IEEE, Barcelona, Spain, 2564--2571. https://doi.org/10.1109/ICCV.2011.6126544

Digital Library

[23]

Chunxiang Wang, Hengrun Zhang, Ming Yang, Xudong Wang, Lei Ye, and Chunzhao Guo. 2014. Automatic Parking based on a Bird's Eye View Vision System. Advances in Mechanical Engineering, Vol. 6 (2014), 847406:1--13. https://doi.org/10.1155/2014/847406

[24]

R. W. M. Wedderburn. 1974. Quasi-Likelihood Functions, Generalized Linear Models, and the Gauss-Newton Method. Biometrika, Vol. 61, 3 (1974), 439--447. https://doi.org/10.2307/2334725

[25]

Jin Xu, Guang Chen, and Ming Xie. 2000. Vision-guided Automatic Parking for Smart Car. In IEEE Intelligent Vehicles Symposium (IVS'00). IEEE, Dearborn, MI, USA, 725--730. https://doi.org/10.1109/IVS.2000.898435

[26]

Lin Zhang, Junhao Huang, Xiyuan Li, and Lu Xiong. 2018. Vision-based Parking-slot Detection: A DCNN-based Approach and a Large-scale Benchmark Dataset. IEEE Trans. Image Processing, Vol. 27, 11 (2018), 5350--5364. https://doi.org/10.1109/TIP.2018.2857407

Digital Library

[27]

Tianjun Zhang, Lin Zhang, Ying Shen, Yong Ma, Shengjie Zhao, and Yicong Zhou. 2020. OECS: Towards Online Extrinsics Correction for The Surround-view System. In IEEE International Conference on Multimedia and Expo (ICME'20). IEEE, London, UK, 1--6. https://doi.org/10.1109/ICME46284.2020.9102803

[28]

Kun Zhao, Uri Iurgel, Mirko Meuter, and Josef Pauli. 2014. An Automatic Online Camera Calibration System for Vehicular Applications. In IEEE Conference on Intelligent Transportation Systems (ITSC'14). IEEE, Qingdao, China, 1490--1492. https://doi.org/10.1109/ITSC.2014.6957643

Cited By

Li JPi JWei PLuo ZYan G(2024)Automatic Multi-Camera Calibration and Refinement Method in Road Scene for Self-Driving CarIEEE Transactions on Intelligent Vehicles10.1109/TIV.2023.33236659:1(2429-2438)Online publication date: Jan-2024
https://doi.org/10.1109/TIV.2023.3323665
Qin LLin CHuang SYang SZhao Y(2024)Camera calibration for the surround-view system: a benchmark and datasetThe Visual Computer10.1007/s00371-024-03275-940:10(7457-7470)Online publication date: 16-Feb-2024
https://doi.org/10.1007/s00371-024-03275-9
Chen YZhang LShen YZhao BZhou Y(2023)Extrinsic Self-Calibration of the Surround-View System: A Weakly Supervised ApproachIEEE Transactions on Multimedia10.1109/TMM.2022.314488925(1622-1635)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3144889

Index Terms

ROECS: A Robust Semi-direct Pipeline Towards Online Extrinsics Correction of the Surround-view System
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Camera calibration

Recommendations

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach
The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. ...
Online Camera Pose Optimization for the Surround-view System
MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Surround-view system is an important information medium for drivers to monitor the driving environment. A typical surround-view system consists of four to six fish-eye cameras arranged around the vehicle. From these camera inputs, a top-down image of ...
Camera calibration for the surround-view system: a benchmark and dataset
Abstract
Surround-view system (SVS) is widely used in the advanced driver assistance system (ADAS). SVS uses four fish-eye lenses to monitor real-time scenes around the vehicle. However, accurate intrinsic and extrinsic parameter estimation is required for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Science Foundation of Shanghai
Natural Science Foundation of China

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)11

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li JPi JWei PLuo ZYan G(2024)Automatic Multi-Camera Calibration and Refinement Method in Road Scene for Self-Driving CarIEEE Transactions on Intelligent Vehicles10.1109/TIV.2023.33236659:1(2429-2438)Online publication date: Jan-2024
https://doi.org/10.1109/TIV.2023.3323665
Qin LLin CHuang SYang SZhao Y(2024)Camera calibration for the surround-view system: a benchmark and datasetThe Visual Computer10.1007/s00371-024-03275-940:10(7457-7470)Online publication date: 16-Feb-2024
https://doi.org/10.1007/s00371-024-03275-9
Chen YZhang LShen YZhao BZhou Y(2023)Extrinsic Self-Calibration of the Surround-View System: A Weakly Supervised ApproachIEEE Transactions on Multimedia10.1109/TMM.2022.314488925(1622-1635)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3144889

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten