Tianjun Zhang
ABSTRACT
Generally, a surround-view system (SVS), which is an indispensable component of advanced driving assistant systems (ADAS), consists of four to six wide-angle fisheye cameras. As long as both intrinsics and extrinsics of all cameras have been calibrated, a top-down surround-view with the real scale can be synthesized at runtime from fisheye images captured by these cameras. However, when the vehicle is driving on the road, relative poses between cameras in the SVS may change from the initial calibrated states due to bumps or collisions. In case that extrinsics' representations are not adjusted accordingly, on the surround-view, obvious geometric misalignment will appear. Currently, the researches on correcting the extrinsics of the SVS in an online manner are quite sporadic, and a mature and robust pipeline is still lacking. As an attempt to fill this research gap to some extent, in this work, we present a novel extrinsics correction pipeline designed specially for the SVS, namely ROECS (Robust Online Extrinsics Correction of the Surround-view system). Specifically, a "refined bi-camera error" model is firstly designed. Then, by minimizing the overall "bi-camera error" within a sparse and semi-direct framework, the SVS's extrinsics can be iteratively optimized and become accurate eventually. Besides, an innovative three-step pixel selection strategy is also proposed. The superior robustness and the generalization capability of ROECS are validated by both quantitative and qualitative experimental results. To make the results reproducible, the collected data and the source code have been released at https://cslinzhang.github.io/ROECS/.
- Roberto Battiti. 1992. First- and Second-order Methods for Learning: Between Steepest Descent and Newton's Method. Neural Computation, Vol. 4, 2 (1992), 141--166. https://doi.org/10.1162/neco.1992.4.2.141 Google Scholar
Digital Library
- Kyoungtaek Choi, Ho Gi Jung, and Jae Kyu Suhr. 2018. Automatic Calibration of an Around View Monitor System Exploiting Lane Markings. Sensors, Vol. 18, 9 (2018), 2956:1--26. https://doi.org/10.3390/s18092956Google Scholar
- Javier Civera, Andrew J. Davison, and J. M. MartÍnez Montiel. 2008. Inverse Depth Parametrization for Monocular SLAM. IEEE Trans. Robotics, Vol. 24, 5 (2008), 932--945. https://doi.org/10.1109/TRO.2008.2003276 Google ScholarDigital Library
- Juan M. Collado, Cristina Hilario, Arturo de la Escalera, and Jose M. Armingol. 2006. Self-calibration of an On-board Stereo-vision System for Driver Assistance Systems. In IEEE Intelligent Vehicles Symposium (IVS'06). IEEE, Meguro--Ku, Japan, 156--162. https://doi.org/10.1109/IVS.2006.1689621Google Scholar
- Thao Dang and Christian Hoffmann. 2006. Tracking Camera Parameters of an Active Stereo Rig. In Joint DAGM Symposium (DAGM'06). Springer, Berlin, Germany, 627--636. https://doi.org/10.1007/11861898_63 Google ScholarDigital Library
- Jakob Engel, Vladlen Koltun, and Daniel Cremers. 2018. Direct Sparse Odometry. IEEE Trans. Pattern Analysis and Machine Intell., Vol. 40, 3 (2018), 611--625. https://doi.org/10.1109/TPAMI.2017.2658577Google ScholarCross Ref
- Jakob Engel, Vsenko Usenko, and Daniel Cremers. 2016. A Photometrically Calibrated Benchmark For Monocular Visual Odometry. CoRR, Vol. abs/1607.02555 (2016). arxiv: 1607.02555 http://arxiv.org/abs/1607.02555Google Scholar
- Markus Gressmann, Günther Palm, and Otto Löhlein. 2011. Surround View Pedestrian Detection Using Heterogeneous Classifier Cascades. In International IEEE Conference on Intelligent Transportation Systems (ITSC'11). IEEE, Washington, DC, USA, 1317--1324. https://doi.org/10.1109/ITSC.2011.6082895Google Scholar
- Kazukuni Hamada, Zhencheng Hu, Mengyang Fan, and Hui Chen. 2015. Surround View based Parking Lot Detection and Tracking. In IEEE Intelligent Vehicles Symposium (IVS'2015). IEEE, Seoul, Korea (South), 1106--1111. https://doi.org/10.1109/IVS.2015.7225832Google ScholarCross Ref
- Peter Hansen, Hatem Alismail, Peter Rander, and Brett Browning. 2012. Online Continuous Stereo Extrinsic Parameter Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). IEEE, Providence, RI, USA, 1059--1066. https://doi.org/10.1109/CVPR.2012.6247784 Google ScholarDigital Library
- Simon Hecker, Dengxin Dai, and Luc Van Gool. 2018. End-to-end Learning of Driving Models with Surround-view Cameras and Route Planners. In European Conference on Computer Vision (ECCV'18). Springer, Munich, Germany, 435--453. https://doi.org/10.1007/978--3-030-01234--2_27Google Scholar
- William C. Hoffman. 1966. The Lie Algebra of Visual Perception. Journal of Mathematical Psychology, Vol. 3, 1 (1966), 65--98. https://doi.org/10.1016/0022--2496(66)90005--8Google ScholarCross Ref
- Stephanie Hold, Steffen Görmer, Anton Kummert, Mirko Meuter, and Stefan Muller-Schneiders. 2009. A Novel Approach for the Online Initial Calibration of Extrinsic Parameters for a Car-mounted Camera. In International IEEE Conference on Intelligent Transportation Systems (ITSC'09). IEEE, St. Louis, MO, USA, 420--425. https://doi.org/10.1109/ITSC.2009.5309853Google ScholarCross Ref
- Cong Hou, Haizhou Ai, and Shihong Lao. 2007. Multiview Pedestrian Detection based on Vector Boosting. In Asian Conference on Computer Vision (ACCV'07). Springer, Berlin, Heidelberg, 210--219. https://doi.org/10.1007/978--3--540--76386--4_19 Google ScholarDigital Library
- Moré Jorge J. 1978. The Levenberg-Marquardt algorithm: Implementation and theory. In Numerical Analysis (Lecture Notes in Mathematics). Springer, Berlin, Heidelberg, 105--116. https://doi.org/10.1007/BFb0067700Google Scholar
- Moritz Knorr, Wolfgang Niehsen, and Christoph Stiller. 2013. Online Extrinsic Multi-camera Calibration Using Ground Plane Induced Homographies. In IEEE Intelligent Vehicles Symposium (IVS'13). IEEE, Gold Coast, QLD, Australia, 236--241. https://doi.org/10.1109/IVS.2013.6629476Google Scholar
- Linshen Li, Lin Zhang, Xiyuan Li, Xiao Liu, Ying Shen, and Lu Xiong. 2017. Vision-based Parking-slot Detection: A Benchmark and a Learning-based Approach. In IEEE International Conference on Multimedia and Expo (ICME'17). IEEE, Hong Kong, China, 649--654. https://doi.org/10.1109/ICME.2017.8019419Google ScholarCross Ref
- Chien-Chuan Lin and Ming-Shi Wang. 2012. A Vision based Top-view Transformation Model for a Vehicle Parking Assistant. Sensors, Vol. 12, 4 (2012), 4431--4446. https://doi.org/10.3390/s120404431Google ScholarCross Ref
- Yonggen Ling and Shaojie Shen. 2016. High-precision Online Markerless Stereo Extrinsic Calibration. In International Conference on Intelligent Robots and Systems (IROS'16). IEEE/RSJ, Daejeon, Korea (South), 1771--1778. https://doi.org/10.1109/IROS.2016.7759283Google Scholar
- Xiao Liu, Lin Zhang, Ying Shen, Shaoming Zhang, and Shengjie Zhao. 2019. Online Camera Pose Optimization for the Surround-view System. In ACM International Conference on Multimedia (MM '19). ACM, New York, the United States, 383--391. https://doi.org/10.1145/3343031.3350885 Google ScholarDigital Library
- Sergiu Nedevschi, Cristian Vancea, Tiberiu Marita, and Thorsten Graf. 2007. Online Extrinsic Parameters Calibration for Stereovision Systems Used in Far-range Detection Vehicle Applications. IEEE Trans. Intell. Transportation Systems, Vol. 8, 4 (2007). https://doi.org/10.1109/TITS.2007.908576 Google ScholarDigital Library
- Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An Efficient Alternative to SIFT or SURF. International Conference on Computer Vision (ICCV'11) (2011), IEEE, Barcelona, Spain, 2564--2571. https://doi.org/10.1109/ICCV.2011.6126544 Google ScholarDigital Library
- Chunxiang Wang, Hengrun Zhang, Ming Yang, Xudong Wang, Lei Ye, and Chunzhao Guo. 2014. Automatic Parking based on a Bird's Eye View Vision System. Advances in Mechanical Engineering, Vol. 6 (2014), 847406:1--13. https://doi.org/10.1155/2014/847406Google Scholar
- R. W. M. Wedderburn. 1974. Quasi-Likelihood Functions, Generalized Linear Models, and the Gauss-Newton Method. Biometrika, Vol. 61, 3 (1974), 439--447. https://doi.org/10.2307/2334725Google Scholar
- Jin Xu, Guang Chen, and Ming Xie. 2000. Vision-guided Automatic Parking for Smart Car. In IEEE Intelligent Vehicles Symposium (IVS'00). IEEE, Dearborn, MI, USA, 725--730. https://doi.org/10.1109/IVS.2000.898435Google Scholar
- Lin Zhang, Junhao Huang, Xiyuan Li, and Lu Xiong. 2018. Vision-based Parking-slot Detection: A DCNN-based Approach and a Large-scale Benchmark Dataset. IEEE Trans. Image Processing, Vol. 27, 11 (2018), 5350--5364. https://doi.org/10.1109/TIP.2018.2857407 Google ScholarDigital Library
- Tianjun Zhang, Lin Zhang, Ying Shen, Yong Ma, Shengjie Zhao, and Yicong Zhou. 2020. OECS: Towards Online Extrinsics Correction for The Surround-view System. In IEEE International Conference on Multimedia and Expo (ICME'20). IEEE, London, UK, 1--6. https://doi.org/10.1109/ICME46284.2020.9102803Google Scholar
- Kun Zhao, Uri Iurgel, Mirko Meuter, and Josef Pauli. 2014. An Automatic Online Camera Calibration System for Vehicular Applications. In IEEE Conference on Intelligent Transportation Systems (ITSC'14). IEEE, Qingdao, China, 1490--1492. https://doi.org/10.1109/ITSC.2014.6957643Google Scholar
Index Terms
- ROECS: A Robust Semi-direct Pipeline Towards Online Extrinsics Correction of the Surround-view System
Recommendations
Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach
The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. ...
Online Camera Pose Optimization for the Surround-view System
MM '19: Proceedings of the 27th ACM International Conference on MultimediaSurround-view system is an important information medium for drivers to monitor the driving environment. A typical surround-view system consists of four to six fish-eye cameras arranged around the vehicle. From these camera inputs, a top-down image of ...
Direct semi-dense SLAM for rolling shutter cameras
2016 IEEE International Conference on Robotics and Automation (ICRA)In this paper, we present a monocular Direct and Semi-dense SLAM (Simultaneous Localization And Mapping) system for rolling shutter cameras. In a rolling shutter camera, the pose is different for each row of each image, and this yields poor pose estimates ...
Comments