Yizhen Lao
ABSTRACT
To augment the TV show in post-production, we propose a novel solution to uncalibrated camera small motion tracking in a dynamic scene that simultaneously reconstructs the sparse 3D scene and computes camera poses and focal lengths of each frame. The critical elements of our approach are a robust image feature tracking strategy in dynamic scenes followed by automatic local-window frames slicing, local and global bundle adjustment optimization initialized by a homography-based uncalibrated relative rotation solver. The proposed method allows us to add the virtual objects (elements) into the reconstructed 3D scene, then composite them back into the original shot while perfectly matched perspective and appear seamless.
The evaluation of a large variety of real TV show sequences demonstrates the merits of our method against state-of-the-art works and commercial software products.
- [n.d.]. Vicon Boujou, howpublished = http://www.vicon/,.Google Scholar
- Alastair Barber, Darren Cosker, Oliver James, Ted Waine, and Radhika Patel. 2016. Camera tracking in visual effects an industry perspective of structure from motion. In Symposium on Digital Production. Google ScholarDigital Library
- Matthew Brown, David G Lowe, et al. 2003. Recognising panoramas.. In ICCV. Google ScholarDigital Library
- Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV.Google Scholar
- Mark Christiansen. 2013. Adobe After Effects CC Visual Effects and Compositing Studio Techniques. Adobe Press. Google ScholarDigital Library
- Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2018. Superpoint: Self-supervised interest point detection and description. In CVPR workshops.Google ScholarCross Ref
- Tim Dobbert. 2006. Matchmoving: the invisible art of camera tracking. John Wiley & Sons. Google ScholarDigital Library
- Hyowon Ha, Sunghoon Im, Jaesik Park, Hae-Gon Jeon, and In So Kweon. 2016. High-quality depth from uncalibrated small motion clip. In CVPR.Google Scholar
- Jungong Han, Dirk Farin, and Peter HN de With. 2007. A real-time augmentedreality system for sports broadcast video enhancement. In ACM MM. Google ScholarDigital Library
- Christopher G Harris, Mike Stephens, et al. 1988. A combined corner and edge detector.. In Alvey vision conference.Google Scholar
- Juan David Hincapié-Ramos, Kasim Ozacar, Pourang P Irani, and Yoshifumi Kitamura. [n.d.]. GyroWand: IMU-based raycasting for augmented reality headmounted displays. In ACM Symposium on Spatial User Interaction. Google ScholarDigital Library
- Sunghoon Im, Hyowon Ha, Gyeongmin Choe, Hae-Gon Jeon, Kyungdon Joo, and In So Kweon. 2015. High quality structure from small motion for rolling shutter cameras. In ICCV. Google ScholarDigital Library
- Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla. 2008. Automatic generator of minimal problem solvers. In ECCV. Google ScholarDigital Library
- Kiriakos N Kutulakos and James R Vallino. 1998. Calibration-free augmented reality. TVCG (1998). Google ScholarDigital Library
- Viktor Larsson, Kalle Astrom, and Magnus Oskarsson. 2017. Efficient solvers for minimal problems by syzygy-based reduction. In CVPR.Google Scholar
- Zahid Mahmood, Tauseef Ali, Nazeer Muhammad, Nargis Bibi, Imran Shahzad, and Shoaib Azmat. 2017. EAR: Enhanced Augmented Reality System for Sports Entertainment Applications. KSII Transactions on Internet & Information Systems (2017).Google Scholar
- Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet. 2016. Openmvg: Open multiple view geometry. In International Workshop on Reproducible Research in Pattern Recognition. Springer, 60--74.Google Scholar
- Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. 2015. ORBSLAM: a versatile and accurate monocular SLAM system. IEEE transactions on robotics (2015).Google Scholar
- David Nistér. 2004. An efficient solution to the five-point relative pose problem. TPAMI (2004). Google ScholarDigital Library
- Rahul Raguram, Ondrej Chum, Marc Pollefeys, Jiri Matas, and Jan-Michael Frahm. 2012. USAC: a universal framework for random sample consensus. TPAMI (2012). Google ScholarDigital Library
- Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018).Google Scholar
- Johannes L Schonberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In CVPR.Google Scholar
- Gerald Schweighofer, Sinisa Segvic, and Axel Pinz. 2008. Online/realtime structure and motion for general camera models. In 2008 IEEE Workshop on Applications of Computer Vision. Google ScholarDigital Library
- Yongduek Seo and Ki Sang Hong. 2000. Calibration-free augmented reality in perspective. TVCG (2000). Google ScholarDigital Library
- Dave Shreiner, Graham Sellers, John Kessenich, and Bill Licea-Kane. 2013. OpenGL programming guide: The Official guide to learning OpenGL, version 4.3. Google ScholarDigital Library
- RA Smith, Andrew W Fitzgibbon, and Andrew Zisserman. 1999. Improving Augmented Reality using Image and Scene Constraints.. In BMVC.Google Scholar
- Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda. 2017. Visual SLAM algorithms: a survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications (2017).Google Scholar
- Graham Thomas. 2006. Mixed reality techniques for TV and their application for on-set/pre-visualisation in film production. In International Workshop on Mixed Reality Technology for Filmmaking.Google Scholar
- Carlo Tomasi and T Kanade Detection. 1991. Tracking of point features. IJCV (1991).Google Scholar
- Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. 1999. Bundle adjustment-a modern synthesis. In International workshop on vision algorithms. Google ScholarDigital Library
- Changchang Wu et al. 2011. VisualSFM: A visual structure from motion system. (2011).Google Scholar
- Ruichao Xiao, Wenxiu Sun, Jiahao Pang, Qiong Yan, and Jimmy Ren. 2018. Dsr: Direct self-rectification for uncalibrated dual-lens cameras. In 3DV.Google Scholar
- Fisher Yu and David Gallup. 2014. 3d reconstruction from accidental motion. In CVPR. Google ScholarDigital Library
- Jung-Jae Yu and Jae-Hean Kim. 2010. Camera motion tracking in a dynamic scene. In 2010 IEEE International Symposium on Mixed and Augmented Reality.Google ScholarCross Ref
- Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong, and Hujun Bao. 2009. Consistent depth maps recovery from a video sequence. TPAMI (2009). Google ScholarDigital Library
- Richard Hartley Andrew Zisserman. 2004. Multiple view geometry in computer vision. (2004). Google ScholarDigital Library
- Stefanie Zollmann, Tobias Langlotz, Moritz Loos, Wei Hong Lo, and Lewis Baker. 2019. Arspectator: Exploring augmented reality for sport events. In SIGGRAPH Asia. Google ScholarDigital Library
Index Terms
- Augmenting TV Shows via Uncalibrated Camera Small Motion Tracking in Dynamic Scene
Recommendations
Efficient, causal camera tracking in unprepared environments
This paper addresses the problem of tracking the 3D pose of a camera in space, using the images it acquires while moving freely in unmodeled, arbitrary environments. A novel feature-based approach for camera tracking is proposed, intended to facilitate ...
Hand-Eye Camera Calibration with an Optical Tracking System
ICDSC '18: Proceedings of the 12th International Conference on Distributed Smart CamerasThis paper presents a method for hand-eye camera calibration via an optical tracking system (OTS) faciltating robotic applications. The camera pose cannot be directly tracked via the OTS. Because of this, a transformation matrix between a marker-plate ...
Real-time camera pose estimation via line tracking
Real-time camera calibration has been intensively studied in augmented reality. However, for texture-less and texture-repeated scenes as well as poorly illuminated scenes, obtaining a stable calibration is still an open problem. In the paper, we propose ...
Comments