Pingping Cai
Skip Abstract Section
Abstract
3D Point Cloud Data (PCD) is an efficient machine representation for surrounding environments and has been used in many applications. But the measured PCD is often incomplete and sparse due to the sensor occlusion and poor lighting conditions. To automatically reconstruct complete PCD from the incomplete ones, we propose DeepPCD, a deep-learning-based system that reconstructs both geometric and color information for large indoor environments. For geometric reconstruction, DeepPCD uses a novel patch based technique that splits the PCD into multiple parts, approximates, extends, and independently reconstructs the parts by 3D planes, and then merges and refines them. For color reconstruction, DeepPCD uses a conditional Generative Adversarial Network to infer the missing color of the geometrically reconstructed PCD by using the color feature extracted from incomplete color PCD. We experimentally evaluate DeepPCD with several real PCD collected from large, diverse indoor environments and explore the feasibility of PCD autocompletion in various ubiquitous sensing applications.
- B.-S. Kim, P. Kohli, and S. Savarese, "3D Scene Understanding by Voxel-CRF," in IEEE International Conference on Computer Vision (ICCV), 2013.Google Scholar
- A. Vincent, "A 3D Perception System for the Mobile Robot Hilare," in IEEE International Conference on Robotics and Automation, 1986.Google Scholar
- T. Vieville, E. Clergue, R. Enciso, and H. Mathieu, "Experimenting with 3D Vision on a Robotic Head," Robotics and Autonomous Systems, vol. 14, no. 1, 1995.Google Scholar
- H. Zhang, G. Wang, Z. Lei, and J.-N. Hwang, "Eye in the Sky: Drone-Based Object Tracking and 3D Localization," in ACM International Conference on Multimedia, 2019.Google Scholar
- Y. Zeng, Y. Hu, S. Liu, J. Ye, Y. Han, X. Li, and N. Sun, "RT3D: Real-Time 3-D Vehicle Detection in LiDAR Point Cloud for Autonomous Driving," IEEE Robotics and Automation Letters, vol. 3, no. 4, 2018.Google Scholar
- K. C. Kwan and H. Fu, "Mobi3DSketch: 3D Sketching in Mobile AR," in ACM CHI, 2019.Google ScholarDigital Library
- Y. Zhou and O. Tuzel, "VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection," in IEEE/CVF CVPR, 2018.Google Scholar
- C. Gotsman, X. Gu, and A. Sheffer, "Fundamentals of Spherical Parameterization for 3D Meshes," in ACM SIGGRAPH, 2003.Google Scholar
- L. Linsen, "Point Cloud Representation," Karlsruhe Institute of Technology, Germany, Tech. Rep., 2001.Google Scholar
- X. Li, S. Du, G. Li, and H. Li, "Integrate Point-Cloud Segmentation with 3D LiDAR Scan-Matching for Mobile Robot Localization and Mapping," MDPI Sensors, vol. 20, no. 1, 2020.Google Scholar
- R. Radkowski, "Object Tracking with a Range Camera for Augmented Reality Assembly Assistance," Journal of Computing and Information Science in Engineering, vol. 16, no. 1, 2016.Google ScholarCross Ref
- Q. Wang and M.-K. Kim, "Applications of 3D Point Cloud Data in the Construction Industry: A Fifteen-Year Review from 2004 to 2018," Advanced Engineering Informatics, vol. 39, 2019.Google Scholar
- S. B. Walsh, D. J. Borello, B. Guldur, and J. F. Hajjar, "Data Processing of Point Clouds for Object Detection for Structural Engineering Applications," Computer-Aided Civil and Infrastructure Engineering, vol. 28, no. 7, 2013.Google Scholar
- D. A. White, "LIDAR, Point Clouds, and Their Archaeological Applications," in Mapping Archaeological Landscapes from Space, 2013.Google Scholar
- M. Labbe and F. Michaud, "RTAB-Map as an Open-Source Lidar and Visual Simultaneous Localization and Mapping Library for Large-Scale and Long-Term Online Operation," Journal of Field Robotics, vol. 36, no. 2, 2019.Google Scholar
- Y. Xiao and Y. Taguchi and V. R. Kamat, "Coupling Point Cloud Completion and Surface Connectivity Relation Inference for 3D Modeling of Indoor Building Environments," Journal of Computing in Civil Engineering, vol. 32, no. 5, 2018.Google Scholar
- A. Dai, C. Ruizhongtai Q., and M. Niessner, "Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis," in IEEE/CVF CVPR, 2017.Google Scholar
- W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, "PCN: Point Completion Network," in International Conference on 3D Vision (3DV), 2018.Google Scholar
- H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun, "GRNet: Gridding Residual Network for Dense Point Cloud Completion," in European Conference on Computer Vision: Computer Vision - ECCV, 2020.Google Scholar
- M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu, "Morphing and Sampling Network for Dense Point Cloud Completion," AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020.Google Scholar
- A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, "ShapeNet: An Information-Rich 3D Model Repository," https://shapenet.org/, 2015.Google Scholar
- L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, "TopNet: Structural Point Cloud Decoder," in IEEE/CVF CVPR, 2019.Google Scholar
- Y. Yang, C. Feng, Y. Shen, and D. Tian, "FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation," in IEEE/CVF CVPR, 2018.Google Scholar
- C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space," in Advances in Neural Information Processing Systems (NIPS), 2017.Google Scholar
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," in International Conference on Learning Representations (ICLR), 2021.Google Scholar
- H. Fan, H. Su, and L. Guibas, "A Point Set Generation Network for 3D Object Reconstruction from a Single Image," in IEEE/CVF CVPR, 2017.Google Scholar
- P. Xiang, X. Wen, Y.-S. Liu, Y.-P. Cao, P. Wan, W. Zheng, and Z. Han, "SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.Google Scholar
- S. Sur, "DeepPCD Project," 2022. [Online]. Available: https://syrex.cse.sc.edu/research/ubiquitous-sensing/deeppcd/Google Scholar
- H. Sobreira, C. M. Costa, I. Sousa, L. Rocha, J. Lima, P. C. Farias, P. Costa, and A. P. Moreira, "Map-matching algorithms for robot self-localization: A comparison between perfect match, iterative closest point and normal distributions transform," J. Intell. Robotics Syst., vol. 93, 2019.Google ScholarDigital Library
- Z. Liu, H. Tang, Y. Lin, and S. Han, "Point-Voxel CNN for Efficient 3D Deep Learning," in Advances in Neural Information Processing Systems (NIPS), 2019.Google Scholar
- W. Shi and R. Rajkumar, "Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud," in IEEE/CVF CVPR, 2020.Google Scholar
- S. I. Kabanikhin, "Definitions and Examples of Inverse and Ill-posed Problems," Journal of Inverse and Ill-posed Problems, vol. 16, no. 4, 2008.Google ScholarCross Ref
- Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based Learning Applied to Document Recognition," Proceedings of the IEEE, vol. 86, no. 11, 1998.Google Scholar
- Y. Liu, B. Fan, S. Xiang, and C. Pan, "Relation-Shape Convolutional Neural Network for Point Cloud Analysis," in IEEE/CVF CVPR, 2019.Google Scholar
- S. Ji, W. Xu, M. Yang, and K. Yu, "3D Convolutional Neural Networks for Human Action Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, 2013.Google Scholar
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is All You Need," in Advances in Neural Information Processing Systems (NIPS), 2017.Google Scholar
- X. Cao and K. Nagao, "Point Cloud Colorization Based on Densely Annotated 3D Shape Dataset," in MMM, 2019.Google Scholar
- J. Liu, S. Dai, and X. Li, "PCCN: Point Cloud Colorization Network," in IEEE International Conference on Image Processing (ICIP), 2019.Google Scholar
- T. Shinohara, H. Xiu, and M. Matsuoka, "Point2Color: 3D Point Cloud Colorization Using a Conditional Generative Network and Differentiable Rendering for Airborne LiDAR," in IEEE/CVF CVPR Workshops, 2021.Google Scholar
- M. Mirza and S. Osindero, "Conditional Generative Adversarial Nets," 2014. [Online]. Available: https://arxiv.org/abs/1411.1784Google Scholar
- I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative Adversarial Nets," in Advances in Neural Information Processing Systems (NIPS), 2014.Google ScholarDigital Library
- AsusTek Computer Inc., "Zenfone AR: Go Beyond Reality," 2021. [Online]. Available: https://www.asus.com/us/Phone/ZenFone-ARZS571KL/Google Scholar
- IntRoLab, "Real-Time Appearance-Based Mappingy," 2021. [Online]. Available: http://introlab.github.io/rtabmap/Google Scholar
- P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and G. Ranzuglia, "MeshLab: an Open-Source Mesh Processing Tool," in Eurographics Italian Chapter Conference, 2008.Google Scholar
- I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, "3D Semantic Parsing of Large-Scale Indoor Spaces," in IEEE/CVF CVPR, 2016.Google Scholar
- R. Bridson, "Fast Poisson Disk Sampling in Arbitrary Dimensions," in ACM SIGGRAPH 2007 Sketches, 2007.Google Scholar
- Open-Source, "PyTroch," 2021. [Online]. Available: https://pytorch.org/Google Scholar
- NVIDIA, "RTX A6000," 2021. [Online]. Available: https://www.nvidia.com/en-us/design-visualization/rtx-a6000/Google Scholar
- L. Li, F. Yang, H. Zhu, D. Li, Y. Li, and L. Tang, "An Improved RANSAC for 3D Point Cloud Plane Segmentation Based on Normal Distribution Transformation Cells," Remote Sensing, vol. 9, no. 5, 2017.Google Scholar
- Geodan, "Generate Synthetic Points to Fill Holes in Point Clouds," 2020. [Online]. Available: https://github.com/Geodan/fill-holespointcloudGoogle Scholar
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, 2004.Google Scholar
- Q. Yang, H. Chen, Z. Ma, Y. Xu, R. Tang, and J. Sun, "Predicting the perceptual quality of point cloud: A 3d-to-2d projection-based exploration," IEEE Transactions on Multimedia, vol. 23, 2021.Google Scholar
- M. A. Fischler and R. C. Bolles, "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography," Commun. ACM, vol. 24, no. 6, 1981.Google ScholarDigital Library
- M. Pauly, N. J. Mitra, J. Wallner, H. Pottmann, and L. J. Guibas, "Discovering Structural Regularity in 3D Geometry," in ACM SIGGRAPH, 2008.Google Scholar
- P. Speciale, M. R. Oswald, A. Cohen, and M. Pollefeys, "A Symmetry Prior for Convex Variational 3D Reconstruction," in European Conference on Computer Vision: Computer Vision - ECCV, 2016.Google Scholar
- Y. Li, A. Dai, L. Guibas, and M. NieBner, "Database-Assisted Object Retrieval for Real-Time 3D Reconstruction," Computer Graphics Forum, 2015.Google Scholar
- V. G. Kim, W. Li, N. J. Mitra, S. Chaudhuri, S. DiVerdi, and T. Funkhouser, "Learning Part-Based Templates from Large Collections of 3D Shapes," ACM Transactions on Graphics, vol. 32, no. 70, 2013.Google Scholar
- X. Zhang, Y. Feng, S. Li, C. Zou, H. Wan, X. Zhao, Y. Guo, and Y. Gao, "View-Guided Point Cloud Completion," in IEEE/CVF CVPR, 2021.Google Scholar
- X. Wang, J. Ang, Marcelo H, and G. H. Lee, "Point Cloud Completion by Learning Shape Priors," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.Google Scholar
- A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Niebner, "ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans," in IEEE/CVF CVPR, 2018.Google Scholar
- Open-Source, "Deploy Machine Learning Models on Mobile and IoT Devices," 2022. [Online]. Available: https://www.tensorflow.org/liteGoogle Scholar
- Open-Source, "PyTorch Mobile," 2022. [Online]. Available: https://pytorch.org/mobile/home/Google Scholar
Index Terms
- DeepPCD: Enabling AutoCompletion of Indoor Point Clouds with Deep Learning
Recommendations
PalGAN: Image Colorization with Palette Generative Adversarial Networks
Computer Vision – ECCV 2022AbstractMultimodal ambiguity and color bleeding remain challenging in colorization. To tackle these problems, we propose a new GAN-based colorization approach PalGAN, integrated with palette estimation and chromatic attention. To circumvent the ...
AutoPCD: Learning-Augmented Indoor Point Cloud Completion
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers3D Point Cloud (PCD) is an efficient machine representation for surrounding environments and has been used in many applications. But a fast reconstruction of complete PCD for large environments remains a challenge. We propose AutoPCD, a machine-learning ...
Automatic Video Colorization Using 3D Conditional Generative Adversarial Networks
Advances in Visual ComputingAbstractIn this work, we present a method for automatic colorization of grayscale videos. The core of the method is a Generative Adversarial Network that is trained and tested on sequences of frames in a sliding window manner. Network convolutional and ...
Comments