Abstract
3D generation has witnessed significant advancements, yet efficiently producing high-quality 3D assets from a single image remains challenging. In this paper, we present a triplane autoencoder, which encodes 3D models into a compact triplane latent space to effectively compress both the 3D geometry and texture information. Within the autoencoder framework, we introduce a 3D-aware cross-attention mechanism, which utilizes low-resolution latent representations to query features from a high-resolution 3D feature volume, thereby enhancing the representation capacity of the latent space. Subsequently, we train a diffusion model on this refined latent space. In contrast to solely relying on image embedding for 3D generation, our proposed method advocates for the simultaneous utilization of both image embedding and shape embedding as conditions. Specifically, the shape embedding is estimated via a diffusion prior model conditioned on the image embedding. Through comprehensive experiments, we demonstrate that our method outperforms state-of-the-art algorithms, achieving superior performance while requiring less training data and time. Our approach enables the generation of high-quality 3D assets in merely 7 s on a single A100 GPU. More results and visualization can be found on our project page: https://compress3d.github.io/.
B. Zhang—Work done during the internship at IDEA.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, Z., Wang, F., Liu, H.: Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.16585 (2023)
Cheng, Y.C., Lee, H.Y., Tulyakov, S., Schwing, A.G., Gui, L.Y.: Sdfusion: multimodal 3d shape completion, reconstruction, and generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4456–4465 (2023)
Deitke, M., et al.: Objaverse-xl: a universe of 10m+ 3d objects. Adv. Neural Inf. Process. Syst. 36 (2024)
Deitke, M., et al.: Objaverse: a universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142–13153 (2023)
Gupta, A., Xiong, W., Nie, Y., Jones, I., Oğuz, B.: 3dgen: triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371 (2023)
He, Z., Wang, T.: Openlrm: open-source large reconstruction models (2023). https://github.com/3DTopia/OpenLRM
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)
Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
Hong, Y., et al.: Lrm: large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023)
Jun, H., Nichol, A.: Shap-e: generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 (2023)
Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4), 139-1 (2023)
Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primitives for high-performance differentiable rendering. ACM Trans. Graph. 39(6) (2020)
Li, M., Duan, Y., Zhou, J., Lu, J.: Diffusion-sdf: text-to-shape via voxelized diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12642–12651 (2023)
Li, R., Li, X., Hui, K.H., Fu, C.W.: Sp-gan: sphere-guided 3d shape generation and manipulation. ACM Trans. Graph. (TOG) 40(4), 1–12 (2021)
Li, S., et al.: Instant-3d: instant neural radiance field training towards on-device ar/vr 3d reconstruction. In: Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1–13 (2023)
Lin, C.H., et al.: Magic3d: high-resolution text-to-3d content creation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300–309 (2023)
Liu, M., et al.: Openshape: scaling up 3d shape representation towards open-world understanding. Adv. Neural Inf. Process. Syst. 36 (2024)
Liu, M., et al.: One-2-3-45: any single image to 3d mesh in 45 seconds without per-shape optimization. Adv. Neural Inf. Process. Syst. 36 (2024)
Liu, R., et al.: Zero-1-to-3: zero-shot one image to 3d object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298–9309 (2023)
Liu, Z., Feng, Y., Black, M.J., Nowrouzezahrai, D., Paull, L., Liu, W.: Meshdiffusion: score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08133 (2023)
Mercier, A., et al.: Hexagen3d: stablediffusion is just one step away from fast and diverse text-to-3d generation. arXiv preprint arXiv:2401.07727 (2024)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)
Mittal, P., Cheng, Y.C., Singh, M., Tulsiani, S.: Autosdf: shape priors for 3d completion, reconstruction and generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 306–315 (2022)
Nash, C., Ganin, Y., Eslami, S.A., Battaglia, P.: Polygen: an autoregressive generative model of 3d meshes. In: International Conference on Machine Learning, pp. 7220–7229. PMLR (2020)
Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-e: a system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 (2022)
Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)
Schaefer, S., Warren, J.: Dual marching cubes: primal contouring of dual grids. In: 12th Pacific Conference on Computer Graphics and Applications, 2004. PG 2004. Proceedings, pp. 70–76. IEEE (2004)
Schuhmann, C., et al.: Laion-5b: an open large-scale dataset for training next generation image-text models. Adv. Neural. Inf. Process. Syst. 35, 25278–25294 (2022)
Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Adv. Neural. Inf. Process. Syst. 34, 6087–6101 (2021)
Shen, T., et al.: Flexible isosurface extraction for gradient-based mesh optimization. ACM Trans. Graph. (TOG) 42(4), 1–16 (2023)
Shim, J., Kang, C., Joo, K.: Diffusion-based signed distance fields for 3d shape generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20887–20897 (2023)
Shue, J.R., Chan, E.R., Po, R., Ankner, Z., Wu, J., Wetzstein, G.: 3d neural field generation using triplane diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20875–20886 (2023)
Siddiqui, Y., et al.: Meshgpt: generating triangle meshes with decoder-only transformers. arXiv preprint arXiv:2311.15475 (2023)
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: Lgm: large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 (2024)
Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 (2023)
Tang, J., et al.: Make-it-3d: high-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184 (2023)
Wang, T., et al.: Rodin: a generative model for sculpting 3d digital avatars using diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4563–4573 (2023)
Wu, L., et al.: Fast point cloud generation with straight flows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9445–9454 (2023)
Yi, T., et al.: Gaussiandreamer: fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529 (2023)
Zeng, X., et al.: Lion: latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978 (2022)
Zhang, B., Nießner, M., Wonka, P.: 3dilg: irregular latent grids for 3d generative modeling. Adv. Neural. Inf. Process. Syst. 35, 21871–21885 (2022)
Zhang, B., Tang, J., Niessner, M., Wonka, P.: 3dshape2vecset: a 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445 (2023)
Zou, Z.X., et al.: Triplane meets gaussian splatting: fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 (2023)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (62072366, U23A20312).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, B., Yang, T., Li, Y., Zhang, L., Zhao, X. (2025). Compress3D: A Compressed Latent Space for 3D Generation from a Single Image. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15076. Springer, Cham. https://doi.org/10.1007/978-3-031-72649-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-72649-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72648-4
Online ISBN: 978-3-031-72649-1
eBook Packages: Computer ScienceComputer Science (R0)