research-article

iOrthoPredictor: model-guided deep prediction of teeth alignment

Authors:

Youyi ZhengAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 39, Issue 6

Article No.: 216, Pages 1 - 15

https://doi.org/10.1145/3414685.3417771

Published: 27 November 2020 Publication History

Abstract

In this paper, we present iOrthoPredictor, a novel system to visually predict teeth alignment in photographs. Our system takes a frontal face image of a patient with visible malpositioned teeth along with a corresponding 3D teeth model as input, and generates a facial image with aligned teeth, simulating a real orthodontic treatment effect. The key enabler of our method is an effective disentanglement of an explicit representation of the teeth geometry from the in-mouth appearance, where the accuracy of teeth geometry transformation is ensured by the 3D teeth model while the in-mouth appearance is modeled as a latent variable. The disentanglement enables us to achieve fine-scale geometry control over the alignment while retaining the original teeth appearance attributes and lighting conditions. The whole pipeline consists of three deep neural networks: a U-Net architecture to explicitly extract the 2D teeth silhouette maps representing the teeth geometry in the input photo, a novel multilayer perceptron (MLP) based network to predict the aligned 3D teeth model, and an encoder-decoder based generative model to synthesize the in-mouth appearance conditional on the original teeth appearance and the aligned teeth geometry. Extensive experimental results and a user study demonstrate that iOrthoPredictor is effective in qualitatively predicting teeth alignment, and applicable to the orthodontic industry.

Supplementary Material

ZIP File (a216-yang.zip)

Supplemental material.

Download
58.67 MB

MP4 File (3414685.3417771.mp4)

Presentation video

Download
19.04 MB

References

[1]

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018. Learning representations and generative models for 3D point clouds. In International Conference on Machine Learning (ICML). 40--49.

[2]

Yazeed Alharbi, Neil Smith, and Peter Wonka. 2019. Latent Filter Scaling for Multimodal Unsupervised Image-to-Image Translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1458--1466.

[3]

Matthew Amodio and Smita Krishnaswamy. 2019. TraVeLGAN: Image-to-image Translation by Transformation Vector Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8983--8992.

[4]

Moab Arar, Yiftach Ginger, Dov Danon, Ilya Leizerson, Amit Bermano, and Daniel Cohen-Or. 2020. Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation. arXiv preprint arXiv:2003.08073 (2020).

[5]

Milton B Asbell. 1990. A brief history of orthodontics. American Journal of Orthodontics and Dentofacial Orthopedics 98, 2 (1990), 176--183.

[6]

Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. 2001. Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing 10, 8 (2001), 1200--1211.

Digital Library

[7]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. In ACM Transactions on Graphics (ToG), Vol. 28. ACM, 24:1--24:11.

[8]

Connelly Barnes, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2010. The generalized PatchMatch correspondence algorithm. In European Conference on Computer Vision. Springer, 29--43.

[9]

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image Inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 417--424.

Digital Library

[10]

Volker Blanz, Curzio Basso, Tomaso Poggio, and Thomas Vetter. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3 (2003), 641--650.

[11]

Chen Cao, Qiming Hou, and Kun Zhou. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (ToG) 33, 4 (2014), 43.

Digital Library

[12]

Duygu Ceylan, Niloy J. Mitra, Youyi Zheng, and Mark Pauly. 2014. Coupled structure-from-motion and 3D symmetry detection for urban facades. ACM Trans. Graph. 33, 1, Article Article 2 (Feb. 2014), 15 pages.

Digital Library

[13]

Menglei Chai, Tianjia Shao, Hongzhi Wu, Yanlin Weng, and Kun Zhou. 2016. AutoHair: Fully automatic hair modeling from a single image. ACM Trans. Graph. 35, 4 (2016), 116:1--116:12.

Digital Library

[14]

Huiwen Chang, Jingwan Lu, Fisher Yu, and Adam Finkelstein. 2018. PairedCycleGAN: Asymmetric style transfer for applying and removing makeup. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 40--48.

[15]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789--8797.

[16]

Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing 13, 9 (2004), 1200--1212.

Digital Library

[17]

Zhiming Cui, Changjian Li, and Wenping Wang. 2019. ToothNet: Automatic tooth instance segmentation and identification from cone beam CT images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6368--6377.

[18]

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision. 764--773.

[19]

Kevin Dale, Kalyan Sunkavalli, Micah K Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video face replacement. ACM Trans. Graph. 30, 6 (2011), 130:1--130:10.

Digital Library

[20]

Yue Deng, Qionghai Dai, and Zengke Zhang. 2011. Graph Laplace for occluded face completion and recognition. IEEE Transactions on Image Processing 20, 8 (2011), 2329--2338.

Digital Library

[21]

Hui Ding, Kumar Sricharan, and Rama Chellappa. 2018. ExprGAN: Facial expression editing with controllable expression intensity. In Thirty-Second AAAI Conference on Artificial Intelligence.

[22]

Iddo Drori, Daniel Cohen-Or, and Hezy Yeshurun. 2003. Fragment-based image completion. ACM Transactions on Graphics (ToG) 22, 3 (2003), 303--312.

Digital Library

[23]

Alexei A Efros and William T Freeman. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 341--346.

Digital Library

[24]

Alexei A Efros and Thomas K Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, Vol. 2. IEEE, 1033--1038.

Digital Library

[25]

Patrick Esser, Ekaterina Sutter, and Björn Ommer. 2018. A variational U-Net for conditional appearance and shape generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8857--8866.

[26]

Yaroslav Ganin, Daniil Kononenko, Diana Sungatullina, and Victor Lempitsky. 2016. Deepwarp: Photorealistic image resynthesis for gaze manipulation. In European Conference on Computer Vision (ECCV). Springer, 311--326.

[27]

Pablo Garrido, Levi Valgaerts, Hamid Sarmadi, Ingmar Steiner, Kiran Varanasi, Patrick Perez, and Christian Theobalt. 2015. VDub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. Computer Graphics Forum 34, 2 (2015), 193--204.

Digital Library

[28]

Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou. 2018. Warp-guided GANs for single-photo facial animation. ACM Trans. Graph. 37, 6, Article 231 (2018), 12 pages.

Digital Library

[29]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS). 2672--2680.

[30]

Lee W Graber, Robert L Vanarsdall, Katherine WL Vig, and Greg J Huang. 2016. Orthodontics: current principles and techniques. Elsevier Health Sciences.

[31]

Qiao Gu, Guanzhi Wang, Mang Tik Chiu, Yu-Wing Tai, and Chi-Keung Tang. 2019. LADN: Local adversarial disentangling network for facial makeup and de-makeup. Proceedings of the IEEE International Conference on Computer Vision (2019).

[32]

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in neural information processing systems. 5767--5777.

[33]

James Hays and Alexei A Efros. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics (ToG) 26, 3 (2007), 4-es.

Digital Library

[34]

Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. 2019. AttGAN: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing (2019).

[35]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems. 6626--6637.

[36]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. IEEE Conference on Computer Vision and Pattern Recognition.

[37]

Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. 2014. Image completion using planar structure guidance. ACM Transactions on Graphics (ToG) 33, 4 (2014), 129.

Digital Library

[38]

Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision. 1501--1510.

[39]

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV). 172--189.

Digital Library

[40]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36, 4 (2017), 107.

Digital Library

[41]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125--1134.

[42]

Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). 2017--2025.

[43]

Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. 2019. Neural style transfer: A review. IEEE transactions on visualization and computer graphics (2019).

[44]

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2019. Analyzing and Improving the Image Quality of StyleGAN. CoRR abs/1912.04958 (2019).

[45]

Masahide Kawai, Tomoyori Iwao, Daisuke Mima, Akinobu Maejima, and Shigeo Morishima. 2013. Photorealistic inner mouth expression in speech animation. In ACM SIGGRAPH 2013 Posters. ACM, 9:1--9:1.

Digital Library

[46]

Masahide Kawai, Tomoyori Iwao, Daisuke Mima, Akinobu Maejima, and Shigeo Morishima. 2014. Data-driven speech animation synthesis focusing on realistic inside of the mouth. Journal of information processing 22, 2 (2014), 401--409.

[47]

Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. 2018. Deep video portraits. ACM Trans. Graph. 37, 4, Article 163 (2018), 14 pages.

Digital Library

[48]

Diederik Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In ICLR.

[49]

Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).

[50]

Norman William Kingsley. 1880. A treatise on oral deformities as a branch of mechanical surgery. D. Appleton.

[51]

Rolf M Koch, Markus H Gross, Friedrich R Carls, Daniel F von Büren, George Fankhauser, and Yoav IH Parish. 1996. Simulating facial surgery using finite element models. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 421--428.

Digital Library

[52]

Rolf Köhler, Christian Schuler, Bernhard Schölkopf, and Stefan Harmeling. 2014. Mask-specific inpainting with deep neural networks. In German Conference on Pattern Recognition. Springer, 523--534.

[53]

Johannes Kopf, Wolf Kienzle, Steven Drucker, and Sing Bing Kang. 2012. Quality prediction for image completion. ACM Transactions on Graphics (ToG) 31, 6 (2012), 131.

Digital Library

[54]

Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. 2017. Fast face-swap using convolutional neural networks. In The IEEE International Conference on Computer Vision. 3697--3705.

[55]

Claudia Kuster, Tiberiu Popa, Jean-Charles Bazin, Craig Gotsman, and Markus Gross. 2012. Gaze correction for home video conferencing. ACM Trans. Graph. 31, 6 (2012), 174:1--174:6.

Digital Library

[56]

Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics (ToG) 24, 3 (2005), 795--802.

Digital Library

[57]

Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DE-NOYER, et al. 2017. Fader Networks: Manipulating Images by Sliding Attributes. In Advances in Neural Information Processing Systems. 5963--5972.

[58]

Anat Levin, Assaf Zomet, and Yair Weiss. 2003. Learning how to inpaint from global image statistics. In International Conference on Computer Vision. IEEE, 305--312.

[59]

Zhanli Li, Jingding Fu, Hongan Li, Kang Zhou, and Qiaojuan Hui. 2019. Automatic arrangement method of misaligned teeth in virtual orthodontic treatment. In Journal of Graphics (in Chinese), Vol. 40. 225--234.

[60]

Ming-Yu Liu, Thomas Breuel, and Jan Kautz. 2017. Unsupervised image-to-image translation networks. In Advances in neural information processing systems. 700--708.

[61]

Si Liu, Xinyu Ou, Ruihe Qian, Wei Wang, and Xiaochun Cao. 2016. Makeup like a superstar: Deep localized makeup transfer network. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16 (2016), 2568--2575.

[62]

Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).

[63]

Umar Mohammed, Simon JD Prince, and Jan Kautz. 2009. Visio-lization: Generating novel facial images. ACM Transactions on Graphics (ToG) 28, 3 (2009), 57.

Digital Library

[64]

Kyle Olszewski, Zimo Li, Chao Yang, Yi Zhou, Ronald Yu, Zeng Huang, Sitao Xiang, Shunsuke Saito, Pushmeet Kohli, and Hao Li. 2017. Realistic dynamic facial textures from a single image using GANs. In IEEE International Conference on Computer Vision (ICCV). 5429--5438.

[65]

Darko Pavić, Volker Schönefeld, and Leif Kobbelt. 2006. Interactive image completion with perspective correction. The Visual Computer 22, 9--11 (2006), 671--681.

Digital Library

[66]

Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. 2016. Invertible conditional gans for image editing. NIPS Workshop on Adversarial Training (2016).

[67]

Robert J Peterman, Shuying Jiang, Rene Johe, and Padma M Mukherjee. 2016. Accuracy of Dolphin visual treatment objective (VTO) prediction software on class III patients treated with maxillary advancement and mandibular setback. Progress in orthodontics 17, 1 (2016), 19.

[68]

G Power, J Breckon, M Sherriff, and F McDonald. 2005. Dolphin Imaging Software: an analysis of the accuracy of cephalometric digitization and orthognathic prediction. International journal of oral and maxillofacial surgery 34, 6 (2005), 619--626.

[69]

William R Proffit, Henry W Fields Jr, and David M Sarver. 2006. Contemporary orthodontics. Elsevier Health Sciences.

[70]

Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Noguer. 2018. GANimation: Anatomically-aware facial animation from a single image. In Proceedings of the European Conference on Computer Vision (ECCV). 818--833.

Digital Library

[71]

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 652--660.

[72]

Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang, Fumin Shen, Chen Qian, and Ran He. 2019. Make a face: Towards arbitrary high fidelity face manipulation. In International Conference on Computer Vision (ICCV).

[73]

Fengchun Qiao, Naiming Yao, Zirui Jiao, Zhihao Li, Hui Chen, and Hongan Wang. 2018. Geometry-contrastive generative adversarial network for facial expression synthesis. arXiv preprint arXiv:1802.01822 (2018).

[74]

Waseem Rawat and Zenghui Wang. 2017. Deep convolutional neural networks for image classification: A comprehensive review. Neural computation 29, 9 (2017), 2352--2449.

[75]

Jimmy SJ Ren, Li Xu, Qiong Yan, and Wenxiu Sun. 2015. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems. 901--909.

[76]

Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. 2019. StructureFlow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE International Conference on Computer Vision. 181--190.

[77]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.

[78]

Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. 2018. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In The European Conference on Computer Vision (ECCV).

Digital Library

[79]

Md Mahfuzur Rahman Siddiquee, Zongwei Zhou, Nima Tajbakhsh, Ruibin Feng, Michael B. Gotway, Yoshua Bengio, and Jianming Liang. 2019. Learning fixed points in generative adversarial networks: From image-to-image translation to disease detection and localization. In The IEEE International Conference on Computer Vision (ICCV).

[80]

Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.

[81]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[82]

Lingxiao Song, Zhihe Lu, Ran He, Zhenan Sun, and Tieniu Tan. 2018. Geometry guided adversarial facial expression synthesis. In ACM Multimedia Conference on Multimedia Conference. ACM, 627--635.

Digital Library

[83]

Jian Sun, Lu Yuan, Jiaya Jia, and Heung-Yeung Shum. 2005. Image completion with structure propagation. ACM Transactions on Graphics (ToG) 24, 3 (2005), 861--868.

Digital Library

[84]

Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG) 36, 4 (2017), 95.

Digital Library

[85]

Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34, 6 (2015), 183:1--183:14.

Digital Library

[86]

Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2019. Appearance capture and modeling of human teeth. ACM Transactions on Graphics (ToG) 37, 6 (2019), 207.

Digital Library

[87]

Shuyang Wang and Yun Fu. 2016. Face behind makeup. In Thirtieth AAAI Conference on Artificial Intelligence. 58--64.

Digital Library

[88]

Nicholas Watters, Loïc Matthey, Christopher P. Burgess, and Alexander Lerchner. 2019. Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs. CoRR abs/1901.07017 (2019). arXiv:1901.07017 http://arxiv.org/abs/1901.07017

[89]

Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-time completion of video. IEEE Transactions on Pattern Analysis & Machine Intelligence 3 (2007), 463--476.

Digital Library

[90]

Oliver Whyte, Josef Sivic, and Andrew Zisserman. 2009. Get out of my picture! Internet-based inpainting. In BMVC, Vol. 2. 5.

[91]

Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus Gross, and Thabo Beeler. 2016. Model-based teeth reconstruction. ACM Transactions on Graphics (ToG) 35, 6 (2016), 220.

Digital Library

[92]

Po-Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, and Shih-Wei Liao. 2019b. RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes. In Proceedings of the IEEE International Conference on Computer Vision. 5914--5922.

[93]

Ruizheng Wu, Xin Tao, Xiaodong Gu, Xiaoyong Shen, and Jiaya Jia. 2019c. Attribute-driven spontaneous motion in unpaired image translation. In The IEEE International Conference on Computer Vision (ICCV).

[94]

Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. 2019a. TransGaGa: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8012--8021.

[95]

Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. 2019. Image inpainting with learnable bidirectional attention maps. In Proceedings of the IEEE International Conference on Computer Vision. 8858--8867.

[96]

Junyuan Xie, Linli Xu, and Enhong Chen. 2012. Image denoising and inpainting with deep neural networks. In Advances in neural information processing systems. 341--349.

[97]

Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 1395--1403.

Digital Library

[98]

Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. 2019. Foreground-aware image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5840--5848.

[99]

Xiaojie Xu, Chang Liu, and Youyi Zheng. 2018. 3D tooth segmentation and labeling using deep convolutional neural networks. IEEE transactions on visualization and computer graphics 25, 7 (2018), 2336--2348.

[100]

Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. Highresolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6721--6729.

[101]

Lingchen Yang, Lumin Yang, Mingbo Zhao, and Youyi Zheng. 2018. Controlling Stroke Size in Fast Style Transfer with Recurrent Convolutional Neural Network. In Computer Graphics Forum, Vol. 37. 97--107.

[102]

Raymond Yeh, Ziwei Liu, Dan B Goldman, and Aseem Agarwala. 2016. Semantic facial expression editing using autoencoded flow. arXiv preprint arXiv:1611.09961 (2016).

[103]

Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision. 2849--2857.

[104]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5505--5514.

[105]

Bo Zhao, Bo Chang, Zequn Jie, and Leonid Sigal. 2018. Modular generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV). 150--165.

Digital Library

[106]

Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. 2019. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems 30, 11 (2019), 3212--3232.

[107]

Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. 2019. Pluralistic image completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1438--1447.

[108]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017a. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.

[109]

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems. 465--476.

[110]

Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2019a. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9308--9316.

[111]

Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. 2019b. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2347--2356.

Cited By

Fan YWei GWang CZhuang SWang WZhou YWooldridge MDy JNatarajan S(2024)Collaborative tooth motion diffusion model in digital orthodonticsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i2.27935(1679-1687)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i2.27935
Chen YGao STu PChen X(2024)Automatic 3D Teeth Reconstruction From Five Intra-Oral Photos Using Parametric Teeth ModelIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.327791430:8(4780-4791)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3277914
Wang CWei GWei GWang WZhou Y(2024)Tooth Alignment Network Based on Landmark Constraints and Hierarchical Graph StructureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.321802830:2(1457-1469)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TVCG.2022.3218028
Show More Cited By

Index Terms

iOrthoPredictor: model-guided deep prediction of teeth alignment
1. Computing methodologies
  1. Artificial intelligence
  2. Computer graphics
    1. Image manipulation

Recommendations

Tooth alignment of the dental cast using 3D thin plate spline
ACST'07: Proceedings of the third conference on IASTED International Conference: Advances in Computer Science and Technology

This article has described the process for tooth alignment using 3D thin plate spline. First step is to simulate the individual tooth and its landmarks and fit it into the dental cast by geometric transformation of individual tooth. Next step is to ...
J-Ortho: an open-source orthodontic treatment simulator
SAC '06: Proceedings of the 2006 ACM symposium on Applied computing

An interactive computer-based training tool for using in Orthodontics is aimed at students and experienced professionals who need to predict orthodontic treatment outcomes. Usually, treatment planning and the choice of a proper appliance model are based ...
Chaos and Graphics: An interactive simulation system for training and treatment planning in orthodontics

We have designed and implemented an interactive simulation system for training and treatment planning in orthodontics. Currently, both treatment planning, and the choice of a proper appliance model, are based exclusively on clinician expertise. Most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 39, Issue 6

December 2020

1605 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3414685

Editor:
Karol Myszkowski
MPI Informatik

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 November 2020

Published in TOG Volume 39, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Grants Council of the Hong Kong Special Administrative Region, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
679
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)9

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fan YWei GWang CZhuang SWang WZhou YWooldridge MDy JNatarajan S(2024)Collaborative tooth motion diffusion model in digital orthodonticsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i2.27935(1679-1687)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i2.27935
Chen YGao STu PChen X(2024)Automatic 3D Teeth Reconstruction From Five Intra-Oral Photos Using Parametric Teeth ModelIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.327791430:8(4780-4791)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3277914
Wang CWei GWei GWang WZhou Y(2024)Tooth Alignment Network Based on Landmark Constraints and Hierarchical Graph StructureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.321802830:2(1457-1469)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TVCG.2022.3218028
Madhura DChandana MRajesh VGokul B(2024)Teeth Alignment Prediction Using U-Net2024 International Conference on Recent Advances in Science and Engineering Technology (ICRASET)10.1109/ICRASET63057.2024.10895643(1-4)Online publication date: 21-Nov-2024
https://doi.org/10.1109/ICRASET63057.2024.10895643
Lei CXia MWang SLiang YYi RWen YLiu Y(2024)Automatic Tooth Arrangement with Joint Features of Point and Mesh Representations via Diffusion Probabilistic ModelsComputer Aided Geometric Design10.1016/j.cagd.2024.102293(102293)Online publication date: Apr-2024
https://doi.org/10.1016/j.cagd.2024.102293
Joshi MChavan YKale AJoshi APatil D(2024)SmileScan - Predictive Dental DetectionIntelligent Computing and Big Data Analytics10.1007/978-3-031-74682-6_22(330-347)Online publication date: 31-Dec-2024
https://doi.org/10.1007/978-3-031-74682-6_22
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Chen BFu HZhou KZheng Y(2023)OrthoAligner: Image-Based Teeth Alignment Prediction via Latent Style ManipulationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.316615929:8(3617-3629)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3166159
Zhang CElgharib MFox GGu MTheobalt CWang W(2022)An Implicit Parametric Morphable Dental ModelACM Transactions on Graphics10.1145/3550454.355546941:6(1-13)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555469
Zhao HWillsey MZhu ANandi CTatlock ZSolomon JSchulz A(2022)Co-Optimization of Design and Fabrication Plans for CarpentryACM Transactions on Graphics10.1145/350849941:3(1-13)Online publication date: 9-Mar-2022
https://dl.acm.org/doi/10.1145/3508499
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents