Skip to main content
Log in

Diminished reality for augmented reality interior design

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

A modular real-time diminished reality pipeline for indoor applications is presented. The pipeline includes a novel inpainting method which requires no prior information of the textures behind the object to be diminished. The inpainting method operates on rectified images and adapts to scene illumination. In typically challenging illumination situations, the method produces more realistic results in indoor scenes than previous approaches. Modularity enables using alternative implementations in different stages and adapting the pipeline for different applications. Finally, practical solutions to problems occurring in diminished reality applications, for example interior design, are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31

Similar content being viewed by others

References

  1. Alvar: A library for virtual and augmented reality. http://www.vtt.fi/multimedia/alvar.html

  2. Bao, S.Y., Furlan, A., Fei-Fei, L., Savarese, S.: Understanding the 3d layout of a cluttered room from multiple images. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), Steamboat Springs, CO, 24–26 March 2014, pp. 690–697 (2014)

  3. Bhatewara, N., Kumar, P., Agrawal, A.: Intelligent video inpainting system for texture reconstruction. In: 2013 4th International Conference on Computer and Communication Technology (ICCCT), Allahabad, 20–22 Sept 2013, pp. 60–65 (2013)

  4. Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: 2013 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 18–20 June 2003, vol. 2, pp. II-721–II-728 (2003)

  5. Dou, M., Fuchs, H.: Temporally enhanced 3d capture of room-sized dynamic scenes with commodity depth cameras. In: Virtual Reality (VR), IEEE, Minneapolis, MN, USA, 29 March-2 April 2014, pp. 39–44 (2014)

  6. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision, vol. 2. ICCV ’99, p. 1033. IEEE Computer Society, Washington, DC (1999)

  7. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  8. Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: Rgb-d mapping: using depth cameras for dense 3d modeling of indoor environments. In: Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics, Springer Tracts in Advanced Robotics, vol. 79, pp. 477–491. Springer, Berlin (2014)

    Google Scholar 

  9. Herling, J., Broll, W.: Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In: 2014 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 207–212 (2010)

  10. Herling, J., Broll, W.: Pixmix: a real-time approach to high-quality diminished reality. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Atlanta, GA, USA, 5–8 Nov 2012, pp. 141–150 (2012)

  11. Herling, J., Broll, W.: High-quality real-time video inpainting with pixmix. IEEE Trans. Vis. Comput. Graph. 20(6), 866–879 (2014). ID: 1

  12. Jarusirisawad, S., Hosokawa, T., Yokoya, N.: Diminished reality using plane-sweep algorithm with weakly-calibrated cameras. Prog. Inform. 7, 11–20 (2010)

    Article  Google Scholar 

  13. Kawai, N., Sato, T., Yokoya, N.: From image inpainting to diminished reality. In: Shumaker, R., Lackey, S. (eds.) Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments. Lecture Notes in Computer Science, vol. 8525, pp. 363–374. Springer International Publishing, Berlin (2014)

  14. Kawai, N., Yamasaki, M., Sato, T., Yokoya, N.: Ar marker hiding based on image inpainting and reflection of illumination changes. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 293–294 (2012)

  15. Kholgade, N., Simon, T., Efros, A., Sheikh, Y.: 3d object manipulation in a single photograph using stock 3d models. ACM Trans. Graph. 33(4), 127:1–127:12 (2014)

  16. Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker hiding for augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 247–248 (2010)

  17. Li, Z., Wang, Y., Guo, J., Cheong, L.F., Zhou, S.: Diminished reality using appearance and 3d geometry of internet photo collections. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Adelaide, SA, 1–4 Oct 2013, pp. 11–19 (2013)

  18. Liu, H., Philipose, M., Sun, M.T.: Automatic objects segmentation with rgb-d cameras. J. Vis. Commun. Image Represent. 25(4), 709–718 (2014)

  19. Lu, B.V., Kakuta, T., Kawakami, R., Oishi, T., Ikeuchi, K.: Foreground and shadow occlusion handling for outdoor augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Seoul, South Korea, 13–16 Oct 2010 pp. 109–118 (2010)

  20. Mann, S.: Mediated reality with implementations for everyday life.PRESENCE: Teleoperators and Virtual Environments, MIT Press (2002)

  21. Mura, C., Mattausch, O., Villanueva, A.J., Gobbetti, E., Pajarola, R.: Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Comput. Graph. 44, 20–32 (2014)

    Article  Google Scholar 

  22. Pintore, G., Gobbetti, E.: Effective mobile mapping of multi-room indoor structures. Vis. Comput. 30(6–8), 707–716 (2014)

    Article  Google Scholar 

  23. Shepard, D.: A two-dimensional interpolation function for irregularly-spaced data. In: Proceedings of the 1968 23rd ACM National Conference. ACM ’68, pp. 517–524. ACM, New York (1968)

  24. Siltanen, S.: Texture generation over the marker area. In: IEEE/ACM International Symposium on Mixed and Augmented Reality, 2006. ISMAR 2006, Santa Barbara, CA, 22–25 Oct 2006, pp. 253–254 (2006)

  25. Siltanen, S., Oksman, V., Ainasoja, M.: User-centered design of augmented reality interior design service. Int. J. Arts Sci. 6(1), 547–563 (2013)

    Google Scholar 

  26. Siltanen, S., Saraspaa, H., Karvonen, J.: [Demo] a complete interior design solution with diminished reality. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 Sept 2014, pp. 371–372 (2014)

  27. Simon, G., Berger, M.O.: Interactive building and augmentation of piecewise planar environments using the intersection lines. Vis. Comput. 27(9), 827–841 (2011)

    Article  Google Scholar 

  28. Yang, M.D., Chao, C.F., Huang, K.S., Lu, L.Y., Chen, Y.P.: Image-based 3d scene reconstruction and exploration in augmented reality. Autom. Constr. 33, 48–60 (2013). Augmented Reality in Architecture, Engineering, and Construction

  29. Zou, C., Peng, X., Lv, H., Chen, S., Fu, H., Liu, J.: Sketch-based 3-d modeling for piecewise planar objects in single images. Comput. Graph. 46, 130–137 (2015)

    Article  Google Scholar 

Download references

Acknowledgments

Dr. Timo Tossavainen provided valuable comments during the writing of this article.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sanni Siltanen.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mpg 42052 KB)

Supplementary material 2 (mpg 902 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siltanen, S. Diminished reality for augmented reality interior design. Vis Comput 33, 193–208 (2017). https://doi.org/10.1007/s00371-015-1174-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-015-1174-z

Keywords

Navigation