Skip to main content
Springer Nature Link
Log in

Unified Model and Framework for Interactive Mixed Entity Systems

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1474))

  • 947 Accesses

Abstract

Mixed reality, natural user interfaces, and the internet of things converge towards an advanced sort of interactive system. These systems enable new forms of interactivity, allowing intuitive user interactions with ubiquitous services in mixed environments. However, they require to synchronize multiple platforms and various technologies. Their heterogeneity makes them complex, and sparsely interoperable or extensible. Therefore, designers and developers require new models, tools, and methodologies to support their creation. We present a unified model of the entities composing these systems, breaking them down into graphs of mixed entities. This model decorrelates real and virtual but still describes their interplay. It characterizes and classifies both the external and internal interactions of mixed entities. We also present a design and implementation framework based on our unified model. Our framework takes advantage of our model to simplify, accelerate, and unify the production of these systems. We showcase the use of our framework by designers and developers in the case of a smart building management system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://z-wavealliance.org/.

  2. 2.

    https://developers.google.com/ar/.

  3. 3.

    https://developer.apple.com/augmented-reality/arkit/.

  4. 4.

    https://github.com/Microsoft/MixedRealityToolkit-Unity.

  5. 5.

    https://unity.com/fr.

  6. 6.

    https://github.com/genielabs/zwave-lib-dotnet.

  7. 7.

    https://lora-alliance.org/.

  8. 8.

    https://developer.vuforia.com/.

  9. 9.

    https://github.com/artoolkit.

References

  1. Alce, G., Roszko, M., Edlund, H., Olsson, S., Svedberg, J., Wallergård, M.: [POSTER] AR as a user interface for the internet of things—comparing three interaction models. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 81–86, October 2017. https://doi.org/10.1109/ISMAR-Adjunct.2017.37

  2. Atzori, L., Iera, A., Morabito, G.: The internet of things: a survey. Comput. Netw. 54(15), 2787–2805 (2010). https://doi.org/10.1016/j.comnet.2010.05.010. http://dx.doi.org/10.1016/j.comnet.2010.05.010

  3. Bataille, G., Gouranton, V., Lacoche, J., Pelé, D., Arnaldi, B.: A unified design & development framework for mixed interactive systems. In: VISIGRAPP, GRAPP, the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Valetta, Malta, pp. 49–60 (2020)

    Google Scholar 

  4. Blackler, A., Hurtienne, J.: Towards a unified view of intuitive interaction: definitions, models and tools across the world. MMI-Interaktiv 13 (2007)

    Google Scholar 

  5. Blake, J.: The natural user interface revolution. In: Natural User Interfaces in .Net, pp. 1–43. Manning publications edn. (2012)

    Google Scholar 

  6. Bouville, R., Gouranton, V., Boggini, T., Nouviale, F., Arnaldi, B.: #FIVE: high-level components for developing collaborative and interactive virtual environments. In: 2015 IEEE 8th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS), pp. 33–40. IEEE, Arles, March 2015. https://doi.org/10.1109/SEARIS.2015.7854099. http://ieeexplore.ieee.org/document/7854099/

  7. Bouzekri, E., Canny, A., Martinie, C., Palanque, P.: A generic software and hardware architecture for hybrid interactive systems. In: EICS 2018 - Workshop on Heterogeneous Models and Modeling Approaches for Engineering of Interactive Systems, Paris, France, June 2018

    Google Scholar 

  8. Claude, G., Gouranton, V., Arnaldi, B.: Versatile scenario guidance for collaborative virtual environments. In: Proceedings of 10th International Conference on Computer Graphics Theory and Applications (GRAPP 2015), Berlin, Germany, March 2015. https://hal-univ-rennes1.archives-ouvertes.fr/hal-01147733

  9. Coutaz, J.: PAC: an object oriented model for implementing user interfaces. ACM SIGCHI Bull. 19(2), 37–41 (1987). https://doi.org/10.1145/36111.1045592. https://dl.acm.org/doi/10.1145/36111.1045592

  10. Coutrix, C., Nigay, L.: Mixed reality: a model of mixed interaction. In: Proceedings of the Working Conference on Advanced Visual Interfaces - AVI 2006, p. 43. ACM Press, Venezia (2006). https://doi.org/10.1145/1133265.1133274. http://portal.acm.org/citation.cfm?doid=1133265.1133274

  11. Dubois, E., Bortolaso, C., Appert, D., Gauffre, G.: An MDE-based framework to support the development of Mixed Interactive Systems. Sci. Comput. Program. 89, 199–221 (2014). https://doi.org/10.1016/j.scico.2013.03.007. https://linkinghub.elsevier.com/retrieve/pii/S0167642313000671

  12. Dubois, E., Gray, P., Nigay, L.: The Engineering of Mixed Reality Systems. Human-Computer Interaction Series, Springer, Heidelberg (2010). https://doi.org/10.1007/978-1-84882-733-2

    Book  Google Scholar 

  13. Fuchs, P., Moreau, G., Guitton, P.: Virtual Reality: Concepts and Technologies. CRC Press (2011)

    Google Scholar 

  14. Gibson, J.J.: The theory of affordances. In: Shaw, R.E., Bransford, J. (ed.) Perceiving, Acting, and Knowing: Toward an Ecological Psychology, pp. 67–82. Lawrence Erlbaum Associates, Hillsdale (1977). https://hal.archives-ouvertes.fr/hal-00692033

  15. Glaessgen, E., Stargel, D.: The digital twin paradigm for future NASA and US Air Force vehicles. In: 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference 20th AIAA/ASME/AHS Adaptive Structures Conference 14th AIAA, p. 1818 (2012)

    Google Scholar 

  16. Greer, C., Burns, M., Wollman, D., Griffor, E.: Cyber-physical systems and internet of things. Technical report NIST SP 1900–202, National Institute of Standards and Technology, Gaithersburg, MD, March 2019. https://doi.org/10.6028/NIST.SP.1900-202. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1900-202.pdf

  17. Grieves, M.W.: Virtually intelligent product systems: digital and physical twins. In: Complex Systems Engineering: Theory and Practice, pp. 175–200 (2019)

    Google Scholar 

  18. Hussey, A., Carrington, D.: Comparing the MVC and PAC architectures: a formal perspective. IEE Proc. - Softw. Eng. 144(4), 224–236 (1997). https://doi.org/10.1049/ip-sen:19971512

    Article  Google Scholar 

  19. Jacob, R.J., et al.: Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 201–210. ACM, New York (2008). https://doi.org/10.1145/1357054.1357089. http://doi.acm.org/10.1145/1357054.1357089

  20. Kelaidonis, D., et al.: Virtualization and cognitive management of real world objects in the internet of things. In: 2012 IEEE International Conference on Green Computing and Communications, pp. 187–194. IEEE, Besancon, November 2012. https://doi.org/10.1109/GreenCom.2012.37. http://ieeexplore.ieee.org/document/6468312/

  21. Kim, K., Bölling, L., Haesler, S., Bailenson, J., Bruder, G., Welch, G.F.: Does a digital assistant need a body? The influence of visual embodiment and social behavior on the perception of intelligent virtual agents in AR, Munich, pp. 105–114 (2018). https://doi.org/10.1109/ISMAR.2018.00039

  22. Lacoche, J., Le Chenechal, M., Villain, E., Foulonneau, A.: Model and tools for integrating iot into mixed reality environments: towards a virtual-real seamless continuum. In: ICAT-EGVE 2019 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments. Tokyo, Japan, September 2019. https://hal.archives-ouvertes.fr/hal-02332096

  23. Lin, S., Cheng, H.F., Li, W., Huang, Z., Hui, P., Peylo, C.: Ubii: physical world interaction through augmented reality. IEEE Trans. Mob. Comput. 16(3), 872–885 (2017). https://doi.org/10.1109/TMC.2016.2567378

    Article  Google Scholar 

  24. Liu, W.: Natural user interface- next mainstream product user interface. In: 2010 IEEE 11th International Conference on Computer-Aided Industrial Design Conceptual Design 1, vol. 1, pp. 203–205, November 2010. https://doi.org/10.1109/CAIDCD.2010.5681374

  25. MacWilliams, A., Sandor, C., Wagner, M., Bauer, M., Klinker, G., Bruegge, B.: Herding sheep: live system development for distributed augmented reality. In: Proceedings of the 2Nd IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR 2003, p. 123. IEEE Computer Society, Washington, DC (2003). http://dl.acm.org/citation.cfm?id=946248.946803

  26. Marchand, E., Uchiyama, H., Spindler, F.: Pose estimation for augmented reality: a hands-on survey. IEEE Trans. Vis. Comput. Graph. 22(12), 2633–2651 (2016). https://doi.org/10.1109/TVCG.2015.2513408. http://ieeexplore.ieee.org/document/7368948/

  27. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D (1994)

    Google Scholar 

  28. Nitti, M., Pilloni, V., Colistra, G., Atzori, L.: The virtual object as a major element of the internet of things: a survey. IEEE Commun. Surv. Tutor. 18(2), 1228–1240 (2016). https://doi.org/10.1109/COMST.2015.2498304. http://ieeexplore.ieee.org/document/7320954/

  29. Norman, D.A.: The Psychology of Everyday Things. (The Design of Everyday Things). Basic Books (1988)

    Google Scholar 

  30. Norman, D.A.: Affordance, conventions, and design. Interactions 6(3), 38–43 (1999). https://doi.org/10.1145/301153.301168. http://doi.acm.org/10.1145/301153.301168

  31. Norman, D.A.: Natural user interfaces are not natural. Interactions 17(3), 6–10 (2010). https://doi.org/10.1145/1744161.1744163. http://doi.acm.org/10.1145/1744161.1744163

  32. Pfeiffer, T., Pfeiffer-Leßmann, N.: Virtual prototyping of mixed reality interfaces with internet of things (IoT) connectivity. i-com 17(2), 179–186 (2018). https://doi.org/10.1515/icom-2018-0025. http://www.degruyter.com/view/j/icom.2018.17.issue-2/icom-2018-0025/icom-2018-0025.xml

  33. Poslad, S.: Ubiquitous Computing: Smart Devices, Environments and Interactions. Wiley, Hoboken (2011)

    Google Scholar 

  34. Sánchez López, T., Ranasinghe, D.C., Harrison, M., Mcfarlane, D.: Adding sense to the internet of things. Pers.Ubiquit. Comput. 16(3), 291–308 (2012). https://doi.org/10.1007/s00779-011-0399-8. http://dx.doi.org/10.1007/s00779-011-0399-8

  35. Turner, P.: Affordance as context. Interact. Comput. 17(6), 787–800 (2005). https://doi.org/10.1016/j.intcom.2005.04.003

    Article  Google Scholar 

  36. Uimonen, M.: Accessing BIM-related information through AR, p. 2 (2018)

    Google Scholar 

  37. Walther, S.: ASP.NET 3.5 unleashed. Sams, Indianapolis, Ind (2008). oCLC: ocn180989631

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Orange Labs, Cesson Sévigné, France

    Guillaume Bataille, Jérémy Lacoche & Danielle Pelé

  2. Univ. Rennes, INSA Rennes, Inria, CNRS, IRISA, Rennes, France

    Guillaume Bataille, Valérie Gouranton & Bruno Arnaldi

Authors
  1. Guillaume Bataille
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valérie Gouranton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jérémy Lacoche
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Danielle Pelé
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bruno Arnaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guillaume Bataille .

Editor information

Editors and Affiliations

  1. IRISA, University of Rennes 1, Rennes, France

    Kadi Bouatouch

  2. Universidade do Porto, Porto, Portugal

    A. Augusto de Sousa

  3. University of Genova, Genova, Italy

    Manuela Chessa

  4. Mines ParisTech, Paris, France

    Alexis Paljic

  5. Linnaeus University, Växjö, Sweden

    Andreas Kerren

  6. French Civil Aviation University (ENAC), Toulouse, France

    Christophe Hurter

  7. Università di Catania, Catania, Italy

    Giovanni Maria Farinella

  8. Universitat de Barcelona, Barcelona, Spain

    Petia Radeva

  9. Escola Superior de Tecnologia de Setúbal, Setúbal, Portugal

    Jose Braz

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bataille, G., Gouranton, V., Lacoche, J., Pelé, D., Arnaldi, B. (2022). Unified Model and Framework for Interactive Mixed Entity Systems. In: Bouatouch, K., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2020. Communications in Computer and Information Science, vol 1474. Springer, Cham. https://doi.org/10.1007/978-3-030-94893-1_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-94893-1_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-94892-4

  • Online ISBN: 978-3-030-94893-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics