Skip to main content
SpringerLink
Log in

Augmenting natural interaction with physical paper in ambient intelligence environments

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Physical paper, in its various forms (e.g. books, leaflets, catalogues), is extensively used in everyday activities, despite any advancements in digital technology and ICT. Inspired by the popularity of this medium, several research efforts have envisioned and pursued a new era of interactive paper, however several challenges still remain to be addressed. On the other hand, recent advancements towards Ambient Intelligence (AmI) and Smart environments, bear the promise of seamless integration of the physical and digital worlds in an intuitive and user-friendly manner. This paper presents InPrinted, a systematic and generic framework supporting physical paper augmentation and user interaction in Ambient intelligence environments. InPrinted enables natural multimodal user interaction with any kind of printed matter in smart environments, providing context aware and anticipation mechanisms, as well as tools and interaction techniques that support the development of applications incorporating printed matter augmentation. InPrinted has been put to practice in the development of various systems, including the Interactive Maps system, facilitating interaction with printed maps as well as their augmentation with digital information. The results of an in-situ observation experiment of the Interactive Maps system are reported, highlighting that interacting with augmented paper is quite easy and natural, while the overall User Experience is positive.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. http://www.jessrules.com/jess/index.shtml.

  2. http://www.drools.org

  3. http://www.dotnetrdf.org

  4. https://virtuoso.openlinksw.com

  5. http://www.crawler-lib.net/nhunspell

  6. https://www.marketingcharts.com/industries/technology-74252

  7. http://www.pewinternet.org/fact-sheet/mobile/

References

  1. Anoto (2018) Forms based on unique pattern. https://support.anoto.com/article/283-forms-based-on-unique-pattern. Accessed 8 May 2018

  2. ARToolKit Introduction to ARToolKit. http://www.hitl.washington.edu/artoolkit/documentation/userintro.htm. Accessed 8 May 2018

  3. Benko H, Jota R, Wilson A (2012) MirageTable: freehand interaction on a projected augmented reality tabletop. Proceedings of the SIGCHI conference on human factors in computing systems (CHI '12), ACM, New York, p 199–208

  4. Billinghurst M, Kato H, Poupyrev I (2001) The MagicBook: moving seamlessly between reality and Virtuality. IEEE Comput Graph 21(3):6–8

    Google Scholar 

  5. Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Foundations and Trends® in Human–Computer Interaction 8(2–3):73–272

    Article  Google Scholar 

  6. Chen H, Finin T, Joshi A (2003) An ontology for context-aware pervasive computing environments. Knowl Eng Rev 18(3):197–207

    Article  Google Scholar 

  7. Crawford Technologies (2015) Paper vs. digital: two sides survey. http://www.crawfordtech.com/news-and-events/the-importance-of-paper. Accessed 8 May 2018

  8. Dachselt R, Al-Saiegh S (2011) Interacting with printed books using digital pens and smart mobile projection. Proceedings of the Workshop on Mobile and Personal Projection (MP2) @ CHI 2011, Vol. 2011

  9. Fields B, Amaldi P, Wong W, Gill S (2007) In use, in situ: extending field research methods. Int J Hum-Comput Interact 22(1–2):1–6

    Google Scholar 

  10. Figueiredo AC, Pinto AL, Branco P, Zagalo N, Coquet E (2013) Bridging book: a not-so-electronic children's picturebook. Proceedings of the 12th International Conference on Interaction Design and Children (IDC '13), ACM, New York, p 569–572

  11. Forsberg AS, LaViola JJ Jr, Zeleznik RC (1998) ErgoDesk: A Framework for Two- and Three-Dimensional Interaction at the ActiveDesk. Proceedings of the Second International Immersive Projection Technology Workshop, p 11–12

  12. Georgalis I (2013) Architectures, methods and tools for creating ambient intelligence environments. Dissertation, Computer Science Department, University of Crete. https://www.didaktorika.gr/eadd/handle/10442/29423. Accessed 8 May 2018

  13. Golemati M, Katifori A, Vassilakis C, Lepouras G, Halatsis C (2007). Creating an ontology for the user profile: Method and applications. Proceedings of the first RCIS conference, No 2007, p 407–412

  14. Gu T, Wang XH, Pung HK, Zhang DQ (2004) An ontology-based context model in intelligent environments. Proceedings of communication networks and distributed systems modeling and simulation conference, Vol 2004, p 270–275

  15. Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '05), ACM, New York, p 591–599

  16. Kjeldsen R, Pinhanez C, Pingali G, Hartman J, Levas T, Podlaseck M (2002) Interacting with steerable projected displays. Proceedings of the fifth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE, p 402–407

  17. Kobayashi M, Koike H (1998) Enhanced desk: integrating paper documents and digital documents. Asia Pacific Computer Human Interaction (APCHI’98) IEEE CS, p 57–62

  18. Korozi M, Leonidis A, Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2012) Ambient educational mini-games. Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12), ACM, New York, p 802–803

  19. Lam A H, Chow K C, Yau E H, Lyu M R (2006) ART: augmented reality table for interactive trading card game. Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications (VRCIA '06), ACM, New York, p 357–360

  20. Liao C, Guimbretière F, Hinckley K, Hollan J (2008) Papiercraft: a gesture-based command system for interactive paper. ACM T Comput-Hum Int 14(4):18 27 pages

    Google Scholar 

  21. Liao C, Liu Q, Liew B, Wilcox L (2010) Pacer: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), ACM, New York, p 2441–2450

  22. Luff P, Heath C, Norrie M, Signer B, Herdman P (2004) Only touching the surface: creating affinities between digital content and paper. Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04), ACM, New York, p 523–532

  23. Mackay WE, Fayard AL (1999) Designing interactive paper: lessons from three augmented reality projects. Proceedings of International Workshop on Augmented Reality: placing artificial objects in real scenes (IWAR) 98:81–90

    Google Scholar 

  24. Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2011) A smart environment for augmented learning through physical books. Proceedings of the 2011 Conference on Multimedia and Expo (ICME 2011), IEEE, p 1–6

  25. Margetis G, Ntoa S, Bouhli M, Stephanidis C (2011) Study-buddy: improving the learning process through technology-augmented studying environments. Posters’ Extended Abstracts of the HCI International Conference 2011, Springer, Berlin, Heidelberg, p 504–508

  26. Margetis G, Ntelidakis A, Zabulis X, Ntoa S, Koutlemanis P, Stephanidis C (2013). Augmenting physical books towards education enhancement. Proceedings of the 1st IEEE Workshop on User-Centered Computer Vision (UCCV '13), IEEE, p 43–49

  27. Margetis G, Zabulis X, Koutlemanis P, Antona M, Stephanidis C (2013) Augmented interaction with physical books in an ambient intelligence learning environment. Multimed Tools Appl 67(2):473–495

    Article  Google Scholar 

  28. Margetis G, Antona M, Stephanidis C (2015) A framework for supporting natural interaction with printed matter in ambient intelligence environments. Proceeding of the Fifth International Conference on Ambient Computing, Applications, Services and Technologies (AMBIENT'15), IARIA, p 72–78

  29. Margetis G, Zabulis X, Ntoa S, Koutlemanis P, Papadaki E, Antona M, Stephanidis C (2015) Enhancing education through natural interaction with physical paper. Univ Access Inf Soc 14(3):427–447

    Article  Google Scholar 

  30. Margetis G, Ntoa S, Antona M, Stephanidis C (2017) Interacting with augmented paper maps: a user experience study. Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (CHItaly '17), ACM, New York, article no. 18

  31. Marshall CC (2005) Reading and interactivity in the digital library: creating an experience that transcends paper. Proceedings of CLIR/Kanazawa institute of technology roundtable 5(4):1–20

    MathSciNet  Google Scholar 

  32. Matsuo Y, Ishizuka M (2004) Keyword extraction from a single document using word co-occurrence statistical information. Int J Artif Intell T 13(01):157–169

    Article  Google Scholar 

  33. Matthews T, Rattenbury T, Carter S (2007) Defining, designing, and evaluating peripheral displays: an analysis using activity theory. Human–computer. Interaction 22(1–2):221–261

    Google Scholar 

  34. Microsoft (2006) Introduction to the windows workflow foundation rules engine. https://msdn.microsoft.com/en-us/library/aa480193.aspx. Accessed 8 May 2018

  35. Microsoft (2018) Ink analysis overview. http://msdn.microsoft.com/en-us/library/ms704040(v=VS.85).aspx. Accessed 8 May 2018

  36. Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77(12):1321–1329

    Google Scholar 

  37. Miller G, Fellbaum C (1998) Wordnet: An electronic lexical database. Language 76(3):706–708

    MATH  Google Scholar 

  38. Nielsen J (1994) Heuristic evaluation. In: Nielsen J, Mack RL (eds) Usability inspection methods. John Wiley & Sons, New York, pp 25–61

    Google Scholar 

  39. Nielsen C M, Overgaard M, Pedersen M B, Stage J, Stenild S (2006) It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles (NordiCHI '06), ACM, New York, p 272–280

  40. Ojala T, Kostakos V, Kukka H (2011) It’s a jungle out there: fantasy and reality of evaluating public displays in the wild. Large Displays in Urban Life 4:1–4

    Google Scholar 

  41. Papadaki E, Zabulis X, Ntoa S, Margetis G, Koutlemanis P, Karamaounas P, Stephanidis C (2013) The book of Ellie: an interactive book for teaching the alphabet to children. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW 2013), IEEE, p 1–6

  42. Pinhanez C (2001) Using a steerable projector and a camera to transform surfaces into interactive displays. CHI'01 Extended Abstracts on Human Factors in Computing Systems, ACM, p 369–370

  43. Preuveneers D, Van den Bergh J, Wagelaar D, Georges A, Rigole P, Clerckx T, De Bosschere K (2004) Towards an extensible context ontology for ambient intelligence. Proceedings of the European Symposium on Ambient Intelligence, Springer, Berlin, Heidelberg, p 148–159

  44. Robertson C, Robinson J (1999) Live paper: video augmentation to simulate interactive paper. Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99), ACM, New York, p 167–170

  45. Rogers Y, Connelly K, Tedesco L, Hazlewood W, Kurtz A, Hall R E, Hursey J, Toscos T (2007) Why it’s worth the hassle: The value of in-situ studies when designing ubicomp. Proceedings of the International Conference on Ubiquitous Computing (UbiComp 2007), Springer, Berlin, Heidelberg, p 336–353

  46. Rosson MB, Carroll JM (2009) Scenario based design. In: Sears A, Jacko J (eds) Human-computer interaction: development process, Boca Raton, pp 145–162

  47. Schmalstieg D, Fuhrmann V, Hesina G, Szalav Z, Encarna LM, Gervautz M, Purgathofer W (2002) The studierstube augmented reality project. Presence: Teleoper. Virtual Environ 11:33–54

    Google Scholar 

  48. Sellen AJ, Harper RH (2003) The myth of the paperless office. MIT press, Cambridge

    Google Scholar 

  49. Signer B, Norrie M C (2007) PaperPoint: a paper-based presentation and interactive paper prototyping tool. Proceedings of the 1st international conference on Tangible and embedded interaction (TEI '07), ACM, New York, p 57–64

  50. Signer B, Norrie MC (2010) Interactive paper: past, present and future. Proceedings of the 1st International Workshop on Paper Computing (PaperComp '2010), Copenhagen Denmark, September 2010

  51. Smith R (2007) An overview of the Tesseract OCR engine. Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), IEEE, Vol 2, p 629–633

  52. Song H, Grossman T, Fitzmaurice G, Guimbretiere F, Khan A, Attar R, Kurtenbach G. (2009) PenLight: combining a mobile projector and a digital pen for dynamic visual overlay. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), ACM, New York, p 143–152

  53. The Bibliographic Ontology (2009) Bibliographic ontology specification. http://bibliontology.com/. Accessed 8 May 2018

  54. Valderrama Bahamóndez EDC, Kubitza T, Henze N, Schmidt A (2013) Analysis of children's handwriting on touchscreen phones. Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13), ACM, New York, p 171–174

  55. Waldner M, Hauber J, Zauner J, Haller M, Billinghurst M (2006) Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup. Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (OZCHI '06), ACM, New York, p 151–158

  56. Wellner P (1993) Interacting with paper on the DigitalDesk. Commun ACM 36(7):87–96

    Article  Google Scholar 

  57. Woods E, Billinghurst M, Looser J, Aldridge G, Brown D, Garrie B, Nelles C (2004) Augmenting the science centre and museum experience. Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia (GRAPHITE '04), ACM, New York, p 230–236

  58. Wu C S A, Robinson S J, Mazalek A (2008) Turning a page on the digital annotation of physical books. Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI '08), ACM, New York, p 109–116

Download references

Acknowledgments

This work is supported by the FORTH-ICS internal RTD Programme ‘Ambient Intelligence Environments’.

Author information

Authors and Affiliations

  1. Foundation for Research and Technology – Hellas (FORTH), Institute of Computer Science, Heraklion, GR-70013, Crete, Greece

    George Margetis, Stavroula Ntoa, Margherita Antona & Constantine Stephanidis

  2. Department of Computer Science, University of Crete, Crete, Greece

    Constantine Stephanidis

Authors
  1. George Margetis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stavroula Ntoa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Margherita Antona
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Constantine Stephanidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George Margetis.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOC 121 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Margetis, G., Ntoa, S., Antona, M. et al. Augmenting natural interaction with physical paper in ambient intelligence environments. Multimed Tools Appl 78, 13387–13433 (2019). https://doi.org/10.1007/s11042-018-7088-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-7088-9

Keywords

Search

Navigation