Skip to main content
SpringerLink
Log in

Mobile PointMe-based spatial haptic interaction with annotated media for learning purposes

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

In this paper, we propose a framework to facilitate handheld device-based PointMe interaction with annotated media content, where the user points his/her handheld device to the media presentation screen in order to access services/information from the annotated media. Each annotation is encoded into customized learning object metadata (LOM) format that provides access point for available information about a specific media on the screen. By incorporating a fusion of infra-red (IR) motion points and accelerometer data of the handheld device, the system determines the learner’s PointMe location on the media screen. By using the pointed location spatial geometrical query is performed over the displayed annotated media in order to deliver learner’s level-specific interactive learning content to the user’s handheld device. The level customization is achieved by incorporating learner’s age, progress, etc., parameters in the media content delivery process. We design intuitive learning scenarios by leveraging the PointMe interaction scheme, where learners are challenged to answer quiz questions in learning-based games. This real-world interaction technique with the annotated media and seamless virtual learning information acquisition process makes the system transparent to young learners and encourage them to engage and concentrate on their learning activities. We develop a prototype and perform experiments in a technology-augmented learning space to show the suitability of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Notes

  1. Annotation, http://www.youtube.com/t/annotations_about.

  2. Java SAX parser; http://sax.sourceforge.net/.

References

  1. Ailisto, H., Pohjanheimo, L., Valkkynen, P., Strmmer, E., Tuomisto, T., Korhonen, I.: Bridging the physical and virtual worlds by local connectivity-based physical selection. Pers. Ubiquitous Comput. 10(6), 333–344

  2. Ballagas, R., Borchers, J., Rohs, M., Sheridan, J.: The smart phone: a ubiquitous input device. IEEE Pervasive Comput. 5(1), 70–77

  3. Bargeron, D., Gupta, A., Grudin, J., Sanocki, E.: Annotations for streaming video on the web. In: CHI ’99: CHI ’99 extended abstracts on Human Factors in Computing Systems, pp. 278–279. ACM, New York, NY, USA (1999). http://doi.acm.org/10.1145/632716.632887

  4. Brewster, S., Chohan, F., Brown, L.: Tactile feedback for mobile interactions. In: Proceedings of ACM CHI, pp. 159–162. ACM Press/Addison-Wesley, San Jose, CA, USA (2007)

  5. Broll, G., Rukzio, E., Paolucci, M., Wagner, M., Schmidt, A., Hubmann, H.: Perci: Pervasive service interaction with the internet of things. IEEE Internet Comput. 13(6), 74–81 (2009)

    Google Scholar 

  6. Brom, C., Šisler, V., Slavík, R.: Implementing digital game-based learning in schools: augmented learning environment of ‘europe 2045’. Multimed. Syst. 16, 23–41 (2010). doi:10.1007/s00530-009-0174-0

    Article  Google Scholar 

  7. Cesar, P., Bulterman, D., Jansen, A.: Usages of the secondary screen in an interactive television environment: Control, enrich, share, and transfer television content. Changing Television Environments, LNCS, Springer, Berlin, pp. 168–177 (2008)

  8. Cesar, P., Bulterman, D.C.A., Jansen, J., Geerts, D., Knoche, H., Seager, W.: Fragment, tag, enrich, and send: Enhancing social sharing of video. ACM Trans. Multimedia Comput. Commun. Appl. 5(3), 1–27 (2009)

    Google Scholar 

  9. Chavira, G., Nava, S.W., Herv, R., Villarreal, V., Bravo, J., Mart, S., Castro, M.: Services through NFC technology in AmI environment (2008)

  10. Chorianopoulos, K., Spinellis, D.: User interface development for interactive television: extending a commercial DTV platform to the virtual channel API. Comput. Graph. 28(2), 157–166 (2004)

    Article  Google Scholar 

  11. Correia, N., Chambel, T.: Active video watching using annotation. In: MULTIMEDIA ’99: Proceedings of the Seventh ACM International Conference on Multimedia (Part 2), pp. 151–154. ACM, New York, NY, USA (1999)

  12. Dale, E.: Audio-Visual Methods in Teaching. Rinehart and Winstor, New York (1969)

    Google Scholar 

  13. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13(3), 319–340 (1989)

    Article  Google Scholar 

  14. Duval, E., Hodgins, W.: A LOM research agenda. In: Proceedings of 12th International Conference on World Wide Web (WWW), pp. 1–9 (2003)

  15. Egenhofer, M.: Spatial information appliances: A next generation of geographic information systems. In: First Brazilian Workshop on GeoInformatics, Campinas, Brazil Spatial Information Appliances: A Next Generation of Geographic Information Systems (1999)

  16. El Saddik, A., Rahman, A.S.M.M., Hossain, M.A.: Suitability of searching and representing multimedia learning resources in a 3d virtual gaming environment. IEEE Trans. Instrum. Meas. (TIM) 57(9), 1830–1839 (2008)

    Article  Google Scholar 

  17. Eppler, M.J.: The image of insight: The use of visual metaphors in the communication of knowledge. In: In Proceedings of I-KNOW ’03. Graz, Austria (2003)

  18. Fahiem, M., Haq, S., Saleemi, F.: A review of 3d reconstruction techniques from 2d orthographic line drawings. In: 3D Reconstruction Techniques from 2D Orthographic Line Drawings. Geometric Modeling and Imaging, 2007. GMAI ’07, pp. 60–66 (2007)

  19. Fallahkhair, S., Pemberton, L., Griffiths, R.: Dual device user interface design for ubiquitous language learning: Mobile phone and interactive television (itv). In: WMTE 2005: Proceedings of the IEEE International Workshop on Wireless and Mobile Technologies in Education, pp. 85–92 (2005)

  20. Faugeras, O.: Three-Dimensional Computer Vision. MIT Press, Cambridge, MA

  21. Frohlich, P., Simon, R., Baillie, L.: Mobile spatial interaction. Pers. Ubiquitous Comput. 13(4), 251–253 (2009)

    Google Scholar 

  22. Gellersen, H., Fischer, C., Guinard, D., Gostner, R., Kortuem, G., Kray, C., Rukzio, E., Streng, S.: Supporting device discovery and spontaneous interaction with spatial references. Pers. Ubiquitous Comput. 13(4) (2008)

  23. Gentes, A., Guyot-Mbodji, A., Demeure, I.: Gaming on the move: urban experience as a new paradigm for mobile pervasive game design. Multimed. Syst. 16, 43–55 (2010). doi:10.1007/s00530-009-0172-2

    Article  Google Scholar 

  24. Gujar, U.G., Nagendra, I.V.: Construction of 3D solid objects from orthographic views. Comput. Graph. 13(4), 505–521 (1989)

    Article  Google Scholar 

  25. Guo, J., Mei, T., Liu, F., Hua, X.S.: Adon: an intelligent overlay video advertising system. In: SIGIR ’09: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 628–629. ACM, New York, NY, USA (2009)

  26. Hoggan, E., Brewster, S.A., Johnston, J.: Investigating the effectiveness of tactile feedback for mobile touchscreens. In: CHI ’08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 1573–1582. ACM, New York, NY, USA (2008)

  27. IEEE-LTSC-LOM: IEEE Standard for Learning Technology-Extensible Markup Language (xml) Schema Definition Language Binding for Learning Object Metadata. IEEE Std 1484.12.3-2005, pp. 1–46 (2005)

  28. Jinqiao, W., Yikai, F., Hanqing, L.: Online video advertising based on users attention relavancy computing. In: IEEE International Conference on Multimedia and Expo, pp. 1161–1164 (2008)

  29. Liu, H., Jiang, S., Huang, Q., Xu, C.: A generic virtual content insertion system based on visual attention analysis. In: Proceeding of the 16th ACM International Conference on Multimedia, pp. 379–388. ACM, New York, NY, USA (2008)

  30. Mackay, W.E., Davenport, G.: Virtual video editing in interactive multimedia applications. Commun. ACM 32(7), 802–810 (1989)

    Article  Google Scholar 

  31. Madhavapeddy, A., Scott, D., Sharp, R., Upton, E.: Using camera phones to enhance human–computer interaction (2004)

  32. Marcos, G., Illarramendi, A., Olaizola, I., Flórez, J.: A middleware to enhance current multimedia retrieval systems with content-based functionalities. Multimed. Syst. 17, 149–164 (2011). doi:10.1007/s00530-010-0217-6

    Article  Google Scholar 

  33. Marr, D., Poggio, T.: A computational theory of human stereo vision. Proc. R. Soc. Lond. Ser. B 204, 301–328 (1979)

    Google Scholar 

  34. Najjar, L.J.: Principles of educational multimedia user interface design. Hum. Factors 40(2), 311–323 (1998)

    Article  Google Scholar 

  35. Nonaka, I.: The knowledge-creating company. Harv. Bus. Rev. 69(6), 96–104 (1991)

    Google Scholar 

  36. Park, T., Hwang, J., Hwang, W.: Facilitating the design of vibration for handheld devices. In: Jacko, J. (eds) Human–Computer Interaction. Novel Interaction Methods and Techniques, Lecture Notes in Computer Science, vol. 5611, pp. 496–502. Springer, Berlin (2009)

  37. Pea, R.D.: Video-as-data and digital video manipulation techniques for transforming learning sciences research, education, and other cultural practices. In: Weiss, J., Nolan, J., Hunsinger, J., Trifonas, P. (eds.) The International Handbook of Virtual Learning Environments, pp. 1321–1393. Springer, Berlin (2006)

  38. Rahman, A.S.M.M., Cha, J., Saddik, A.E.: Authoring edutainment content through video annotations and 3d model augmentation. In: IEEE International Conference on Virtual Environments, Human–Computer Interfaces and Measurement Systems (VECIMS), pp. 370–374. Hong Kong, China (2009)

  39. Rahman, A.S.M.M., El Saddik, A.: A 3d notice board metaphor for visualizing, categorizing, and sharing the learning objects. In: The Second International Conference on Systems (ICONS 2007). Martinique, French Caribbean (2007)

  40. Rahman, A.S.M.M., Hossain, M.A., Parra, J., Saddik, A.E.: Motion-path based gesture interaction with smart home services. In: Proceedings of the 17th ACM International Conference on Multimedia. ACM, Beijing, China (2009)

  41. Rahman, A.S.M.M., Hossain, M.A., Saddik, A.E.: Spatial-geometric approach to physical mobile interaction based on accelerometer and ir sensory data fusion. ACM Trans. Multimedia Comput. Commun. Appl. 6, 28:1–28:23 (2010)

    Google Scholar 

  42. Rekimoto, J.: A multiple device approach for supporting whiteboard-based interactions. In: CHI ’98: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 344–351. ACM Press/Addison-Wesley, New York, NY, USA (1998)

  43. Riekki, J., Salminen, T., Alakarppa, I.: Requesting pervasive services by touching rfid tags. Pervasive Comput., IEEE 5(2), 40–46 (2006)

    Google Scholar 

  44. Robertson, S., Wharton, C., Ashworth, C., Franzke, M.: Dual device user interface design: PDAs and interactive television. In: CHI ’96: Proceedings of the SIGCHI Conference on Human factors in Computing Systems, pp. 79–86. ACM, New York, NY, USA (1996)

  45. Rohs, M.: Visual code widgets for marker-based interaction. In: 25th IEEE International Conference on Distributed Computing Systems Workshops, pp. 506–513 (2005)

  46. Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P., Schmidt, A., Chin, J.: An Experimental Comparison of Physical Mobile Interaction Techniques: Touching, Pointing and Scanning, pp. 87–104. Springer, Berlin (2006)

    Google Scholar 

  47. Ruzkio, E., Broll, G., Leichtenstern, K., Schmidt, A.: Mobile interaction with the real world: An evaluation and comparison of physical mobile interaction techniques (2007)

  48. Sanchez, I., Riekki, J., Pyykknen, M.: Touch and control: Interacting with services by touching RFID tags. In: Proceedings of IWRT, vol. 8, pp. 12–13 (2008)

  49. Sharples, M.: The design of personal mobile technologies for lifelong learning. Comput. Educ. 34 177–193 (2000)

    Google Scholar 

  50. Simon, R., Frohlich, P., Grechenig, T.: Geopointing: evaluating the performance of orientation-aware location-based interaction under real-world conditions. J. Locat. Based Serv. 2(1), 24–40

  51. Steggink, J., Snoek, C.G.M.: Adding semantics to image-region annotations with the name-it-game. Multimed. Syst. 17(5), 367–378 (2011)

    Google Scholar 

  52. Stone, R.: Learning and the importance of interactivity information design becomes interaction design. In: Fifth International Conference on Information Visualisation, pp. 624 –629 (2001)

  53. Thompson, S.C.: Will it hurt less if i can control it? a complex answer to a simple question. Psychol. Bull. 90(1), 89–101 (1981)

    Article  Google Scholar 

  54. Tseng, A.H., Hsia, J.W.: The impact of internal locus of control on perceived usefulness and perceived ease of use in e-learning: An extension of the technology acceptance model. In: International Conference on Cyberworlds, pp. 815–819 (2008)

  55. Valkkynen, P., Tuomisto, T.: Physical browsing research. In: Proceedings of Pervasive Mobile Interaction Devices (PERMID 2005). Munich, Germany (2005)

  56. Van Schaik, P., Ling, J.: Design parameters of rating scales for web sites. ACM Trans. Comput.-Hum. Interact. 14(1), 4 (2007)

    Article  Google Scholar 

  57. Vetter, J., Hamard, J., Paolucci, M., Rukzio, E., Schmidt, A.: Physical mobile interaction with dynamic physical object. In: MobileHCI ’07: Proceedings of the 9th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 339–340. ACM, New York, NY, USA (2007)

  58. Want, R., Fishkin, K.P., Gujar, A., Harrison, B.L.: Bridging physical and virtual worlds with electronic tags. In: Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit. ACM pp. 370–377 (1999)

Download references

Author information

Authors and Affiliations

  1. Discover Research Lab, University of Ottawa, Ottawa, Canada

    Abu Saleh Md. Mahfujur Rahman & Abdulmotaleb El Saddik

Authors
  1. Abu Saleh Md. Mahfujur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdulmotaleb El Saddik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abu Saleh Md. Mahfujur Rahman.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rahman, A.S.M.M., El Saddik, A. Mobile PointMe-based spatial haptic interaction with annotated media for learning purposes. Multimedia Systems 19, 131–149 (2013). https://doi.org/10.1007/s00530-012-0285-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-012-0285-x

Keywords

Search

Navigation