Skip to main content
SpringerLink
Log in

Providing multimodal and multi-user interactions for digital tv applications

  • 1215: Multimodal Interaction and IoT Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Listing 1
Fig. 2
Listing 2
Fig. 3
Fig. 4
Fig. 5
Listing 3
Fig. 6
Fig. 7
Listing 4
Fig. 8
Listing 5
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Notes

  1. https://assistant.google.com/platforms/tv/

  2. The Ginga-NCL extension implementation presented in this work can be accessed at https://github.com/marinaivanov/ginga-mulsemedia/tree/multimodal

  3. https://github.com/TeleMidia/ginga

  4. https://gaming.tobii.com/tobii-eye-tracker-4c/

  5. https://help.tobii.com/hc/en-us/articles/213414285-Specifications-for-the-Tobii-Eye-Tracker-4C

  6. Video of the application running at: https://1drv.ms/u/s!AvKXFJ6ipz46hqgcEWo39TPNhFC4JQ?e=xyetJ1

  7. NCL applications and the script can be accessed at http://bit.do/fHiw4

  8. https://forumsbtvd.org.br/tv3_0/

References

  1. ABNT (2021) Digital terrestrial television - Data coding and transmission specification for digital broadcasting - Part 2: Ginga-NCL for fixed and mobile receivers - XML application language for application coding (In portuguese)

  2. Barreto F, de Abreu RS, Montevecchi EBB, Josué MI, Valentim PA, Muchaluat-Saade DC (2020) Extending ginga-ncl to specify multimodal interactions with multiple users (in portuguese). In: Proceedings of the brazilian symposium on multimedia and the web, pp. 281–288

  3. Bolt RA (1980) “Put-that-there” voice and gesture at the graphics interface. In: Proceedings of the 7th annual conference on Computer graphics and interactive techniques, pp. 262–270

  4. Card S, MORAN T, Newell A (1986) The model human processor- an engineering model of human performance. Handbook of perception and human performance 2(45–1)

  5. Carvalho LAMC, Guimarães AP, Macêdo HT (2008) Architectures for interactive vocal environment to brazilian digital tv middleware. In: Proceedings of the 2008 euro american conference on telematics and information systems, EATIS ’08. ACM, New York, NY, USA

  6. Casanova MA, Tucherman L, Lima MJD, Rangel Netto JL, Rodriquez N, Soares LF (1991) The nested context model for hyperdocuments. In: Proceedings of the third annual ACM conference on Hypertext, pp. 193–201

  7. Costa RMdR, Moreno MF, Soares LFG (2009) Ginga-ncl: Supporting multiple devices. In: Proceedings of the XV brazilian symposium on multimedia and the web, webmedia ’09. association for computing machinery, New York, NY, USA

  8. Danesh A, Saini M, El Saddik A (2015) A proxemic multimedia interaction over the internet of things International conference on multimedia modeling, pp. 1–12. Springer

  9. de Farias BC, de Lima Filho EB, Maia OB, Souto E (2020) Extensions to middleware ginga for integration with iot environments. In: 2020 IEEE International conference on consumer electronics (ICCE), pp. 1–5. IEEE

  10. de Lima ES, Feijó B, Barbosa S, da Silva FG, Furtado AL, Ciarlini AE, Pozzer CT (2011) Multimodal, multi-user and adaptive interaction for interactive storytelling applications. In: 2011 Brazilian symposium on games and digital entertainment, pp. 206–214. IEEE

  11. Feit AM, Williams S, Toledo A, Paradiso A, Kulkarni H, Kane S, Morris MR (2017) Toward everyday gaze input: Accuracy and precision of eye tracking and implications for design. In: Proceedings of the 2017 chi conference on human factors in computing systems, chi ’17, p. 1118–1130. association for computing machinery, New York, NY, USA. https://doi.org/10.1145/3025453.3025599

  12. Furht B (2008) Encyclopedia of multimedia Springer Science & Business Media

  13. Gowing M, Ahmadi A, Destelle F, Monaghan DS, O’Connor NE, Moran K (2014) Kinect vs. low-cost inertial sensing for gesture recognition. In: International conference on multimedia modeling, pp. 484–495. Springer

  14. Guedes ÁLV, de Albuquerque Azevedo RG, Barbosa SDJ (2017) Extending multimedia languages to support multimodal user interactions. Multimedia Tools and Applications 76(4):5691–5720

    Article  Google Scholar 

  15. Hunkeler U, Truong HL, Stanford-Clark A (2008) Mqtt-s—a publish/subscribe protocol for wireless sensor networks. In: 2008 3Rd international conference on communication systems software and middleware and workshops (COMSWARE’08), pp. 791–798. IEEE

  16. Ierusalimschy R (2006) Programming in lua Roberto Ierusalimschy

  17. ITU (2009) Nested context language (ncl) and ginga-ncl for iptv services. http://www.itu.int/rec/T-REC-H.761-200904-S. ITU-T Rec. H.761

  18. Klyne G, Carroll JJ (2006) Resource description framework (rdf): Concepts and abstract syntax

  19. Luque FP, Galloso I, Feijoo C, Martín CA, Cisneros G (2014) Integration of multisensorial stimuli and multimodal interaction in a hybrid 3dtv system. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 11(1s):1–22

    Article  Google Scholar 

  20. Mo S, Yang W, Wang G, Liao Q (2020) Emotion recognition with facial landmark heatmaps. In: International conference on multimedia modeling, pp. 278–289. Springer

  21. Montevecchi EBB, Josué MI, Barreto F, de Abreu RS, Muchaluat-Saade DC (2020) Providing eye gaze interaction for ginga-ncl applications (in portuguese). In: Proceedings of the brazilian symposium on multimedia and the web, pp. 297–303

  22. Muchaluat-Saade DC, Soares LFG (2002) Xconnector & xtemplate: Improving the expressiveness and reuse in web authoring languages. New Rev Hypermedia Multimed J 8(1):139–169

    Article  Google Scholar 

  23. Pedrosa D, Martins Jr JAC, Melo EL, Teixeira CA (2011) A multimodal interaction component for digital television. In: Proceedings of the 2011 ACM Symposium on Applied Computing, pp. 1253–1258

  24. Pereira DMG, e Silva FJdS, Carlos de Salles SN, dos Santos DV, Coutinho LR, Guedes ÁL (2021) An ontology-based approach to integrate tv and iot middlewares. Multimed Tools Appl 80(2):1813–1837

    Article  Google Scholar 

  25. Silva ECO, dos Santos JA, Muchaluat-Saade DC (2013) Jns: an alternative authoring language for specifying ncl multimedia documents. In: 2013 IEEE International conference on multimedia and expo workshops (ICMEW), pp. 1–6. IEEE

  26. Soares LFG, Rodrigues RF (2005) Nested context model 3.0: Part 1–ncm core. Technical Report of the Informatics Department of PUC-Rio 12

  27. Soares LFG, Rodrigues RF, Moreno MF (2007) Ginga-ncl: the declarative environment of the brazilian digital tv system. J Brazilian Comput Soc 12 (4):37–46

    Article  Google Scholar 

  28. Soares LFG, Rodrigues RF, Muchaluat-Saade DC (2000) Modeling, authoring and formatting hypermedia documents in the hyperprop system. Multimedia systems 8(2):118–134

    Article  MATH  Google Scholar 

  29. Tobii (2020) This is eye tracking. https://www.tobii.com/group/about/this-is-eye-tracking. Acessed in: May 2020

  30. Turabzadeh S, Meng H, Swash RM, Pleva M, Juhar J (2018) Facial expression emotion detection for real-time embedded systems. Technologies 6(1):17

    Article  Google Scholar 

  31. Turk M (2014) Multimodal interaction: a review. Pattern Recogn Lett 36:189–195

    Article  Google Scholar 

  32. Turunen M, Kallinen A, Sànchez I., Riekki J, Hella J, Olsson T, Melto A, Rajaniemi JP, Hakulinen J, Mäkinen E et al (2009) Multimodal interaction with speech and physical touch interface in a media center application. In: Proceedings of the international conference on advances in computer enterntainment technology, pp. 19–26

  33. Union IT (2009) Itu-r h761: Nested context language (ncl) and ginga-ncl for iptv services

  34. W3C (2008) Synchronized multimedia integration language - smil 3.0 specification. http://www.w3c.org/TR/SMIL3. World-Wide Web Consortium Rec.

  35. W3C (2014) Html5: A vocabulary and associated apis for html and xhtml. https://www.w3.org/TR/html5/. World-Wide Web Consortium Rec.

Download references

Acknowledgments

The authors would like to thank CAPES, CAPES PRINT, CNPq, INCT-MACC and FAPERJ for the partial financial support of this work.

Author information

Authors and Affiliations

  1. MídiaCom Lab, Fluminense Federal University, Niterói, Brazil

    Fábio Barreto, Raphael S. de Abreu, Marina I. P. Josué, Eyre Brasil B. Montevecchi, Pedro Alves Valentim & Débora C. Muchaluat-Saade

  2. Unilasalle-RJ, Niterói, Brazil

    Fábio Barreto & Raphael S. de Abreu

Authors
  1. Fábio Barreto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raphael S. de Abreu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marina I. P. Josué
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eyre Brasil B. Montevecchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pedro Alves Valentim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Débora C. Muchaluat-Saade
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fábio Barreto.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barreto, F., de Abreu, R., Josué, M.I.P. et al. Providing multimodal and multi-user interactions for digital tv applications. Multimed Tools Appl 82, 4821–4846 (2023). https://doi.org/10.1007/s11042-021-11847-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11847-3

Keywords

Search

Navigation