Skip to main content
SpringerLink
Log in

Users’ viewpoint of usability and user experience testing procedure - gaining methodological insights in a case of an interactive HbbTV application

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We present a novel meta-methodological approach for the user experience and usability methodology procedure evaluation, shown in an example of the user experience and usability study of an interactive HbbTV application. The idea behind this research is not only to evaluate and improve the TV-WEB service but also to gain insights how the participants perceived the whole Ux evaluation procedure itself. The research questions focused mainly on the time complexity (temporal demand) and the frustration level of the TV-WEB evaluation procedure. Additionally the appropriateness of the selected content used, interface and interaction design, and the service impressions and satisfaction/payment related questions was sought. A special questionnaire based on the NASA TLX standard test is presented. The concept has been successfully implemented in several live field trials in three countries (BiH, Serbia and Montenegro). In the user experience and usability study of the service itself more than 150 participants were involved, of those 35 took part in the meta-methodology study. The feedback was quantitatively evaluated on a 7-point Likert scale, with “1” indicating the best positive feedback, “4” neutral/undecided feedback and “7” the most negative feedback. The quantitative average summary results obtained in the three evaluation studies were 1.30 for both the BiH and Serbia test study cases and the score of 1.22 for the test study case in Montenegro. These results show that a great majority of the participants found the whole evaluation procedure time-wise and frustration-wise undemanding, with appropriate content, presentation style and overall attitude towards them. By using this approach it was possible to improve the user experience and usability methodology used, producing more reliable results and providing better user experience in the final version of the product as well as providing a pleasant experience during the testing of the product.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Albert, William, Thomas, Tullis (2013) Measuring the user experience: collecting, analyzing, and presenting usability metrics. Elsevier

  2. Bachmayer S, Lugmayr A, Kotsis G (2010) Convergence of collaborative web approaches and interactive TV program formats. Int J Web Inf Syst 6(1):74–94

    Article  Google Scholar 

  3. Brooke J (2013) SUS: a retrospective. J Usability Stud 8(2):29–40

    MathSciNet  Google Scholar 

  4. Chorianopoulos K (2008) User interface design principles for interactive television applications. Int J Hum Comput Interact 24(6):556–573

    Article  Google Scholar 

  5. ETSI (2012) Hybrid Broadcast Broadband TV, ETSI TS 102 796 v1.2.1 (2012–11)

  6. ETSI Digital Video Broadcasting (DVB) (2010) Signalling and carriage of interactive applications and services in Hybrid broadcasting/broadband environments, ETSI TS 102 809 V1.1.1 (2010–01)

  7. Guna J, Stojmenova E, Hauswirth D, Winkler R, Ninaus M, Pogačnik M (2013) WEB on TV: designing the user experience. V: STOJMENOVA, Emilija (ur.), LUGMAYR, Artur (ur.). Proceedings of the 1st Workshop on Defining a European Research Agenda on Information Systems and Management in eMedia Industries, Bled, Slovenia, 9th-13th June 2013, (International series on information systems and management in creative emedia, ISSN 2341–5576, no. 2013/1). Tampere: IugYmedia, str. 5–7.

  8. Guna J, Kos A, Pogačnik M (2014) Gaining methodological insights from the user experience and usability study of an interactive HbbTV application. Proc. of the 7th Intern. Workshop on Semantic Ambient Media Experiences (SAME 2014) – Ambient Media Usability, Interaction, and Smart Media Technologies, In conjunction with NORDCHI 2014, Helsinki, Finland, 26th Oct

  9. Hart, Sandra G (2006) "NASA-task load index (NASA-TLX); 20 years later." Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 50. No. 9. Sage Publications

  10. Kalli S, Lugmayr A, Niiranen S (2004) Digital interactive TV and Metadata: future broadcast multimedia. Springer-Verlag, New York, doi:10.1007/978-1-4757-3953-4

  11. Lowdermilk T (2013) User-centered design: a Developer's guide to building user-friendly applications. O'Reilly Media, Inc., Newton

    Google Scholar 

  12. Lugmayr A, Serral E, Scherp A, Pogorelc B, Mustaquim M (2014) Ambient media today and tomorrow. Multimed Tools App 71(1):7–37

    Article  Google Scholar 

  13. Pauzié A (2008) "Evaluating driver mental workload using the driving activity load index (DALI)." Proc. of European Conference on Human Interface Design for Intelligent Transport Systems

  14. SEE TV-WEB, (2015) Project brochure http://www.see-tvweb.eu/communication-material (lastaccessed on 18.2.2015)

  15. Stojmenova E, Guna J, Hauswirth D, Winkler R, Ninaus M, Pogačnik M (2013) WEB on TV: cross-national user study for designing better user experience. V: LUGMAYR, Artur (ur.). Making sense of covering media. New York: Association for Computing Machinery, cop. 2007, str. 17–19

  16. the TV-WEB project - Tackling the”(2015) Digital Divide” in SEE, http://www.see-tvweb.eu (last accessed on 18.2.2015)

Download references

Acknowledgments

We would like to kindly thank all the participants in the evaluation study for their participation and valuable feedback given.

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, University of Ljubljana, Tržaška 25, 1000, Ljubljana, Slovenia

    Jože Guna, Emilija Stojmenova-Duh & Matevž Pogačnik

Authors
  1. Jože Guna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emilija Stojmenova-Duh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matevž Pogačnik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jože Guna.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guna, J., Stojmenova-Duh, E. & Pogačnik, M. Users’ viewpoint of usability and user experience testing procedure - gaining methodological insights in a case of an interactive HbbTV application. Multimed Tools Appl 76, 16125–16143 (2017). https://doi.org/10.1007/s11042-016-3898-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3898-9

Keywords

Search

Navigation