Skip to main content
SpringerLink
Log in

A virtual reality approach for usability assessment: case study on a wheelchair-mounted robot manipulator

  • Original Article
  • Published:
Engineering with Computers Aims and scope Submit manuscript

Abstract

The aim of the paper is to address an innovative methodology for assessing the usability of a product. This methodology is particularly suitable for designing products that provide their main functions through their control interfaces. In particular, this case study relates to the usability assessment of two control devices for a wheelchair-mounted robot manipulator to assist physically disabled people. The study focuses on defining a synthetic usability index on the basis of two currently used methods: the multi criteria decision analysis and the Saaty’s analytic hierarchy process. Several virtual reality (VR)-based experiments have been conducted, set up in accordance with a cross-array experimental plan, that adequately caters for both control and noise factors. Quantitative measures and subjective user evaluations have been collected to maximize the effectiveness, the efficiency and the satisfaction perceived by users while using the product. Compared to the literature on the subject, the proposed approach provides both more flexibility in defining quantitative indexes and more adequate results, even when involving only a small sample of users in the participatory design session. The use of VR technologies for the collection of the experimental data has been essential in terms of safety, costs and repeatability of the tests, as well as of the robustness with respect to noise factors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. It is understood that those devices actually control the computer-generated viewpoint only, without affecting the physical position of the wheelchair in the laboratory.

References

  1. Abbattista F, Degemmis M, Licchelli O, Lops P, Semeraro G, Zambetta F (2002) Improving the usability of an e-commerce web site through personalization. In: Ricci F, Smith B (eds) Recommendation and personalization in Ecommerce, 2nd International conference on adaptive hypermedia and adaptive web based systems, pp 20–29

  2. Bevan N (2009) Extending quality in use to provide a framework for usability measurement. In: HCD 09: proceedings of the 1st international conference on human centered design. Springer, Berlin, pp 13–22

  3. Caputo F, Di Gironimo G (2007) Vrtest: a virtual reality sysTEm for tranSporTation design. Centre of Competence for Transport Systems of the Campania Region, An experience of innovation and training

  4. Caputo F, Di Gironimo G, Papa S (2006) A virtual reality system for ergonomics and usability validation of equipment controls. Anales de Ingenierìa Gràfica 18:47–64

    Google Scholar 

  5. Das H, Zak H, Johnson J, Crouch J, Frambach D (1999) Evaluation of a telerobotic system to assist surgeons in microsurgery. Comput Aided Surg 15–25

  6. Davies R (2004) Adapting virtual reality for the participatory design of work environments. Comput. Supported Coop. Work 13(1):1–33. doi:10.1023/B:COSU.0000014985.12045.9c

  7. De Santis A, Di Gironimo G, Marzano A, Siciliano B, Tarallo A (2008) A virtual-reality-based evaluation environment for wheelchair-mounted manipulators. In: Proceedings of the 6th EUROGRAPHICS Italian Chapter 2008 conference. Salerno, Italy, pp 1–8

  8. Di Gironimo G, Lanzotti A (2009) Designing in VR. Int J Interact Design Manuf (IJIDeM) 3(2):51–53

    Article  Google Scholar 

  9. Di Gironimo G, Marzano A, Tarallo A (2007) Human robot interaction in virtual reality. In: 5th EUROGRAPHICS Italian chapter conference. Trento, Italy, pp 1–8

  10. Di Gironimo G, Patalano S, Tarallo A (2009) Innovative assembly process for modular train and feasibility analysis in virtual environment. Int J Interact Design Manuf (IJIDeM) 3(2):93–101

    Article  Google Scholar 

  11. Dinh HQ, Walker N, Song C, Kobayashi A, Hodges LF (1999) Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. Virtual Reality Conference, IEEE 0, 222. doi:10.1109/VR.1999.756955

  12. Eftring H, Boschian K (1999) Technical results from manus user trials. In: Proceedings of the international conference on rehabilitation robotics (ICORR). Stanford, CA, pp 136–141

  13. Eraslan E, Akay D, Kurt M (2006) Usability ranking of intercity bus passenger seats using fuzzy axiomatic design theory. In: CDVE, pp 141–148

  14. Eriksson J, Johansson G (1996) Adaptation of workplaces and homes for disabled people using computer aided design. Int J Ind Ergonomics 17(2):153–162. doi:1016/0169-8141(95)00046-1

    Article  Google Scholar 

  15. Figueira J, Greco S, Ehrgott M (2005) Multiple criteria decision analysis: state of the art surveys. Springer, Berlin

    Google Scholar 

  16. Frøkjær E, Hertzum M, Hornbæk K (2000) Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In: CHI’00: proceedings of the SIGCHI conference on Human factors in computing systems. ACM, New York, pp 345–352. doi:10.1145/332040.332455

  17. Ghosh R, Dekhil M (2009) Discovering user profiles. In: Proceedings of the 18th international conference on World wide web, WWW’09. ACM, New York, pp 1233–1234. doi:10.1145/1526709.1526944

  18. Hassenzahl M (2003) The thing and i: understanding the relationship between user and product. In: Blythe MA, Overbeeke K, Monk AF (eds) Funology: from usability to enjoyment (human–computer interaction series). Kluwer Academic Publishers, Norwell, pp 31–42

    Google Scholar 

  19. Hillman M, Gammie A (1994) The bath institute of medical engineering assistive robot. In: Proceedings of ICORR 94. Wilmington, pp 211–212

  20. Hornbaek K (2006) Current practice in measuring usability: Challenges to usability studies and research. Int J Hum Comput Stud 64(2):79–102. doi:10.1016/j.ijhcs.2005.06.002

    Google Scholar 

  21. ISO/IEC standard (1998) Ergonomic requirements for office work with visual display terminals—guidance on usability. ISO 9241-11:1998

  22. Jaspers M (2009) A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. I. J Med Inf 78(5):340–353

    Article  Google Scholar 

  23. Jolliffe I (2002) Principal component analysis. Springer Verlag, Secaucus

    MATH  Google Scholar 

  24. Kalawsky R (1998) New methodologies and techniques for evaluating user performance in advanced 3d virtual interfaces. IEE Digest. 1998(437), 5/1–5/8. doi:10.1049/ic:19980654

  25. Kanai S, Horiuchi S, Kikuta Y, Yokoyama A, Shiroma Y (2007) An integrated environment for testing and assessing the usability of information appliances using digital and physical mock-ups. Springer, Berlin. doi:10.1007/978-3-540-73335-5

  26. Kilic Delice E, Güngör Z (2009) The usability analysis with heuristic evaluation and analytic hierarchy process. Int J Ind Ergonomics 39(6):934–939. doi:10.1016/j.ergon.2009.08.005

    Article  Google Scholar 

  27. Kim J, Han S (2008) A methodology for developing a usability index of consumer electronic products. Int J Ind Ergonomics 38(3–4):333–345. doi:10.1016/j.ergon.2007.12.002

    Google Scholar 

  28. Kuutti K, Battarbee K, Säde S, Mattelmäki T, Keinonen T, Teirikko T, Tornberg A (2001) Virtual prototypes in usability testing. In: HICSS’01: proceedings of the 34th annual Hawaii international conference on system sciences (HICSS-34), vol 5, p 5029. IEEE Computer Society, Washington

  29. Lanzotti A, Matrone G, Tarantino P, Vanacore A (2008) Statistical design for innovation in VR. In: Erto P (eds) Statistics for innovation. Springer, Milan, pp 27–41

    Google Scholar 

  30. Liljegren E (2004) Cognitive engineering methods as usability evaluation tools for medical equipment. Int J Ind Ergonomics 34(1):49–62. doi:10.1016/j.ergon.2004.01.008

    Google Scholar 

  31. Lin M, Wang C, Chen M, Chang C (2008) Using ahp and topsis approaches in customer-driven product design process. Comput Ind 59(1):17–31. doi:10.1016/j.compind.2007.05.013

    Google Scholar 

  32. Mahdjoub M, Monticolo D, Gomes S, Sagot J (2009) A collaborative design for usability approach supported by virtual reality a multi-agent system embedded in a plm environment. Computer-Aided Design. doi:10.1016/j.cad.2009.02.009

  33. Montero F, Lozano M, Gonzalez P (2008) Usability-oriented quality model based on ergonomic criteria. Handbook of research on web information systems quality

  34. Muzzupappa M, Bruno F, Mattanò RM, Pina M (2006) A new approach to participatory design: usability tests in virtual environment. In: Research in interactive design, vol 1, Cap. X. SpringerVerlag., New York, pp 80–90

  35. Nagarajan U, Kantor G, Hollis RL (2009) Human–robot physical interaction with dynamically stable mobile robots. In: Proceedings of the 4th ACM/IEEE international conference on human robot interaction, HRI’09. ACM, New York, pp 281–282. doi:10.1145/1514095.1514176

  36. Nielsen J (1993) Usability engineering. Morgan Kaufmann Publishers Inc., San Francisco

    MATH  Google Scholar 

  37. Norman D (2004) Emotional design. Perseus Publishing, New York

    Google Scholar 

  38. Park S (1996) Robust design and analysis for quality engineering. Kluwer Academic Publishers, London

    Google Scholar 

  39. Petrie H, Hamilton F, King N, Pavan P (2006) Remote usability evaluations with disabled people. In: CHI’06: proceedings of the SIGCHI conference on human factors in computing systems, pp 1133–1141. ACM, New York. doi:10.1145/1124772.1124942

  40. Pithon T, Weiss T, Richir S, Klinger E (2009) Wheelchair simulators: a review. Technol Disabil 21(1):1–s10

    Google Scholar 

  41. Saaty L (2008) Decision making with the analytic hierarchy process. Int J Services Sci 1(1):83–98. doi:10.1504/IJSSCI.2008.017590

    Google Scholar 

  42. Sauer J, Seibel K, Rüttinger B (2009) The influence of user expertise and prototype fidelity in usability tests. Appl Ergonomics. doi:10.1016/j.apergo.2009.06.003

  43. Sauro J, Kindlund E (2005) A method to standardize usability metrics into a single score. In: CHI’05: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 401–409. doi:10.1145/1054972.1055028

  44. Shackel B, Richardson S (2008) Human factors for informatics usability. Cambridge University Press, New York

    Google Scholar 

  45. Sharma V, Simpson R, LoPresti E, Mostowy C, Olson J, Puhlman J, Hayashi S, Cooper R, Konarski E, Kerley B (2008) Participatory design in the development of the wheelchair convoy system. J NeuroEng Rehabil 5(1):1 (2008). doi:10.1186/1743-0003-5-1. http://www.jneuroengrehab.com/content/5/1/1

    Google Scholar 

  46. Tullis T, Albert B (2008) Measuring the user experience. Morgan Kauffmann Publishers, San Francisco

    Google Scholar 

  47. Wallergard M, Eriksson J, Johansson G (2008) A suggested virtual reality methodology allowing people with cognitive disabilities to communicate their knowledge and experiences of public transport systems. Technol Disabil 20(1):9–24

    Google Scholar 

Download references

Acknowledgments

The present work was developed with the contribution of the Italian Ministry of Education, University and Research (MIUR) within the framework of the PRIN 2008 project called “Innovation in service quality management: statistical approach and application in some fields of national interest”. The authors also thank “Polo ST UniNa” that has partially funded the work within the framework of “Faro Project”. Finally, the authors would like to thank Agostino De Santis for his interesting discussions about assistive robotics and Human–Robot Interaction and Amedeo Sgura for his helpful research work during his master degree thesis and the anonymous referees for their suggestions. We would also like to mention Neatech s.r.l. for having provided the physical wheelchair with the corresponding CAD model used for VR experiments.

Author information

Authors and Affiliations

  1. PRO-I Research Group, University of Naples Federico II, Naples, Italy

    G. Di Gironimo, G. Matrone, A. Tarallo, M. Trotta & A. Lanzotti

Authors
  1. G. Di Gironimo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G. Matrone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Tarallo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Trotta
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. A. Lanzotti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G. Di Gironimo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Di Gironimo, G., Matrone, G., Tarallo, A. et al. A virtual reality approach for usability assessment: case study on a wheelchair-mounted robot manipulator. Engineering with Computers 29, 359–373 (2013). https://doi.org/10.1007/s00366-012-0274-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00366-012-0274-x

Keywords

Search

Navigation