Skip to main content

Advertisement

SpringerLink
Log in

Automatic validation and failure diagnosis of human-device interfaces using task analytic models and model checking

  • SI: BRIMS 2011
  • Published:
Computational and Mathematical Organization Theory Aims and scope Submit manuscript

Abstract

When evaluating designs of human-device interfaces for safety critical systems, it is very important that they support the goal-directed tasks they were designed to facilitate. This paper describes a novel method that generates task-related temporal logic properties from task analytic models created early in the system design process. This allows analysts to use model checking (a means of performing exhaustive mathematical proofs) to automatically validate that formal models of human-device interfaces will let human operators successfully perform the necessary tasks with the system. This paper also presents an algorithm that uses the method to diagnose why a particular task is not valid for a given design. The application of both the method and algorithm are illustrated with a patient-controlled analgesia pump programming example. The method and algorithm are discussed and avenues for future work are described.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The analyst can stop the algorithm at any time during its execution if he or she feels enough information has been obtained.

  2. The task analysis findings presented here are theoretical and are used only for the purpose of illustrating the method presented in this paper.

  3. The presented device is based heavily on an actual PCA pump used in hospitals. Thus the presented HDI does not necessarily reflect the design philosophies of the author.

  4. Machine specifications are reported because they influence reported verification time statistics.

  5. All model checking verifications discussed in this list were performed in 24 seconds or less.

References

  • Abowd GD, Wang H, Monk AF (1995) A formal technique for automated dialogue development. In: Proceedings of the 1st conference on designing interactive systems. ACM Press, New York, pp 219–226

    Google Scholar 

  • Aït-Ameur Y, Baron M (2006) Formal and experimental validation approaches in HCI systems design based on a shared event B model. Int J Softw Tools Technol Transf 8(6):547–563

    Article  Google Scholar 

  • Aït-Ameur Y, Baron M, Girard P (2003) Formal validation of HCI user tasks. In: Proceedings of the international conference on software engineering research and practice. CSREA Press, Las Vegas, pp 732–738

    Google Scholar 

  • Amant R, Freed A, Ritter F (2005) Specifying act-r models of user interaction with a goms language. Cogn Syst Res 6(1):71–88

    Article  Google Scholar 

  • Basnyat S, Palanque P, Schupp B, Wright P (2007) Formal socio-technical barrier modelling for safety-critical interactive systems design. Saf Sci 45(5):545–565

    Article  Google Scholar 

  • Basnyat S, Palanque PA, Bernhaupt R, Poupart E (2008) Formal modelling of incidents and accidents as a means for enriching training material for satellite control operations. In: Proceedings of the joint ESREL 2008 and 17th SRA-Europe conference. Taylor and Francis, London, CD–ROM

    Google Scholar 

  • Bass EJ, Bolton ML, Feigh K, Griffith D, Gunter E, Mansky W, Rushby J (2011) Toward a multi-method approach to formalizing human-automation interaction and human-human communications. In: Proceedings of the IEEE international conference on systems, man, and cybernetics. IEEE, Piscataway, pp 1817–1824

    Google Scholar 

  • Basuki TA, Cerone A, Griesmayer A, Schlatte R (2009) Model-checking user behaviour using interacting components. Form Asp Comput 21(6):571–588

    Article  Google Scholar 

  • Bolton ML (2010) Using task analytic behavior modeling, erroneous human behavior generation, and formal methods to evaluate the role of human-automation interaction in system failure. PhD thesis, University of Virginia, Charlottesville

  • Bolton ML, Bass EJ (2009) A method for the formal verification of human interactive systems. In: Proceedings of the 53rd annual meeting of the human factors and ergonomics society. HFES, Santa Monica, pp 764–768

    Google Scholar 

  • Bolton ML, Bass EJ (2010a) Formally verifying human-automation interaction as part of a system model: limitations and tradeoffs. Innov Syst Softw Eng 6(3):219–231

    Article  Google Scholar 

  • Bolton ML, Bass EJ (2010b) Using task analytic models to visualize model checker counterexamples. In: Proceedings of the 2010 IEEE international conference on systems, man, and cybernetics. IEEE, Piscataway, pp 2069–2074

    Chapter  Google Scholar 

  • Bolton ML, Bass EJ, Siminiceanu RI (2012) Using formal verification to evaluate human-automation interaction in safety critical systems, a review. IEEE Trans Syst Man Cybern, Part A, Syst Hum (accepted)

  • Bolton ML, Siminiceanu RI, Bass EJ (2011) A systematic approach to model checking human-automation interaction using task-analytic models. IEEE Trans Syst Man Cybern, Part A, Syst Hum 41(5):961–976

    Article  Google Scholar 

  • Booher H, Minninger J (2003) Human systems integration in army systems acquisition. In: Booher HR (ed) Handbook of human systems integration. Wiley, Hoboken, pp 663–698

    Chapter  Google Scholar 

  • Bredereke J, Lankenau A (2005) Safety-relevant mode confusions–modelling and reducing them. Reliab Eng Syst Saf 88(3):229–245

    Article  Google Scholar 

  • Brito R (2009) The algorithms bundle. http://carroll.aset.psu.edu/pub/CTAN/macros/latex/contrib/algorithms/algorithms.pdf

  • Burch JR, Clarke EM, Dill DL, Hwang J, McMillan KL (1992) Symbolic model checking: 1020 states and beyond. Inf Comput 98(2):142–171

    Article  Google Scholar 

  • Byrne M, Kirlik A (2005) Using computational cognitive modeling to diagnose possible sources of aviation error. Int J Aviat Psychol 15(2):135–155

    Article  Google Scholar 

  • Campos JC, Harrison M (1997) Formally verifying interactive systems: a review. In: Proceedings of the fourth international Eurographics workshop on the design, specification, and verification of interactive systems. Springer, Berlin, pp 109–124

    Google Scholar 

  • Campos JC, Harrison MD (2008) Systematic analysis of control panel interfaces using formal tools. In: Proceedings of the 15th international workshop on the design, verification and specification of interactive systems. Springer, Berlin, pp 72–85

    Chapter  Google Scholar 

  • Campos JC, Harrison MD (2009) Interaction engineering using the ivy tool. In: Proceedings of the 1st ACM SIGCHI symposium on engineering interactive computing systems. ACM Press, New York, pp 35–44

    Chapter  Google Scholar 

  • Cerone A, PA Lindsay, Connelly S (2005) Formal analysis of human-computer interaction using model-checking. In: Proceedings of the 3rd IEEE international conference on software engineering and formal methods. IEEE Computer Society, Los Alamitos, pp 352–362

    Chapter  Google Scholar 

  • Chu RW, Mitchell CM, Jones PM (1995) Using the operator function model and OFMspert as the basis for an intelligent tutoring system: towards a tutor/aid paradigm for operators of supervisory control systems. IEEE Trans Syst Man Cybern, Part A, Syst Hum 25(7):1054–1075

    Article  Google Scholar 

  • Clarke E, Grumberg O, Jha S, Lu Y, Veith H (2003) Counterexample-guided abstraction refinement for symbolic model checking. J ACM 50(5):752–794

    Article  Google Scholar 

  • Clarke EM, Enders R, Filkorn T, Jha S (1996) Exploiting symmetry in temporal logic model checking. Form Methods Syst Des 9(1):77–104

    Article  Google Scholar 

  • Clarke EM, Grumberg O, Peled DA (1999) Model checking. MIT Press, Cambridge

    Google Scholar 

  • Cobleigh J, Giannakopoulou D, Păsăreanu C (2003) In: Proceedings of the 9th international conference on tools and algorithms for the construction and analysis of systems. Springer, Berlin, pp. 331–346

    Chapter  Google Scholar 

  • De Moura L, Owre S, Shankar N (2003) The SAL language manual. Tech. Rep. CSL-01-01, Computer Science Laboratory, SRI International, Menlo Park

  • Dwyer MB, Carr V, Hines L (1997) Model checking graphical user interfaces using abstractions. In: Proceedings of the sixth European software engineering conference. Springer, New York, pp 244–261

    Google Scholar 

  • Dwyer MB, Tkachuk O, Robby, Visser W (2004) Analyzing interaction orderings with model checking. In: Proceedings of the 19th IEEE international conference on automated software engineering. IEEE Computer Society, Los Alamitos, pp 154–163

    Google Scholar 

  • Emerson EA (1990) Temporal and modal logic. In: van Leeuwen J, Meyer AR, Nivat M, Paterson M, Perrin D (eds) Handbook of theoretical computer science. MIT Press, Cambridge, Chap 16, pp 995–1072

    Google Scholar 

  • Feary M (2007) Automatic detection of interaction vulnerabilities in an executable specification. In: Proceedings of the 7th international conference on engineering psychology and cognitive ergonomics. Springer, Berlin, pp 487–496

    Chapter  Google Scholar 

  • Fields RE (2001) Analysis of erroneous actions in the design of critical systems. PhD thesis, University of York, York

  • Göknur S, Bolton ML, Bass EJ (2004) Adding a motor control component to the operator function model expert system to investigate air traffic management concepts using simulation. In: Proceedings of the IEEE international conference and systems, man, and cybernetics. IEEE, Piscataway, pp 886–892

    Google Scholar 

  • Hamon G, De Moura L, Rushby J (2005) Automated test generation with SAL. Tech. rep., Menlo Park. http://www.csl.sri.com/users/rushby/papers/salatg.pdf

  • Hartson HR, Siochi AC, Hix D (1990) The UAN: a user-oriented representation for direct manipulation interface designs. ACM Trans Inf Syst 8(3):181–203

    Article  Google Scholar 

  • Holzmann G, Peled D (1994) An improvement in formal verification. In: Proceedings of the 7th international conference on formal description techniques. Chapman and Hall, London, pp 197–211

    Google Scholar 

  • John BE, Kieras DE (1996) Using GOMS for user interface design and evaluation: which technique? ACM Trans Comput-Hum Interact 3(4):287–319

    Article  Google Scholar 

  • Jonker CM, Schut MC, Treur J, Yolum P (2007) Analysis of meeting protocols by formalisation, simulation, and verification. Comput Math Organ Theory 13(3):283–314

    Article  Google Scholar 

  • Joshi A, Miller SP, Heimdahl MP (2003) Mode confusion analysis of a flight guidance system using formal methods. In: Proceedings of the 22nd digital avionics systems conference. IEEE, Piscataway, pp 2.D.1-1–2.D.1-12

    Google Scholar 

  • Kieras D (2003) Goms models for task analysis. Lawrence Erlbaum Associates, Mahwah, pp 83–116

    Google Scholar 

  • Kirwan B, Ainsworth LK (1992) A guide to task analysis. Taylor and Francis, London

    Google Scholar 

  • Lecerof A, Paternò F (1998) Automatic support for usability evaluation. IEEE Trans Softw Eng 24(10):863–888

    Article  Google Scholar 

  • Limbourg Q, Vanderdonckt J (2003) Comparing task models for user interface design. In: Diaper D, Stanton N (eds) The handbook of task analysis for human-computer interaction. Lawrence Erlbaum Associates, Mahwah, pp 135–154

    Google Scholar 

  • Loer K, Harrison MD (2006) An integrated framework for the analysis of dependable interactive systems (IFADIS): its tool support and evaluation. Autom Softw Eng 13(4):469–496

    Article  Google Scholar 

  • Mansouri-Samani M, Pasareanu CS, Penix JJ, Mehlitz PC, O’Malley O, Visser WC, Brat GP, Markosian LZ, Pressburger TT (2007) Program model checking: a practitioner’s guide. Tech. rep., Intelligent Systems Division, NASA Ames Research Center, Moffett Field

  • Mitchell CM, Miller RA (1986) A discrete control model of operator function: a methodology for information display design. IEEE Trans Syst Man Cybern, Part A, Syst Hum 16(3):343–357

    Article  Google Scholar 

  • Mueller S, Simpkins B, Anno G, Fallon C, Price O, McClellan G (2011) Adapting the task-taxon-task methodology to model the impact of chemical protective gear. Comput Math Organ Theory 17:251–271

    Article  Google Scholar 

  • Palanque PA, Bastide R, Senges V (1996) Validating interactive system design through the verification of formal task and system models. In: Proceedings of the IFIP TC2/WG2.7 working conference on engineering for human-computer interaction. Chapman and Hall, London, pp 189–212

    Google Scholar 

  • Parnas DL (1969) On the use of transition diagrams in the design of a user interface for an interactive computer system. In: Proceedings of the 24th national ACM conference. ACM Press, New York, pp 379–385

    Google Scholar 

  • Paternò F (1997) Formal reasoning about dialogue properties with automatic support. Interact Comput 9(2):173–196

    Article  Google Scholar 

  • Paternò F, Santoro C (2001) Integrating model checking and HCI tools to help designers verify user interface properties. In: Proceedings of the 7th international workshop on the design, specification, and verification of interactive systems. Springer, Berlin, pp 135–150

    Chapter  Google Scholar 

  • Paternò F, Mancini C, Meniconi S (1997) Concurtasktrees: a diagrammatic notation for specifying task models. In: Proceedings of the IFIP TC13 international conference on human-computer interaction. Chapman and Hall, London, pp 362–369

    Google Scholar 

  • Paternò F, Santoro C, Tahmassebi S (1998) Formal model for cooperative tasks: concepts and an application for en-route air traffic control. In: Proceedings of the 5th international conference on the design, specification, and verification of interactive systems. Springer, Vienna, pp 71–86

    Google Scholar 

  • Pew R, Mavor A (2007) Human-system integration in the system development process: a new look. National Academies Press, Washington

    Google Scholar 

  • Ritter F, Kukreja U, Amant R (2007) Including a model of visual processing with a cognitive architecture to model a simple teleoperation task. J Cogn Eng Decis Mak 1(2):121

    Article  Google Scholar 

  • Ritter FE, Van Rooy D, Amant RS, Simpson K (2006) Providing user models direct access to interfaces: an exploratory study of a simple interface with implications for HRI and HCI. IEEE Trans Syst Man Cybern, Part A, Syst Hum 36(3):592–601

    Article  Google Scholar 

  • Rukšenas R, Back J, Curzon P, Blandford A (2009) Verification-guided modelling of salience and cognitive load. Form Asp Comput 21(6):541–569

    Article  Google Scholar 

  • Rushby J (2002) Using model checking to help discover mode confusions and other automation surprises. Reliab Eng Syst Saf 75(2):167–177

    Article  Google Scholar 

  • Shankar N (2000) Symbolic analysis of transition systems. In: Proceedings of the international workshop on abstract state machines, theory and applications. Springer, London, pp 287–302

    Chapter  Google Scholar 

  • Wing JM (1990) A specifier’s introduction to formal methods. Computer 23(9):8, 10–22, 24

    Article  Google Scholar 

Download references

Acknowledgements

The author would like to thank Dorrit Billman and Michael Feary for the feedback they provided during the preparation of this manuscript. The work documented here was completed while the author was a Senior Research Associate for the San José State University Research Foundation at the NASA Ames Research Center. This work was supported in part by NASA cooperative agreement NNX08AX12A, the NASA Aviation Safety program, and the FAA-NASA Nextgen flight research project. The content is solely the responsibility of the author and does not necessarily represent the official views of NASA or the FAA.

Author information

Authors and Affiliations

  1. Department of Mechanical and Industrial Engineering, University of Illinois at Chicago, Chicago, IL, 60607, USA

    Matthew L. Bolton

Authors
  1. Matthew L. Bolton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew L. Bolton.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bolton, M.L. Automatic validation and failure diagnosis of human-device interfaces using task analytic models and model checking. Comput Math Organ Theory 19, 288–312 (2013). https://doi.org/10.1007/s10588-012-9138-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10588-012-9138-6

Keywords

Search

Navigation