Skip to main content
SpringerLink
Log in

A mixed integer programming (MIP) model for evaluating navigation and task planning of human–robot interactions (HRI)

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Exercise of robotics in many applications brings in concerns of human–robot interaction. This paper offers a mathematical model-based mission planning tool for optimizing operator workload and platform utilization in human/multi-robot (H/M-R) teams. None of the earlier methods consistently predicts fan-out (number and configuration of robots that can be operated simultaneously and effectively, a critical H/M-R design decision). In this research, a mixed integer programming (MIP) model and solution framework are proposed to provide better estimates of fan-out while explicitly considering the performance, mission characteristics, objective and task/environment complexity. The extent of each robot’s waiting time is restricted by a utilization threshold in the MIP model. The effect of environment’s complexity on the task effectiveness is considered, where robots’ performances deteriorate during switch and neglect times. Simulation results show that fan-out effect is dependent on interaction efficiency, neglect tolerance, as well as other parameters. Performance is most sensitive to environment’s complexity and least sensitive to utilization threshold. In addition, the MIP model reveals optimal control sequence of robots to prevent switching confusions and maximize team performance. Empirical evaluations show that this approach holds great promise for real-world scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

(Adapted from [10])

Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Liu C, Tomizuka M (2014) Modeling and controller design of cooperative robots in workspace sharing human–robot assembly teams. In: Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems, Chicago, IL, USA, pp 1386–1391

  2. Chai T, Ding J, Yu G, Wang H (2014) Integrated optimization for the automation systems of mineral processing. IEEE Trans Autom Sci Eng 11(4):965–982

    Article  Google Scholar 

  3. Kim HJ, Lee JH, Lee TE (2014) Non-cyclic scheduling of a wet station. IEEE Trans Autom Sci Eng 11(4):1262–1274

    Article  MathSciNet  Google Scholar 

  4. Shibata T, Wada K, Saito T, Tanie K (2005) Human interactive robot for psychological enrichment and therapy. In: Proceedings of the AISB 2005 symposium on robot companion hard problem and open challenges in human robot interaction, pp 98–109

  5. Husain A, Jones H, Kannan B, Wong U, Pimentel T, Tang S, Daftry S, Huber S, Whittaker WL (2012) Mapping planetary caves with an autonomous, heterogeneous robot team. IEEE, pp 1–13

  6. Chen F, Sekiyama K, Cannella F, Fukuda T (2014) Optimal subtask allocation for human and robot collaboration within hybrid assembly system. IEEE Trans Autom Sci Eng 11(4):1065–1075

    Article  Google Scholar 

  7. Chen F, Sekiyama K, Cannella F, Fukuda T (2013) Cooperative human robot interaction systems: IV. Communication of shared plans with Naive humans using gaze and speech. In: Proceedings of the 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS), Tokyo, Japan. IEEE, pp 129–136

  8. Abbott KA, Slotte SM, Stimson DK (1996) Federal Aviation Administration Human Factors team report on: the interfaces between flightcrews and modern flight deck systems. Federal Aviation Administration, Washington DC, Technical Report, June 1996, pp D1–D3. http://www.faa.gov/avr/afs/interfac.pdf

  9. Espinosa JA, Cadiz J, Rico-Gutierrez L, Kraut RE, Scherlis W, Lautenbacher G (2000) Coming to the wrong decision quickly: why awareness tools must be matched with appropriate tasks. In: Proceedings of the human factors in computing systems, computer–human interaction (CHI) conference. ACM Press

  10. Pires JN (2005) Semi-autonomous manufacturing systems: the role of human–machine interface software and of the manufacturing tracking software. Mechatron Int J 15(10):1191–1205

    Article  Google Scholar 

  11. Neto P, Pires JN, Moreira AP (2009) Accelerometer-based control of an industrial robotic arm. In: Proceedings of the 18th IEEE international symposium on robot and human interactive communication, Toyama, Japan, pp 1192–1197

  12. Hillenbrand U, Brunner B, Borst C, Hirzinger G (2004) The robutler: a vision-controlled hand-arm system for manipulating bottles and glasses. In: Proceedings of the 35th international symposium on robotics, Paris, France

  13. Yamamoto S, Valin JM, Nakadai K, Rouat J, Michaud F, Ogata T, Okuno HG (2005) Enhanced robot speech recognition based on microphone array source separation and missing feature theory. In: Proceedings of the 2005 ieee international conference on robotics and automation (ICRA 2005), Barcelona, Spain, pp 1477–1482

  14. Burke JL, Murphy RR, Rogers E, Lumelsky VJ, Scholtz J (2004) Final report for the DARPA/NSF interdisciplinary study on human–robot interaction. IEEE Syst Man Cybern C Appl Rev Spec Issue Hum Robot Interact 34(2):103–112

    Article  Google Scholar 

  15. Schreckenghost DK (1999) Checklists for human–robot collaboration during space operations. In: Proceedings of the 43rd annual meeting of the human factors and ergonomics society, Santa Monica, CA, pp 46–50

  16. Scholtz J, Bahrami S (2003) Human–Robot interaction: development of an evaluation methodology for the bystander role of interaction. Retrieved 23 Feb 2004, from http://www.isd.mel.ist.gov/documents/scholtz/Scholtz_Bahrami.pdf

  17. Murphy RR (2004) Human–robot interaction in rescue robotics. IEEE Syst Man Cybern C Appl Rev Spec Issue Hum Robot Interact 34(2):138–153

    Article  MathSciNet  Google Scholar 

  18. Fong T, Kaber D, Lewis M, Scholtz J, Shultz A, Steinfeld A (2004) Common metrics for human–robot interaction. In: Proceedings of IEEE international conference on intelligent robots and systems, Sendai, Japan

  19. Chen JYC, Haas EC, Pillalamarri K, Jacobson CN (2006) Human–robot interface: issues in operator performance interface design, and technologies. Army Research Laboratory, ARL-TR-3834

  20. Goodrich M, Olsen D (2003) Seven principles of efficient human–robot interaction. In: Proceedings of the 2003 IEEE international conference on systems, man and cybernetics, Washington DC, pp 3943–3948

  21. Dixon SR, Wickens CD, Chang D (2003) Comparing quantitative model predictions to experimental data in multiple-UAV flight control. In: Proceedings of the human factors and ergonomics society 47th annual meeting Santa Monica, CA, pp 104–108

  22. Schipani SP (2003) An evaluation of operator workload during partially-autonomous vehicle operations. In: Proceedings of PerMIS

  23. Gombolay M, Bair A, Huang C, Shah J (2017) Computational design of mixed-initiative human–robot teaming that considers human factors: situational awareness, workload, and workflow preferences. Int J Robot Res 36(5–7):597–617

    Article  Google Scholar 

  24. H. Tempelmeier,”Simultaneous buffer and workload optimization for asynchronous flow production systems. Department of Production Management, University of Cologne, Koeln, Germany. http://www.spw.uni-koeln.de/

  25. Crandall JW, Cummings M, Penna MD, de Jong PMA (2011) Computing the effects of operator attention allocation in human control of multiple robots. IEEE Trans Syst Man Cybern A Syst Hum 41(3):385–397

    Article  Google Scholar 

  26. Crandall JW, Cummings M (2007) Developing performance metrics for the supervisory control of multiple robots. In: Proceedings of the 2007 ACM/IEEE conference on human robot interaction, Arlington, VA

  27. Olsen D, Goodrich M (2003) Metrics for evaluating human–robot interactions. In: Proceedings of PERMIS

  28. Olsen D, Wood S (2004) Fan-out: measuring human control of multiple robots. In: Proceedings of the SIGCHI conference on human factors in computer systems, Vienna, Austria, pp 231–238

  29. Crandall JW, Goodrich M, Olsen D, Nielsen C (2005) Validating human–robot interaction schemes in multitasking environments. IEEE Trans Syst 35(4):438–448

    Google Scholar 

  30. Crandall JW, Cummings M (2007) Identifying predictive metrics for supervisory control of multiple robots. IEEE Trans Robot Autom 23(5):942–951

    Article  Google Scholar 

  31. Mau S, Dolan J (2006) Scheduling to minimize downtime in human–multirobot supervisory control. In: Proceedings of the 5th international workshop on planning and scheduling for space, pp 268–277

  32. Mau S, Dolan J (2007) Scheduling for humans in multirobot supervisory control. In: Proceedings of the 2007 IEEE/RSJ international conference on intelligent robots and systems, San Diego, CA, pp 1637–1643

  33. Donmez B, Pina P, Cummings M (2008) Evaluation criteria for human automation performance metrics. In: Proceedings of the PerMIS

  34. Geoffrion AM (1972) Generalized benders decomposition. J Optim Theory Appl 10(4):237–260

    Article  MathSciNet  MATH  Google Scholar 

  35. Duran MA, Grossmann LE (1986) An outer-approximation algorithm for a class of mixed-integer non-linear programs. Math Program 36(3):307–339

    Article  MATH  Google Scholar 

  36. Fletcher R, Leyffer S (1994) Solving mixed integer nonlinear programs by outer approximation. Math Program 66(1–3):327–349

    Article  MathSciNet  MATH  Google Scholar 

  37. Westerlund T, Petersson F (1995) An extended cutting plane method for solving convex MINLP problems. Comput Chem Eng 19(1):131–136

    Article  Google Scholar 

  38. Borchers B, Mitchell JE (1991) An improved branch and bound algorithm for mixed integer non-linear programs.RPI Math Report No 200

  39. Leyffer S (2001) Integrating SQP and branch-and-bound for mixed integer nonlinear programming. Comput Optim Appl 18:295–309

    Article  MathSciNet  MATH  Google Scholar 

  40. Floudas CA, Gounaris CE (2008) A review of recent advances in global optimization. J Glob Optim 45:3–38

    Article  MathSciNet  MATH  Google Scholar 

  41. Floudas CA, Akrotirianakis IG, Caratzoulas S, Meyer CA, Kallrath J (2005) Global optimization in the 21st century: advances and challenges. Comput Chem Eng 29:1185–1202

    Article  Google Scholar 

  42. Liao T, Socha K, Montes de Oca MA, Stutzle T, Dorigo M (2014) Ant colony optimization for mixed-variable optimization problems. IEEE Trans Evol Comput 18(4):503–518

    Article  Google Scholar 

  43. IBM ILOG CPLEX Optimization Studio 12.8. https://www.ibm.com/products/ilog-cplex-optimization-studio. Accessed 11 Dec 2018

  44. Hart SG, Staveland LE (1988) Development of the NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. North Holland, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  45. Sert O, Freedy E, McDonough J, Weltman G, Tambe M, Gupta T, Grayson W, Cabrera P (2008) Multiagent adjustable autonomy framework (MAAF) for multi-robot, multi-human teams. IEEE, pp 498–505

  46. Sherali HD, Adams WP (1998) Reformulation–linearization techniques for discrete optimization problems. In: Du DZ, Pardalos PM (eds) Handbook of combinatorial optimization. Springer, Boston

    Google Scholar 

Download references

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, Gazi Universiy, Maltepe, Ankara, Turkey

    Mehmet Burak Şenol

Authors
  1. Mehmet Burak Şenol
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehmet Burak Şenol.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Şenol, M.B. A mixed integer programming (MIP) model for evaluating navigation and task planning of human–robot interactions (HRI). Intel Serv Robotics 12, 231–242 (2019). https://doi.org/10.1007/s11370-019-00275-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-019-00275-w

Keywords

Search

Navigation