Skip to main content
SpringerLink
Log in

Adjustable autonomy: a systematic literature review

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Developing autonomous systems that operate successfully in dynamic environments entails many challenges. Researchers introduce the concept of adjustable autonomy to mitigate some of these challenges. Adjustable autonomy enables a system to operate in different autonomic conditions and transfers control between the system’s operators. To gauge the extent to which such autonomy has been studied, this paper presents a systematic literature review of adjustable autonomy. It reviews 171 research papers and examines, in detail, 78 research papers. The review provides a fundamental understanding of adjustable autonomy and its application in multi-agent systems. The paper contributes to (1) identifying adjustable autonomy approaches and evaluating their utility, (2) specifying the requirements of formulating adjustable autonomy, (3) presenting adjustable autonomy assessment techniques, and (4) exploring the adjustable autonomy research and identify the research gaps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  • Alan A, Costanza E, Fischer J, Ramchurn SD, Rodden T, Jennings NR (2014) A field study of human-agent interaction for electricity tariff switching. In: Proceedings of the 2014 international conference on autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multi-agent Systems, pp 965–972

  • Alzahrani A, Callaghan V, Gardner M (2013) Towards adjustable autonomy in adaptive course sequencing. In: Intelligent environments (workshops), pp 466–477

  • Amalberti R (2006) Optimum system safety and optimum system resilience: agonistic or antagonistic concepts. In: Hollnagel E et al (eds) Resilience engineering: concepts and precepts. Ashgate, Farnham, pp 253–271

    Google Scholar 

  • Ball M, Callaghan V (2011) Introducing intelligent environments, agents and autonomy to users. In: 7th international conference on intelligent environments (IE)IEEE, pp 382–385

  • Ball M, Callaghan V (2012) Explorations of autonomy: an investigation of adjustable autonomy in intelligent environments. In: 2012 8th international conference on intelligent environments (IE)IEEE, pp 114–121

  • Beal J, Webb J, Atighetchi M (2010) Adjustable autonomy for cross-domain entitlement decisions. In: Proceedings of the 3rd ACM workshop on artificial intelligence and security. ACM, pp 65–71

  • Bibel W (2010) General aspects of intelligent autonomous systems. In: Pratihar DK, Lakhmi CJ (eds) Intelligent autonomous systems. Springer, Berlin, pp 5–27

  • Birk A, Pfingsthorn M (2006) A HMI supporting adjustable autonomy of rescue robots. In: Robocup 2005: Robot Soccer World Cup IX. Springer, Berlin, pp 255–266

  • Bradshaw JM, Sierhuis M, Gawdiak Y, Jeffers R, Suri N, Greaves M (2001) Teamwork and adjustable autonomy for the personal satellite assistant. In: Workshop on autonomy, delegation and control: interacting with autonomous agents. Seventeenth international joint conference on artificial inelligence (IJCAI-2001), Seattle, WA, vol 6

  • Bradshaw JM, Boy G, Durfee E, Gruninger M, Hexmoor H, Suri N, Tambe M, Uschold M, Vitek J (2002) Software agents for the warfighter. ITAC consortium report. AAAI Press/The MIT Press, Cambridge, MA. (14) Bradshaw, vol 1, pp 89–95

  • Bradshaw JM, Sierhuis M, Acquisti A, Feltovich P, Hoffman R, Jeffers R, Prescott D, Suri N, Uszok A, Van Hoof R (2003) Adjustable autonomy and human-agent teamwork in practice: an interim report on space applications. In: Hexmoor H, Castelfranchi C, Falcone R (eds) Agent autonomy. Springer, US, pp 243–280

  • Bradshaw JM, Jung H, Kulkarni S, Allen J, Bunch L, Chambers N, Feltovich P, Galescu L, Jeffers R, Johnson M, Taysom W (2004) Toward trustworthy adjustable autonomy and mixed-initiative interaction in KAoS. In: Proceedings of the AAMAS 2004 trust workshop, New York City, NY

  • Brereton B, Kitchenham A, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583

    Article  Google Scholar 

  • Brookshire J, Singh S, Simmons R (2004) Preliminary results in sliding autonomy for coordinated teams. In: Proceedings of the 2004 spring symposium series

  • Bush LA, Wang AJ, Williams BC (2012) Risk-based sensing in support of adjustable autonomy. In: Aerospace conference, 2012. IEEE, pp 1–18

  • Calero C, Bertoa M, Angeles Moraga M (2013) A systematic literature review for software sustainability measures. In: Proceedings of the 2nd international workshop on green and sustainable Software (GREENS), pp 46–53

  • Carlin A, Ayers J, Rousseau J, Schurr N. (2010). Agent-based coordination of human-multi-robot teams in complex environments. In: 9th international conference on autonomous agents and multi-agent systems: industry track. International Foundation for Autonomous Agents and Multi-agent Systems, pp 1747–1754

  • Ceballos A, Bensalem S, Cesta A, De Silva L, Fratini S, Ingrand F, Ocon J, Orlandini A, Py F, Rajan K, Rasconi R (2011) A goal-oriented autonomous controller for space exploration. ASTRA, p 11

  • Cheng MY, Cohen R (2005) A hybrid transfer of control model for adjustable autonomy multi-agent systems. In: Proceedings of the fourth international joint conference on autonomous agents and multi-agent systems. ACM, pp 1149–1150

  • Cheng MY, Cohen R (2006) Integrating information gathering interaction into transfer of control strategies in adjustable autonomy multi-agent systems. In: Lamontagne L, Marchand M (eds) Advances in artificial intelligence. Springer, Berlin, pp 1–12

  • Codognet P (2011) A simple language for describing autonomous agent behaviors. In: The seventh international conference on autonomic and autonomous systems (ICAS 2011)

  • Cote N, Bouzid M, Mouaddib AI (2013) Integrating human recommendations in the decision process of autonomous agents. In: 2013 IEEE/WIC/ACM international joint conferences on web intelligence (WI) and intelligent agent technologies (IAT)IEEE, vol 2, pp 195–200

  • Dallaire P, Giguère P, Émond D, Chaib-Draa B (2014) Autonomous tactile perception: a combined improved sensing and Bayesian nonparametric approach. Robot Autonom Syst 62(4):422–435

    Article  Google Scholar 

  • de Brun ML, Moffitt VZ, Franke JL, Yiantsios D, Housten T, Hughes A, Fouse S, Housten D (2008) Mixed-initiative adjustable autonomy for human/unmanned system teaming. In: AUVSI unmanned systems North America conference

  • De Visser EJ, LeGoullon M, Freedy A, Freedy E, Weltman G, Parasuraman R (2008) Designing an adaptive automation system for human supervision of unmanned vehicles: a bridge from theory to practice. In: Proceedings of the human factors and ergonomics society annual meeting, vol 52, no. 4. SAGE Publications, pp 221–225

  • De Visser EJ, LeGoullon M, Horvath D, Weltman G, Freedy A, Durlach P, Parasuraman R. (2010). TECRA: C2 application of adaptive automation theory. In: Aerospace conference, 2010 IEEE, pp 1–12. IEEE

  • Dorais G, Bonasso, RP, Kortenkamp D, Pell B, Schreckenghost D (1999) Adjustable autonomy for human-centered autonomous systems. In: Working notes of the sixteenth international joint conference on artificial intelligence workshop on adjustable autonomy systems, pp 16–35

  • Durand B, Godary-Dejean K, Lapierre L, Crestani D (2009) Inconsistencies evaluation mechanisms for a hybrid control architecture with adaptive autonomy. In: CAR’09: 4th national conference on control architectures of robots

  • Ermon S, Gomes C, Selman B, Vladimirsky A (2012) Probabilistic planning with non-linear utility functions and worst-case guarantees. In: Proceedings of the 11th international conference on autonomous agents and multiagent systems, vol 2, pp 965–972

  • Fagnant DJ, Kockelman KM (2014) The travel and environmental implications of shared autonomous vehicles, using agent-based model scenarios. Transp Res Part C Emerg Technol 40:1–13

    Article  Google Scholar 

  • Falcone R, Castelfranchi C (1999) Levels of delegation and levels of help for agents with adjustable autonomy. In: 1999 AAAI spring symposium on agents with adjustable autonomy, Mar 22–24, 1999. Stanford University, pp 25–32

  • Ferguson G, Allen JF, Miller BW (1996) TRAINS-95: towards a mixed-initiative planning assistant. In: AIPS, pp 70–77

  • Fleming M, and Cohen R (2004) A decision procedure for autonomous agents to reason about interaction with humans. In: Proceedings of the AAAI 2004 spring symposium on interaction between humans and autonomous systems over extended operation, pp 81–86

  • Freedy A, Sert O, Freedy E, McDonough J, Weltman G, Tambe M, Gupta T, Grayson W, Cabrera P (2008) Multi-agent adjustable autonomy framework (MAAF) for multi-robot, multi-human teams. In: International symposium on collaborative technologies and systems. CTS 2008 IEEE, pp 498–505

  • Goodrich MA, Olsen DR, Crandall JW, Palmer TJ (2001) Experiments in adjustable autonomy. In: Proceedings of IJCAI workshop on autonomy, delegation and control: interacting with intelligent agents, pp 1624–1629

  • Hexmoor H, Castelfranchi C, Falcone R (eds) (2012) Agent autonomy, vol 7. Springer, Berlin

    MATH  Google Scholar 

  • Huang HM, Messina E, Jacoff A (2009) Performance measures framework for unmanned systems (PerMFUS): initial perspective. In: Proceedings of the 9th workshop on performance metrics for intelligent systems, pp 65–72. ACM

  • Huber MJ (2007) Agent autonomy: social integrity and social independence. In: Fourth international conference on information technology, 2007. ITNG’07. IEEE, pp 282–290

  • Insaurralde CC, Lane DL (2014) Metric assessment of autonomous capabilities in unmanned maritime vehicles. Eng Appl Artif Intell 30:41–48

    Article  Google Scholar 

  • Inyiama HC, Ufoaroh SU, Chijindu VC. (2012). Intelligent agents autonomy issues. Afr J Comput ICT 5(4)

  • Jatoth C, Gangadharan GR, Buyya R (2015) Computational intelligence based QoS-aware web service composition: a systematic literature review. IEEE Trans Serv Comput, pp 1–18

  • Jennings NR, Moreau L, Nicholson D, Ramchurn S, Roberts S, Rodden T, Rogers A (2014) Human–agent collectives. Commun ACM 57(12):80–88

    Article  Google Scholar 

  • Johnson M, Bradshaw JM, Feltovich PJ, Jonker C, Van Riemsdijk B, Sierhuis M (2012) Autonomy and interdependence in human–agent-robot teams. Intell Syst IEEE 27(2):43–51

    Article  Google Scholar 

  • Johnson M, Bradshaw JM, Feltovich PJ, Jonker CM, Van Riemsdijk MB, Sierhuis M (2014) Coactive design: designing support for interdependence in joint activity. J Human Robot Interact 3(1):2014

    Article  Google Scholar 

  • Lewis B, Tastan B, Sukthankar G (2013) An adjustable autonomy paradigm for adapting to expert-novice differences. In: 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 1656–1662. IEEE

  • Lin L, Goodrich MA (2015) Sliding autonomy for UAV path-planning: adding new dimensions to autonomy management. In: Proceedings of the 2015 international conference on autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multi-agent Systems, pp 1615–1624

  • Luck M, D’Inverno M, Munroe S (2003) Autonomy: variable and generative. In: Hexmoor H, Castelfranchi, C, Falcone R (eds) Agent autonomy. Springer US, pp 11–28

  • Magill K, Erden YJ (2012) Autonomy and desire in machines and cognitive agent systems. Cognit Comput 4(3):354–364

    Article  Google Scholar 

  • Mercier S, Dehais F, Lesire C, Tessier C (2008a) Resources as basic concepts for authority sharing. Humans Operating Unmanned Systems, HUMOUS

  • Mercier S, Tessier C, Dehais F (2008b) Adaptive autonomy for a human-robot architecture. In: 3rd national conference on control architectures of robots, Bourges

  • Miller CA, Parasuraman R (2007) Designing for flexible interaction between humans and automation: delegation interfaces for supervisory control. Hum Fact J Hum Fact Ergon Soc 49(1):57–75

    Article  Google Scholar 

  • Moffitt VZ, Franke JL, Lomas M (2006) Mixed-initiative adjustable autonomy in multi-vehicle operations. In: Proceedings of AUVSI, Orlando, Florida

  • Mostafa SA, Ahmad MS, Annamalai M, Ahmad A, Gunasekaran SS (2013a) A conceptual model of layered adjustable autonomy. Adv Intell Syst Comput 206:619–630

    Google Scholar 

  • Mostafa SA, Ahmad MS, Annamalai M, Ahmad A, Gunasekaran SS (2013b) A dynamically adjustable autonomic agent framework. Adv Intell Syst Comput 206:631–642

    Google Scholar 

  • Mostafa SA, Ahmad MS, Annamalai M, Ahmad A, Basheer GS (2013c) Frontiers in artificial intelligence and applications. A layered adjustable autonomy approach for dynamic autonomy distribution. IOS Publisher, New York

    Google Scholar 

  • Mostafa SA, Ahmad MS, Ahmad A, Annamalai M, Mustapha A (2014a) A dynamic measurement of agent autonomy in the layered adjustable autonomy model. Stud Comput Intell 513(513):25–35

    Google Scholar 

  • Mostafa SA, Ahmad MS, Tang AY, Ahmad A, Annamalai M, Mustapha A (2014b) Agent’s autonomy adjustment via situation awareness. Lect Notes Comput Sci 8397:443–453

    Article  Google Scholar 

  • Mostafa S, Gunasekaran SS, Ahmad MS, Ahmad A, Annamalai M, Mustapha A (2014c) Defining tasks and actions complexity-levels via their deliberation intensity measures in the layered adjustable autonomy model. In: International conference on intelligent environments (IE), 2014 IEEE, pp 52–55

  • Mostafa SA, Ahmad MS, Ahmad A, Annamalai M, Gunasekaran SS (2015a) An autonomy viability assessment matrix for agent-based autonomous systems. In: International symposium on agents, multi-agent systems and robotics (ISAMSR), pp 53–58. IEEE

  • Mostafa SA, Ahmad MS, Annamalai M, Ahmad A, Gunasekaran SS (2015b) Formulating dynamic agents’ operational state via situation awareness assessment. Adv Intell Syst Comput 320:545–556

    Google Scholar 

  • Musliner D, Pell B (1999) Agents with adjustable autonomy. Papers from the AAAI spring symposium. Technical Report SS-99-06, AAAI Press, Menlo Park, CA

  • Myers KL, Morley DN (2003) Policy-based agent directability. In: Hexmoor H, Castelfranchi C, Falcone R (eds) Agent autonomy. Springer, US, pp 185–209

  • Naderpour M, Lu J (2013) A human situation awareness support system to avoid technological disasters. In: Vitoriano B, Montero J, Ruan D (eds) Decision aid models for disaster management and emergencies. Atlantis Press, pp 307–325

  • Pantelis PC, Baker CL, Cholewiak SA, Sanik K, Weinstein A, Wu CC, Tenenbaum JB, Feldman J (2014) Inferring the intentional states of autonomous virtual agents. Cognition 130(3):360–379

    Article  Google Scholar 

  • Parasuraman R, Sheridan TB, Wickens CD (2008) Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J Cognitive Eng Decis Mak 2(2):140–160

    Article  Google Scholar 

  • Pătraşcu M, Drăgoicea M (2014) Integrating agents and services for control and monitoring: managing emergencies in smart buildings. In: Borangiu T, Trentesaux D, Thomas A (eds) Service orientation in holonic and multi-agent manufacturing and robotics. Springer, pp 209–224

  • Petersen K, Von Stryk O (2011) Towards a general communication concept for human supervision of autonomous robot teams. In: Proceedings of the fourth international conference on advances in computer-human interactions (ACHI). pp 228–235

  • Pratihar DK, Jain LC (2010) Towards intelligent autonomous systems. In: Intelligent autonomous systems. Springer, Berlin, pp 1–4

  • Py F, Rajan K, McGann C (2010) A systematic agent framework for situated autonomous systems. In: Proceedings of the 9th international conference on autonomous agents and multi-agent systems: volume 2—Volume 2. International Foundation for Autonomous Agents and Multi-agent Systems, pp 583–590

  • Rajabzadeh RA (2014) Measuring autonomy and solving general stabilization problems with multi-agent systems. Master diss., University of Nebraska, Lincoln

  • Rau PLP, Li Y, Liu J (2013) Effects of a social robot’s autonomy and group orientation on human decision-making. Adv Hum Comput Interact 2013(11):1–13

  • Reed NE (2005) A user controlled approach to adjustable autonomy. In: Proceedings of the 38th annual Hawaii international conference on system sciences, 2005. HICSS’05. IEEE, pp 295b–295b

  • Roehr TM, Shi Y (2010) Using a self-confidence measure for a system-initiated switch between autonomy modes. In: Proceedings of the 10th international symposium on artificial intelligence, robotics and automation in space, Sapporo, Japan, pp 507–514

  • Rogers A, Ramchurn S, Jennings NR (2012) Delivering the smart grid: challenges for autonomous agents and multi-agent systems research. In: Twenty-sixth AAAI conference on artificial intelligence (AAAI-12), Toronto, CA, 22–26 July 2012, pp 2166–2172

  • Sadaghiyani JS (2011) Concepts of decision making under uncertain, risky and deterministic situations. J Math Comput Sci 2(3):529–545

    Article  Google Scholar 

  • Scerri P, Reed N (2001) Designing agents for systems with adjustable autonomy. Department of Computer and Information Science, Linköpings Universitet, Linköpings

    MATH  Google Scholar 

  • Scerri P, Sycara K, Tambe M (2009) Safety in the context of coordination via adjustable autonomy. In: Safety and security in multi-agent systems. Springer, Berlin, pp 208–224

  • Scerri P, Kannan B, Velagapudi P, Macarthur K, Stone P, Taylor M, Dolan J, Farinelli A, Chapman A, Dias B, Kantor G (2012) Flood disaster mitigation: a real-world challenge problem for multi-agent unmanned surface vehicles. In: Dechesne F, Hattori H, ter Mors A, Such JM, Weyns D, Dignum F (eds) Advanced agent technology. Springer, Berlin, pp 252–269

  • Schermerhorn P, Scheutz M (2009) Dynamic robot autonomy: investigating the effects of robot decision-making in a human–robot team task. In: Proceedings of the 2009 international conference on Multimodal interfaces. ACM, pp 63–70

  • Scholtz J, Antonishek B, Young J. (2004). Evaluation of a human–robot interface: development of a situational awareness assessment methodology (ETSIB03). In: International conference on annual Hawaii system sciences, pp 130–130

  • Schurr N (2007) Toward human–multi-agent teams. ProQuest, Ann Arbor

    Google Scholar 

  • Schurr N., Marecki J, Tambe M. (2008). RIAACT: a robust approach to adjustable autonomy for human-multi-agent teams. In: Proceedings of the 7th international joint conference on autonomous agents and multi-agent systems, Volume 3. International Foundation for Autonomous Agents and Multi-agent Systems, pp 1429–1432

  • Schurr N, Marecki J, Tambe M (2009) Improving adjustable autonomy strategies for time-critical domains. In: Proceedings of the 8th international conference on autonomous agents and multi-agent systems, Volume 1. International Foundation for Autonomous Agents and Multi-agent Systems, pp 353–360

  • Schwarz M, Stückler J, Behnke S (2014) Mobile teleoperation interfaces with adjustable autonomy for personal service robots. In: Proceedings of the 2014 ACM/IEEE international conference on human–robot interaction, pp 288–289. ACM

  • Sellner B, Heger FW, Hiatt LM, Simmons R, Singh S (2006) Coordinated multi-agent teams and sliding autonomy for large-scale assembly. Proc IEEE 94(7):1425–1444

    Article  Google Scholar 

  • Sycara K, Sukthankar G (2006) Literature review of teamwork models. Robotics Institute, Carnegie Mellon University, Pittsburgh

    Google Scholar 

  • Tipaldi M, Glielmo L (2015) A Markovian based approach for autonomous space systems. In: Metrology for aerospace (metroaerospace), 2015 IEEE, pp 426–430. IEEE

  • Truszkowski W, Hallock H, Rouff C, Karlin J, Rash J, Hinchey M, Sterritt R (2009) Autonomous and autonomic systems: with applications to NASA intelligent spacecraft operations and exploration systems. Springer, Berlin

    Google Scholar 

  • Valero-Gomez A, De La Puente P, Hernando M (2011) Impact of two adjustable autonomy models on the scalability of single-human/multiple-robot teams for exploration missions. Hum Factors J Hum Factors Ergon Soc 53(6):703–716

    Article  Google Scholar 

  • van der Vecht, B. (2009) Adjustable autonomy: controlling influences on decision making (No. 2009–20). Utrecht University

  • van der Vecht B, Dignum F, Meyer JJC, Neef M (2008) A dynamic coordination mechanism using adjustable autonomy. Springer, Berlin

    Google Scholar 

  • Wallace SA, Henry M (2008) Towards a generic infrastructure to adjust the autonomy of soar agents. In: FLAIRS conference, pp 119–120

  • Wardziński A. (2006). The role of situation awareness in assuring safety of autonomous vehicles. In: Górski J (ed) Computer safety, reliability, and security. Springer, Berlin, pp 205–218

  • Waytz A, Heafner J, Epley N (2014) The mind in the machine: anthropomorphism increases trust in an autonomous vehicle. J Exp Soc Psychol 52:113–117

    Article  Google Scholar 

  • Wilson MA, McMahon J, Aha DW (2014) Bounded expectations for discrepancy detection in goal-driven autonomy. In: Workshops at the twenty-eighth AAAI conference on artificial intelligence

  • Witkowski M, Stathis K (2004) A dialectic architecture for computational autonomy. In: Nickles M, Rovatsos M, Weiss G (eds) Agents and computational autonomy. Springer, Berlin, pp 261–273

  • Zieba S, Polet P, Vanderhaegen F, Debernard S (2010) Principles of adjustable autonomy: a framework for resilient human-machine cooperation. Cognit Technol Work 12(3):193–203

    Article  Google Scholar 

  • Zieba S, Polet P, Vanderhaegen F (2011) Using adjustable autonomy and human–machine cooperation to make a human–machine system resilient-application to a ground robotic system. Inf Sci 181(3):379–397

    Article  Google Scholar 

  • Zilberstein S (2015) Building strong semi-autonomous systems. In: AAAI, pp 4088–4092

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Batu Pahat, Johor, Malaysia

    Salama A. Mostafa & Aida Mustapha

  2. College of Information Technology, Universiti Tenaga Nasional, Selangor, Malaysia

    Mohd Sharifuddin Ahmad

Authors
  1. Salama A. Mostafa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohd Sharifuddin Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aida Mustapha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salama A. Mostafa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mostafa, S.A., Ahmad, M.S. & Mustapha, A. Adjustable autonomy: a systematic literature review. Artif Intell Rev 51, 149–186 (2019). https://doi.org/10.1007/s10462-017-9560-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-017-9560-8

Keywords

Search

Navigation