Incorporating Bayesian learning in agent-based simulation of stakeholders’ negotiation

https://doi.org/10.1016/j.compenvurbsys.2014.07.003Get rights and content

Highlights

  • An agent-based model was developed to simulate stakeholders’ negotiation regarding land development.

  • A Bayesian learning approach was used to enable the Developer agent with learning capability.

  • Two scenarios which included change in location and land-use composition were run both with and without learning capability.

  • The results indicate that with Bayesian learning an agreement can be reached in a fewer number of negotiation rounds.

Abstract

This paper describes the incorporation of a Bayesian learning algorithm into an agent-based model designed to simulate stakeholders’ negotiation when evaluating scenarios of land development. The objective is to facilitate reaching an agreement at an earlier stage in the negotiation by providing the opportunity to the proposer agent to learn his opponents’ preferences. The modeling approach is tested in the Elbow River watershed, in southern Alberta, Canada, that is under considerable pressure for land development due to the proximity of the fast growing city of Calgary. Five agents are included in the model respectively referred to as the Developer agent, the Planner agent, the Citizen agent, the AgricultureConcerned agent, and the WaterConcerned agent. Two types of land development scenarios are evaluated; in the first case, only the geographical location is considered while in the second case, the internal land-use composition is also varied. The Developer agent that is equipped with the Bayesian learning capability attempts to approximate its opponents’ fuzzy evaluation functions based on the responses he receives from them at each round of the negotiation. The results indicate that using this approach, an agreement can be reached in fewer number of negotiation rounds than in the case where the Developer agent selects the subsequent offers based merely on its own utility. The model also indicates how the satisfaction of each agent evolves during the negotiation. This information is very useful for decision makers who wish to consider stakeholders’ perspectives when dealing with multiple objectives in a spatial context.

Introduction

Global urbanization and the resulting concerns about land sustainability have generated an urgent need for examining scenarios of land development (Wu, 1996). These scenarios are images of future land-use patterns if certain land development regulations were to be adopted by decision makers (Xiang & Clarke, 2003). Different “what if” scenarios based on stakeholder inputs and feedback facilitate the investigation of possible land development patterns without bearing the costs of implementing them (Van Noordwijk, Tomich, & Verbist, 2003). This process of incorporating multiple views and coping with pluralistic wishes requires negotiation (Forester, 1999). Negotiation is a complex decision-making process where each party autonomously represents its viewpoints and interacts with the other parties to resolve conflicts and reach an agreement while attempting to maximize all parties’ payoffs (Choi et al., 2001, Jennings et al., 2001). It typically involves a combination of objective facts along with values and emotions and can be highly deviated from rationality due to individual and competitive biases (Bazerman & Moore, 2008). Computer models can facilitate human negotiation by processing a wide range of alternatives and examining their outcomes in the presence of biases (Oliver, 1997).

While land development scenarios have been in practice for years, it is only in the past two decades that the employment of computer models for creating and evaluating them has become possible. These models vary from GIS functionalities (Almeida et al., 2005, Batty and Xie, 1994, Hilferink and Rietveld, 1999, Joao and Walsh, 1992) to sophisticated computational approaches, such as agent-based modeling (ABM) in which the spatial capabilities of GIS are combined to Artificial Intelligence techniques (Benenson and Torrens, 2003, Ligmann-Zielinska and Jankowski, 2010, Matthews et al., 2007). Software agents as autonomous problem solving entities can support the automation of complex negotiations by negotiating on the behalf of stakeholders and providing adequate strategies to achieve realistic, win–win agreements (Rahwan, Kowalczyk, & Pham, 2002).

Agent-based modeling (ABM), which has roots in Artificial Intelligence, possesses outstanding features for simulating and testing scenarios to support decision making (Mensonides, Huisman, & Dignum, 2008). It employs a bottom-up approach in which the interactions of the individual decision makers are simulated (Bone, Dragicevic, & White, 2011). In ABMs, entities of the system being investigated are represented as autonomous individual agents that are intelligent and purposeful and act based on their own interests, values and goals (Matthews et al., 2007). They are aware of their environment, can communicate with each other and adapt their behavior (Beck, Kempener, Cohen, & Petrie, 2008). This modeling approach is particularly adapted to deal with situations where the agents seek their own benefit in the usage of a limited common resource and where a solution needs to be reached to ensure the sustainability of this resource (Marceau, 2008). The capability of these models to connect heterogeneous individual behaviors to collectively emerging patterns makes them suitable for modeling land development scenarios, which requires considering a pluralistic standpoint towards the problem in hand (Lempert, 2002).

Agent-based automated negotiation refers to negotiation conducted with computer agents using artificial intelligence techniques in which two or more agents multilaterally bargain resources for mutual intended gain (Beam & Segev, 1997). A computer agent is situated in some environment and is capable of flexible problem solving behavior to fulfil a specific purpose (Jennings et al., 2001). It has been demonstrated that negotiating agents may obtain significantly improved outcomes compared to results achieved by humans (Jonker et al. 2012). Different agent-based negotiation models have been proposed (Lopes, Wooldridge, & Novais, 2009). Game-theoretic models are particularly interesting in the context of land development. In these models, the parties choose a strategy to maximize the negotiation outcome by an iterative exchange of proposals. If the preference information of a player is known to all other players, then the game is one with complete information; otherwise it is called a game with incomplete information (Ausubel, Cramton, & Deneckere, 2002). In a multi-objective negotiation regarding shared environmental resources such as land, dealing with incomplete information is typically the case.

In the absence of complete information, learning techniques can be used by the agents to acquire knowledge about the other agents’ preferences or changes in the environment. Incorporating learning techniques in negotiation offers two main advantages (Gerding, van Bragt, & La Poutre, 2000). First, an agent can adjust its own negotiating strategies to obtain better deals based on its previous negotiation experiences. Second, learning can be used to update expectations regarding other parties’ strategies. A suitable conflict management approach such as a negotiation must foster learning among the parties (Lee, 1994). This is vital to the sustainability of decisions in any natural system (Daniels & Walker, 1996). The elements of such systems need to adapt to changing environments and such adaptation is done through learning. While it is an inherent feature of human decision making process (Daniels & Walker, 1996), a computer model which attempts at simulating such decision making needs to accommodate learning as well. Learning in this context not only improves the negotiation outcomes, but also provides insights into the possible avenues for agreement in real world negotiations. Small changes in learned behaviors can often result in unpredictable changes in the resulting macro-level emergent properties of the multi-agent group as a whole (Panait & Luke, 2005).

Due to the semi-cooperative nature of land development, in which agents compete over a resource but also attempt to perform a common task, the notion of learning is particularly significant. While learning is a missing component in many real world negotiations of land development (Forester, 1999), a simulation model aimed at improving such negotiations need to explicitly incorporate learning. The stakeholders as users of a negotiation support system equipped with learning capability can investigate the evolution of opinions among the opponents that results from the learning capability. They can understand the significance of learning the opponents’ perspectives and how it enhances the negotiation outcomes.

Several learning approaches have been used in agent-based negotiation to facilitate the agreement among agents (Panait and Luke, 2005, Weiß, 1996). They aim at obtaining a better performance in the future based on the experiences gained in the past (Alpaydin, 2004, Kulkarni, 2012). One of the popular learning approaches in agent-based negotiation is Reinforcement Learning (RL). In RL, a numerical performance measure representing an objective is being maximized (Szepesvári, 2010). At each iteration, the agent takes an action that changes the state of the environment; such transition is communicated to the agent through a scalar reward called reinforcement signal that evaluates the quality of the transition (Kaelbling, Littman, & Moore, 1996). The study of Bone and Dragićević (2010) is a good example of the use of RL to improve the negotiation results in a multi-stakeholder agent-based forest management model. However, a common issue with RL is to find a balance between exploration that consists in taking sub-optimal actions to discover new features, and exploitation that involves using the knowledge currently available about the world (Coggan, 2004). Each action must be repeated several times to obtain a reliable estimate of its expected reward (Kulkarni, 2012). Generalization is another issue in RL in which a function approximator such as neural network is needed to generalize between similar situations and actions (Boyan and Moore, 1995, Sutton, 1996).

Other learning techniques have been employed in agent-based negotiation. Choi et al. (2001) used a genetic algorithm to enable an agent to learn its opponents‘ preferences based on the counter-offers received during the previous rounds of negotiation. This approach requires a large number of rounds to obtain meaningful results. Carbonneau, Kersten, and Vahidov (2008) used a neural network to predict the opponents‘ negotiation moves in electronic negotiations. Other than the requirement for a large number of negotiation rounds, the generalizability of the approach presented in this study is also limited.

A promising approach to deal with the issue of learning in agent-based negotiation is Bayesian learning in which the probability of a hypothesis is updated based on acquired evidence. In other words, the posterior probability distribution of a hypothesis is computed conditioned to the evidence obtained through new data. It has been demonstrated that Bayesian learning provides the opportunity to learn an opponent‘s evaluation function in a fewer negotiation rounds in comparison with a no-learning scenario (Hindriks & Tykhonov, 2008). Moreover Bayesian learning is not data intensive and can yield noticeable results in a reasonable number of negotiation rounds (Kotsiantis, 2007). Domingos and Pazzani (1997) performed a large-scale comparison of the Bayesian approach with a number of algorithms for decision tree induction, instance-based learning, and rule induction on standard benchmark datasets, and found it to be superior in comparison with other learning schemes, even on datasets with substantial feature dependencies. They also concluded that the Bayesian approach can be a better method than most powerful alternatives when the sample size is small. This is important in land management negotiation, in which the number of land development scenarios that a developer can propose is limited and therefore the search space, i.e. the number of possible alternatives, is not large. In such a case, the learning needs to be accomplished in a few rounds of negotiation. This is in contrast with cases where several negotiation rounds must be completed for the learning to be achieved.

One of the initial attempts to incorporate Bayesian learning in agent negotiation was made by Zeng and Sycara (1998) who developed a sequential decision making model called Bazaar in which the agents were able to learn the opponents‘ preferences. However, the model was not suitable for a negotiation problem in a dynamic environment where the agents‘ actions change throughout the negotiation (Li & Cao, 2004). In another study conducted by Bui, Venkatesh, and Kieronska (1999), agents equipped with Bayesian learning capability work together to book meetings on the behalf of their respective users. When using the learning techniques, accurate predictions are made by the agents about their opponents‘ utility functions resulting in overall better performance. Ren and Anumba (2002) employed the Bayesian learning approach to facilitate negotiation among participants in a multi-agent system called MASCOT designed for constructing claims negotiation. While it improved the negotiation results, it was conditioned to the fact that the agents could gain enough prior knowledge about their opponents. Buffett and Spencer (2007) examined the effectiveness of Bayesian learning for learning an opponent’s preferences during a bilateral multi-issue negotiation. Using a hypothetical negotiation scenario, they were able to determine the opponents‘ preferences in a few rounds of negotiation.

Jacobs and Kruschke (2011) argue that many aspects of human learning can be captured by the Bayesian approach. The ability of people to learn from limited data can be addressed in a Bayesian framework by strong constraints on the prior beliefs. Moreover the assumptions made in Bayesian models are often expressed as well-defined mathematical expressions which make them easy to examine, evaluate, and modify. Additionally, since the Bayesian models update probabilities for all possible values of the variables, they yield a distribution over all possible outcomes rather than a fixed point, which enables them to update several competing hypotheses.

This paper describes the incorporation of Bayesian learning in an ABM to simulate the negotiation process of multiple stakeholders regarding land development scenarios. This work is a continuation of a previous study conducted by the authors (Pooyandeh & Marceau, 2013) which aimed at building a spatial negotiation support system for stakeholders using an agent-based model in a web-based participatory environment. While the model adequately captured different aspects of the negotiation process, it lacked the important notion of learning among agents, which is an essential process in real-world stakeholders’ negotiation. In this study, our goal is to evaluate the impact of adding a learning component to the achievement of agreement among agents and to examine how it affects the negotiation behavior of the agents. A learning component is added into the negotiation process to better mimic human behavior and facilitate the result of the negotiation.

Section snippets

Methodology

In this section, the study area is introduced along with its significance in the context of land development. In Sections 2.2.1 The agents and their preferences, 2.2.2 The agents‘ evaluation functions, the essential components of the agent-based model are presented and the criteria used by the agents are described. The updated negotiation model and the learning of the agents are described in Section 2.2.3.

Results

Fig. 7 displays the results of the negotiation for different cases. Fig. 7a illustrates the values obtained for the LocationChange_No-learning case. As it can be seen, it takes 11 rounds for the agents to reach an agreement. The utilities of the Planner and Developer agent change more smoothly compared to the other agents. Moreover the Planner’s utility values are closer to those of the Developer agent compared to the other agents. The WaterConcerned agent’s utility values increase smoothly

Conclusion

This study was undertaken to examine the impact of incorporating a learning technique to improve the achievement of agreement in agent-based negotiation regarding land development in the Elbow River watershed in southern Alberta. The results indicate that the learning module enhances the negotiation and reduces the number of rounds required by the agents to achieve an agreement. They highlight the significance of learning among the parties by considering the opponents’ perspectives in the

Acknowledgements

This research was funded by a research Grant awarded by Tecterra to D. Marceau and by University of Calgary’s scholarships awarded to M. Pooyandeh. We are grateful to the stakeholders for their invaluable contribution to this project. We would also like to thank Dr. Scott Heckbert for his constructive comments and intellectual feedback.

References (66)

  • E.M. Joao et al.

    GIS implications for hydrologic modeling: Simulation of nonpoint pollution generated as a consequence of watershed development scenarios

    Computers, Environment and Urban Systems

    (1992)
  • A. Ligmann-Zielinska et al.

    Exploring normative scenarios of land use development decisions with an agent-based simulation laboratory

    Computers, Environment and Urban Systems

    (2010)
  • M. Pooyandeh et al.

    A spatial web/agent-based model to support stakeholders’ negotiation regarding land development

    Journal of Environmental Management

    (2013)
  • Z. Ren et al.

    Learning in multi-agent systems: A case study of construction claims negotiation

    Advanced Engineering Informatics

    (2002)
  • I.B. Turksen

    Measurement of membership functions and their acquisition

    Fuzzy Sets and Systems

    (1991)
  • S. Wang

    Generating fuzzy membership functions: A monotonic neural network model

    Fuzzy Sets and Systems

    (1994)
  • F. Wu

    A linguistic cellular automata simulation approach for sustainable land development in a fast growing region

    Computers, Environment and Urban Systems

    (1996)
  • D. Zeng et al.

    Bayesian learning in negotiation

    International Journal of Human–Computer Studies

    (1998)
  • Alberta Government (2000). Guidelines for the Approval and Design of Natural and Constructed Treatment Wetlands for...
  • C.M.D. Almeida et al.

    GIS and remote sensing as tools for the simulation of urban land-use change

    International Journal of Remote Sensing

    (2005)
  • Alpaydin, E. (2004). Introduction to Machine Learning. MIT...
  • M. Batty et al.

    Research Article. Modelling inside GIS: Part 1. Model structures, exploratory spatial data analysis and aggregation

    International Journal of Geographical Information Systems

    (1994)
  • M.H. Bazerman et al.

    Judgment in managerial decision making

    (2008)
  • C. Beam et al.

    Automated negotiations: A survey of the state of the art

    Wirtschaftsinformatik

    (1997)
  • I. Benenson et al.

    Geographic automata systems: A new paradigm for integrating GIS and geographic simulation

  • Bilgiç, T., & Turksen, I. B. (1997). Elicitation of membership functions: How far can theory take us? In Proceedings of...
  • Bilgiç, T., & Türkşen, I. B. (2000). Measurement of membership functions: Theoretical and empirical work. In D. Dubois,...
  • C. Bone et al.

    Modeling-in-the-middle: Bridging the gap between agent-based modeling and multi-objective decision-making for land use change

    International Journal of Geographical Information Science

    (2011)
  • J. Boyan et al.

    Generalization in reinforcement learning: Safely approximating the value function

    Advances in Neural Information Processing Systems

    (1995)
  • H.H. Bui et al.

    Learning other agents’ preferences in multi-agent negotiation using the Bayesian classifier

    International Journal of Cooperative Information Systems

    (1999)
  • Civic Census Results City of Calgary (2012)....
  • Coggan, M. (2004). Exploration and exploitation in reinforcement learning. Research supervised by Prof. Doina Precup,...
  • P. Domingos et al.

    On the optimality of the simple Bayesian classifier under zero-one loss

    Machine Learning

    (1997)
  • Cited by (17)

    • An intelligent web-based spatial group decision support system to investigate the role of the opponents’ modeling in urban land use planning

      2022, Land Use Policy
      Citation Excerpt :

      They used MAS along with fuzzy set theory to represent the knowledge of decision-makers. In a follow-up study, they studied the influence of using learning approaches to better support the negotiation process on land developments (Pooyandeh and Marceau, 2014). Given this brief review of approaches, the question may be raised whether providing information about the stakeholders is beneficial for group decision-making in a ULUP process.

    • Socially rational agents in spatial land use planning: A heuristic proposal based negotiation mechanism

      2016, Computers, Environment and Urban Systems
      Citation Excerpt :

      In an application, they examined the impact of incorporating a learning technique to improve the achievement of agreement in agent-based negotiation regarding land development in southern Alberta. The agents were equipped with Bayesian learning capability to predict the evaluation functions of other agents (Pooyandeh & Marceau, 2014). Kamps (2013) developed a MAS based simulation model that links the form of residential developments to actor's bargain and negotiation power.

    • ATABS: A technique for automatically training agent-based simulators

      2016, Simulation Modelling Practice and Theory
      Citation Excerpt :

      Moreover, some works apply learning techniques in ABSs, but are aimed at purposes different from obtaining the similarity between real and simulated results. For example, Pooyandeh and Marceau [13] incorporate a Bayesian learning algorithm into an ABS for simulating stakeholders’ negotiations in scenarios of land development. Specifically, the proposer agent uses this algorithm to learn their opponents’ preferences.

    • Adaptive Probabilistic Behavioural Learning System for the effective behavioural decision in cloud trading negotiation market

      2016, Future Generation Computer Systems
      Citation Excerpt :

      In the cloud based e-commerce negotiation framework, optimal management of broker negotiation strategy poses a series of challenges due to lack of behavioural prediction regarding the opponent’s information [17]. The automated negotiation strategy can be classified into two types such as negotiation strategy without learning ability and with learning ability [18]. Lots of research works are available in case of negotiation strategy without learning ability where the negotiation attributes like price, speed, and time-slot constantly changes over the time [19–21].

    • Automated negotiation in environmental resource management: Review and assessment

      2015, Journal of Environmental Management
      Citation Excerpt :

      With the widespread use of the internet and the World Wide Web, it has received a lot of attention in domains such as supply chain management (Fink, 2006), political studies (Aragonès and Dellunde, 2009), and especially e-commerce (Ramchurn et al., 2007; Jazayeriy et al., 2011). However, its application in modeling stakeholders' negotiation over environmental resources is still in its infancy (Akhbari and Grigg, 2013; Okumura et al., 2013; Pooyandeh and Marceau, 2013, 2014). This is largely due to the characteristics of environmental issues, such as the amount of uncertainty involved, the high-stakes decisions, the diversity of perspectives, and the inexistence of optimum solutions.

    View all citing articles on Scopus
    View full text