Incorporating Bayesian learning in agent-based simulation of stakeholders’ negotiation
Introduction
Global urbanization and the resulting concerns about land sustainability have generated an urgent need for examining scenarios of land development (Wu, 1996). These scenarios are images of future land-use patterns if certain land development regulations were to be adopted by decision makers (Xiang & Clarke, 2003). Different “what if” scenarios based on stakeholder inputs and feedback facilitate the investigation of possible land development patterns without bearing the costs of implementing them (Van Noordwijk, Tomich, & Verbist, 2003). This process of incorporating multiple views and coping with pluralistic wishes requires negotiation (Forester, 1999). Negotiation is a complex decision-making process where each party autonomously represents its viewpoints and interacts with the other parties to resolve conflicts and reach an agreement while attempting to maximize all parties’ payoffs (Choi et al., 2001, Jennings et al., 2001). It typically involves a combination of objective facts along with values and emotions and can be highly deviated from rationality due to individual and competitive biases (Bazerman & Moore, 2008). Computer models can facilitate human negotiation by processing a wide range of alternatives and examining their outcomes in the presence of biases (Oliver, 1997).
While land development scenarios have been in practice for years, it is only in the past two decades that the employment of computer models for creating and evaluating them has become possible. These models vary from GIS functionalities (Almeida et al., 2005, Batty and Xie, 1994, Hilferink and Rietveld, 1999, Joao and Walsh, 1992) to sophisticated computational approaches, such as agent-based modeling (ABM) in which the spatial capabilities of GIS are combined to Artificial Intelligence techniques (Benenson and Torrens, 2003, Ligmann-Zielinska and Jankowski, 2010, Matthews et al., 2007). Software agents as autonomous problem solving entities can support the automation of complex negotiations by negotiating on the behalf of stakeholders and providing adequate strategies to achieve realistic, win–win agreements (Rahwan, Kowalczyk, & Pham, 2002).
Agent-based modeling (ABM), which has roots in Artificial Intelligence, possesses outstanding features for simulating and testing scenarios to support decision making (Mensonides, Huisman, & Dignum, 2008). It employs a bottom-up approach in which the interactions of the individual decision makers are simulated (Bone, Dragicevic, & White, 2011). In ABMs, entities of the system being investigated are represented as autonomous individual agents that are intelligent and purposeful and act based on their own interests, values and goals (Matthews et al., 2007). They are aware of their environment, can communicate with each other and adapt their behavior (Beck, Kempener, Cohen, & Petrie, 2008). This modeling approach is particularly adapted to deal with situations where the agents seek their own benefit in the usage of a limited common resource and where a solution needs to be reached to ensure the sustainability of this resource (Marceau, 2008). The capability of these models to connect heterogeneous individual behaviors to collectively emerging patterns makes them suitable for modeling land development scenarios, which requires considering a pluralistic standpoint towards the problem in hand (Lempert, 2002).
Agent-based automated negotiation refers to negotiation conducted with computer agents using artificial intelligence techniques in which two or more agents multilaterally bargain resources for mutual intended gain (Beam & Segev, 1997). A computer agent is situated in some environment and is capable of flexible problem solving behavior to fulfil a specific purpose (Jennings et al., 2001). It has been demonstrated that negotiating agents may obtain significantly improved outcomes compared to results achieved by humans (Jonker et al. 2012). Different agent-based negotiation models have been proposed (Lopes, Wooldridge, & Novais, 2009). Game-theoretic models are particularly interesting in the context of land development. In these models, the parties choose a strategy to maximize the negotiation outcome by an iterative exchange of proposals. If the preference information of a player is known to all other players, then the game is one with complete information; otherwise it is called a game with incomplete information (Ausubel, Cramton, & Deneckere, 2002). In a multi-objective negotiation regarding shared environmental resources such as land, dealing with incomplete information is typically the case.
In the absence of complete information, learning techniques can be used by the agents to acquire knowledge about the other agents’ preferences or changes in the environment. Incorporating learning techniques in negotiation offers two main advantages (Gerding, van Bragt, & La Poutre, 2000). First, an agent can adjust its own negotiating strategies to obtain better deals based on its previous negotiation experiences. Second, learning can be used to update expectations regarding other parties’ strategies. A suitable conflict management approach such as a negotiation must foster learning among the parties (Lee, 1994). This is vital to the sustainability of decisions in any natural system (Daniels & Walker, 1996). The elements of such systems need to adapt to changing environments and such adaptation is done through learning. While it is an inherent feature of human decision making process (Daniels & Walker, 1996), a computer model which attempts at simulating such decision making needs to accommodate learning as well. Learning in this context not only improves the negotiation outcomes, but also provides insights into the possible avenues for agreement in real world negotiations. Small changes in learned behaviors can often result in unpredictable changes in the resulting macro-level emergent properties of the multi-agent group as a whole (Panait & Luke, 2005).
Due to the semi-cooperative nature of land development, in which agents compete over a resource but also attempt to perform a common task, the notion of learning is particularly significant. While learning is a missing component in many real world negotiations of land development (Forester, 1999), a simulation model aimed at improving such negotiations need to explicitly incorporate learning. The stakeholders as users of a negotiation support system equipped with learning capability can investigate the evolution of opinions among the opponents that results from the learning capability. They can understand the significance of learning the opponents’ perspectives and how it enhances the negotiation outcomes.
Several learning approaches have been used in agent-based negotiation to facilitate the agreement among agents (Panait and Luke, 2005, Weiß, 1996). They aim at obtaining a better performance in the future based on the experiences gained in the past (Alpaydin, 2004, Kulkarni, 2012). One of the popular learning approaches in agent-based negotiation is Reinforcement Learning (RL). In RL, a numerical performance measure representing an objective is being maximized (Szepesvári, 2010). At each iteration, the agent takes an action that changes the state of the environment; such transition is communicated to the agent through a scalar reward called reinforcement signal that evaluates the quality of the transition (Kaelbling, Littman, & Moore, 1996). The study of Bone and Dragićević (2010) is a good example of the use of RL to improve the negotiation results in a multi-stakeholder agent-based forest management model. However, a common issue with RL is to find a balance between exploration that consists in taking sub-optimal actions to discover new features, and exploitation that involves using the knowledge currently available about the world (Coggan, 2004). Each action must be repeated several times to obtain a reliable estimate of its expected reward (Kulkarni, 2012). Generalization is another issue in RL in which a function approximator such as neural network is needed to generalize between similar situations and actions (Boyan and Moore, 1995, Sutton, 1996).
Other learning techniques have been employed in agent-based negotiation. Choi et al. (2001) used a genetic algorithm to enable an agent to learn its opponents‘ preferences based on the counter-offers received during the previous rounds of negotiation. This approach requires a large number of rounds to obtain meaningful results. Carbonneau, Kersten, and Vahidov (2008) used a neural network to predict the opponents‘ negotiation moves in electronic negotiations. Other than the requirement for a large number of negotiation rounds, the generalizability of the approach presented in this study is also limited.
A promising approach to deal with the issue of learning in agent-based negotiation is Bayesian learning in which the probability of a hypothesis is updated based on acquired evidence. In other words, the posterior probability distribution of a hypothesis is computed conditioned to the evidence obtained through new data. It has been demonstrated that Bayesian learning provides the opportunity to learn an opponent‘s evaluation function in a fewer negotiation rounds in comparison with a no-learning scenario (Hindriks & Tykhonov, 2008). Moreover Bayesian learning is not data intensive and can yield noticeable results in a reasonable number of negotiation rounds (Kotsiantis, 2007). Domingos and Pazzani (1997) performed a large-scale comparison of the Bayesian approach with a number of algorithms for decision tree induction, instance-based learning, and rule induction on standard benchmark datasets, and found it to be superior in comparison with other learning schemes, even on datasets with substantial feature dependencies. They also concluded that the Bayesian approach can be a better method than most powerful alternatives when the sample size is small. This is important in land management negotiation, in which the number of land development scenarios that a developer can propose is limited and therefore the search space, i.e. the number of possible alternatives, is not large. In such a case, the learning needs to be accomplished in a few rounds of negotiation. This is in contrast with cases where several negotiation rounds must be completed for the learning to be achieved.
One of the initial attempts to incorporate Bayesian learning in agent negotiation was made by Zeng and Sycara (1998) who developed a sequential decision making model called Bazaar in which the agents were able to learn the opponents‘ preferences. However, the model was not suitable for a negotiation problem in a dynamic environment where the agents‘ actions change throughout the negotiation (Li & Cao, 2004). In another study conducted by Bui, Venkatesh, and Kieronska (1999), agents equipped with Bayesian learning capability work together to book meetings on the behalf of their respective users. When using the learning techniques, accurate predictions are made by the agents about their opponents‘ utility functions resulting in overall better performance. Ren and Anumba (2002) employed the Bayesian learning approach to facilitate negotiation among participants in a multi-agent system called MASCOT designed for constructing claims negotiation. While it improved the negotiation results, it was conditioned to the fact that the agents could gain enough prior knowledge about their opponents. Buffett and Spencer (2007) examined the effectiveness of Bayesian learning for learning an opponent’s preferences during a bilateral multi-issue negotiation. Using a hypothetical negotiation scenario, they were able to determine the opponents‘ preferences in a few rounds of negotiation.
Jacobs and Kruschke (2011) argue that many aspects of human learning can be captured by the Bayesian approach. The ability of people to learn from limited data can be addressed in a Bayesian framework by strong constraints on the prior beliefs. Moreover the assumptions made in Bayesian models are often expressed as well-defined mathematical expressions which make them easy to examine, evaluate, and modify. Additionally, since the Bayesian models update probabilities for all possible values of the variables, they yield a distribution over all possible outcomes rather than a fixed point, which enables them to update several competing hypotheses.
This paper describes the incorporation of Bayesian learning in an ABM to simulate the negotiation process of multiple stakeholders regarding land development scenarios. This work is a continuation of a previous study conducted by the authors (Pooyandeh & Marceau, 2013) which aimed at building a spatial negotiation support system for stakeholders using an agent-based model in a web-based participatory environment. While the model adequately captured different aspects of the negotiation process, it lacked the important notion of learning among agents, which is an essential process in real-world stakeholders’ negotiation. In this study, our goal is to evaluate the impact of adding a learning component to the achievement of agreement among agents and to examine how it affects the negotiation behavior of the agents. A learning component is added into the negotiation process to better mimic human behavior and facilitate the result of the negotiation.
Section snippets
Methodology
In this section, the study area is introduced along with its significance in the context of land development. In Sections 2.2.1 The agents and their preferences, 2.2.2 The agents‘ evaluation functions, the essential components of the agent-based model are presented and the criteria used by the agents are described. The updated negotiation model and the learning of the agents are described in Section 2.2.3.
Results
Fig. 7 displays the results of the negotiation for different cases. Fig. 7a illustrates the values obtained for the LocationChange_No-learning case. As it can be seen, it takes 11 rounds for the agents to reach an agreement. The utilities of the Planner and Developer agent change more smoothly compared to the other agents. Moreover the Planner’s utility values are closer to those of the Developer agent compared to the other agents. The WaterConcerned agent’s utility values increase smoothly
Conclusion
This study was undertaken to examine the impact of incorporating a learning technique to improve the achievement of agreement in agent-based negotiation regarding land development in the Elbow River watershed in southern Alberta. The results indicate that the learning module enhances the negotiation and reduces the number of rounds required by the agents to achieve an agreement. They highlight the significance of learning among the parties by considering the opponents’ perspectives in the
Acknowledgements
This research was funded by a research Grant awarded by Tecterra to D. Marceau and by University of Calgary’s scholarships awarded to M. Pooyandeh. We are grateful to the stakeholders for their invaluable contribution to this project. We would also like to thank Dr. Scott Heckbert for his constructive comments and intellectual feedback.
References (66)
- et al.
Bargaining with incomplete information
Handbook of Game Theory with Economic Applications
(2002) - et al.
A complex systems approach to planning, optimization and decision making for energy networks
Energy Policy
(2008) - et al.
Simulation and validation of a reinforcement learning agent-based model for multi-stakeholder forest management
Computers, Environment and Urban Systems
(2010) - et al.
A Bayesian classifier for learning opponents’ preferences in multi-object automated negotiation
Electronic Commerce Research and Applications
(2007) - et al.
Predicting opponent’s moves in electronic negotiations using neural networks
Expert Systems with Applications
(2008) - et al.
Membership functions I: Comparing methods of measurement
International Journal of Approximate Reasoning
(1987) - et al.
Fuzzy clustering analysis for optimizing fuzzy membership functions
Fuzzy Sets and Systems
(1999) - et al.
A genetic agent-based negotiation system
Computer Networks
(2001) - et al.
Collaborative learning: Improving public deliberation in ecosystem-based management
Environmental Impact Assessment Review
(1996) - et al.
Bayesian learning in social networks
Games and Economic Behavior
(2003)
GIS implications for hydrologic modeling: Simulation of nonpoint pollution generated as a consequence of watershed development scenarios
Computers, Environment and Urban Systems
Exploring normative scenarios of land use development decisions with an agent-based simulation laboratory
Computers, Environment and Urban Systems
A spatial web/agent-based model to support stakeholders’ negotiation regarding land development
Journal of Environmental Management
Learning in multi-agent systems: A case study of construction claims negotiation
Advanced Engineering Informatics
Measurement of membership functions and their acquisition
Fuzzy Sets and Systems
Generating fuzzy membership functions: A monotonic neural network model
Fuzzy Sets and Systems
A linguistic cellular automata simulation approach for sustainable land development in a fast growing region
Computers, Environment and Urban Systems
Bayesian learning in negotiation
International Journal of Human–Computer Studies
GIS and remote sensing as tools for the simulation of urban land-use change
International Journal of Remote Sensing
Research Article. Modelling inside GIS: Part 1. Model structures, exploratory spatial data analysis and aggregation
International Journal of Geographical Information Systems
Judgment in managerial decision making
Automated negotiations: A survey of the state of the art
Wirtschaftsinformatik
Geographic automata systems: A new paradigm for integrating GIS and geographic simulation
Modeling-in-the-middle: Bridging the gap between agent-based modeling and multi-objective decision-making for land use change
International Journal of Geographical Information Science
Generalization in reinforcement learning: Safely approximating the value function
Advances in Neural Information Processing Systems
Learning other agents’ preferences in multi-agent negotiation using the Bayesian classifier
International Journal of Cooperative Information Systems
On the optimality of the simple Bayesian classifier under zero-one loss
Machine Learning
Cited by (17)
An intelligent web-based spatial group decision support system to investigate the role of the opponents’ modeling in urban land use planning
2022, Land Use PolicyCitation Excerpt :They used MAS along with fuzzy set theory to represent the knowledge of decision-makers. In a follow-up study, they studied the influence of using learning approaches to better support the negotiation process on land developments (Pooyandeh and Marceau, 2014). Given this brief review of approaches, the question may be raised whether providing information about the stakeholders is beneficial for group decision-making in a ULUP process.
An agent-based model for investigating the impact of distorted supply–demand information on China's resale housing market
2018, Journal of Computational ScienceSocially rational agents in spatial land use planning: A heuristic proposal based negotiation mechanism
2016, Computers, Environment and Urban SystemsCitation Excerpt :In an application, they examined the impact of incorporating a learning technique to improve the achievement of agreement in agent-based negotiation regarding land development in southern Alberta. The agents were equipped with Bayesian learning capability to predict the evaluation functions of other agents (Pooyandeh & Marceau, 2014). Kamps (2013) developed a MAS based simulation model that links the form of residential developments to actor's bargain and negotiation power.
ATABS: A technique for automatically training agent-based simulators
2016, Simulation Modelling Practice and TheoryCitation Excerpt :Moreover, some works apply learning techniques in ABSs, but are aimed at purposes different from obtaining the similarity between real and simulated results. For example, Pooyandeh and Marceau [13] incorporate a Bayesian learning algorithm into an ABS for simulating stakeholders’ negotiations in scenarios of land development. Specifically, the proposer agent uses this algorithm to learn their opponents’ preferences.
Adaptive Probabilistic Behavioural Learning System for the effective behavioural decision in cloud trading negotiation market
2016, Future Generation Computer SystemsCitation Excerpt :In the cloud based e-commerce negotiation framework, optimal management of broker negotiation strategy poses a series of challenges due to lack of behavioural prediction regarding the opponent’s information [17]. The automated negotiation strategy can be classified into two types such as negotiation strategy without learning ability and with learning ability [18]. Lots of research works are available in case of negotiation strategy without learning ability where the negotiation attributes like price, speed, and time-slot constantly changes over the time [19–21].
Automated negotiation in environmental resource management: Review and assessment
2015, Journal of Environmental ManagementCitation Excerpt :With the widespread use of the internet and the World Wide Web, it has received a lot of attention in domains such as supply chain management (Fink, 2006), political studies (Aragonès and Dellunde, 2009), and especially e-commerce (Ramchurn et al., 2007; Jazayeriy et al., 2011). However, its application in modeling stakeholders' negotiation over environmental resources is still in its infancy (Akhbari and Grigg, 2013; Okumura et al., 2013; Pooyandeh and Marceau, 2013, 2014). This is largely due to the characteristics of environmental issues, such as the amount of uncertainty involved, the high-stakes decisions, the diversity of perspectives, and the inexistence of optimum solutions.