Abstract
Learning in automated negotiations, while successful for many tasks in recent years, is still hard when coping with different types of opponents with unknown strategies. It is critically essential to learn about the opponents from observations and then find the best response in order to achieve efficient agreements. In this paper, we propose a novel framework named Deep BPR+ (DBPR+) negotiating agent framework, which includes two key components: a learning module to learn a new coping policy when encountering an opponent using a previously unseen strategy, and a policy reuse mechanism to efficiently detect the strategy of an opponent and select the optimal response policy from the policy library. The performance of the proposed DBPR+ agent is evaluated against winning agents of ANAC competitions under varied negotiation scenarios. The experimental results show that DBPR+ agent outperforms existing state-of-the-art agents, and is able to make efficient detection and optimal response against unknown opponents.
This work is supported by National Natural Science Foundation of China (Grant Nos.: 61602391, 62106172).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Ponpoko (2017 winner), Caduceus (2016 winner), ParsCat (2016 \( 2^{nd} \) position), Atlas3 (2015 winner), ParsAgent (2015 \( 2^{nd} \) position), The Fawkes (2013 winner), CUHKAgent (2012 winner) and HardHeaded (2011 winner).
- 2.
Due to the space limitation, we only present the statistics of baseline agent in this control experiment. Mean utility, average rounds and average agreement achievement rate are \(0.4573\pm 0.0040\), \(49.54\pm 0.07\) and \(0.57\pm 0.00\) respectively.
- 3.
We also conducted other configures and found similar results, so we only report this evaluation.
References
Aydogan, R.: Anac 2016 - automated negotiating agents competition 2016. Website (2016). http://web.tuat.ac.jp/~katfuji/ANAC2016/
Aydoğan, R., Fujita, K., Baarslag, T., Jonker, C.M., Ito, T.: ANAC 2017: repeated multilateral negotiation league. In: Ito, T., Zhang, M., Aydoğan, R. (eds.) ACAN 2018. SCI, vol. 905, pp. 101–115. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-5869-6_7
Baarslag, T., Aydogan, R., Hindriks, K., Fujita, K., Ito, T., Jonker, C.: The automated negotiating agents competition, 2010–2015. AI Mag. 36, 115–118 (2015). https://doi.org/10.1609/aimag.v36i4.2609
Baarslag, T., Hendrikx, M.J.C., Hindriks, K.V., Jonker, C.M.: Learning about the opponent in automated bilateral negotiation: a comprehensive survey of opponent modeling techniques. Auton. Agents Multi-Agent Syst. 30(5), 849–898 (2015). https://doi.org/10.1007/s10458-015-9309-1
Baarslag, T., Hindriks, K., Jonker, C.: Effective acceptance conditions in real-time automated negotiation. Decis. Support Syst. 60, 68–77 (2014)
Baarslag, T., Hindriks, K., Hendrikx, M., Dirkzwager, A., Jonker, C.: Decoupling negotiating agents to explore the space of negotiation strategies. In: Marsa-Maestre, I., Lopez-Carmona, M.A., Ito, T., Zhang, M., Bai, Q., Fujita, K. (eds.) Novel Insights in Agent-based Complex Automated Negotiation. SCI, vol. 535, pp. 61–83. Springer, Tokyo (2014). https://doi.org/10.1007/978-4-431-54758-7_4
Bagga, P., Paoletti, N., Alrayes, B., Stathis, K.: A deep reinforcement learning approach to concurrent bilateral negotiation. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 297–303 (2020)
Bakker, J., Hammond, A., Bloembergen, D., Baarslag, T.: RLBOA: a modular reinforcement learning framework for autonomous negotiating agents. In: Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pp. 260–268, May 2019
Chang, H.C.H.: Multi-issue bargaining with deep reinforcement learning. arXiv preprint arXiv:2002.07788 (2020)
Chen, S., Ammar, H.B., Tuyls, K., Weiss, G.: Using conditional restricted boltzmann machine for highly competitive negotiation tasks. In: Proceedings of the 23th International Joint Conference on Artificial Intelligence, pp. 69–75. AAAI Press (2013)
Chen, S., Weiss, G.: An intelligent agent for bilateral negotiation with unknown opponents in continuous-time domains. ACM Trans. Auton. Adapt. Syst. 9(3), 16:1-16:24 (2014). https://doi.org/10.1145/2629577
Choi, S.P., Liu, J., Chan, S.P.: A genetic agent-based negotiation system. Comput. Netw. 37(2), 195–204 (2001)
de Jonge, D., Sierra, C.: GANGSTER: an automated negotiator applying genetic algorithms. In: Fukuta, N., Ito, T., Zhang, M., Fujita, K., Robu, V. (eds.) Recent Advances in Agent-based Complex Automated Negotiation. SCI, vol. 638, pp. 225–234. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30307-9_14
Fujita, K., et al.: Modern Approaches to Agent-Based Complex Automated Negotiation. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51563-2
van Galen Last, N.: Agent smith: opponent model estimation in bilateral multi-issue negotiation. In: Ito, T., Zhang, M., Robu, V., Fatima, S., Matsuo, T. (eds.) New Trends in Agent-based Complex Automated Negotiations. Studies in Computational Intelligence, vol. 383, pp. 167–174. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24696-8_12
Gerding, E.H., van Bragt, D.D.B., La Poutré, J.A.: Scientific approaches and techniques for negotiation: a game theoretic and artificial intelligence perspective. Centrum voor Wiskunde en Informatica (2000)
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning, pp. 1861–1870. PMLR (2018)
Hernandez-Leal, P., Kaisers, M.: Learning against sequential opponents in repeated stochastic games. In: The 3rd Multi-disciplinary Conference on Reinforcement Learning and Decision Making, Ann Arbor, vol. 25 (2017)
Hernandez-Leal, P., Rosman, B., Taylor, M.E., Sucar, L.E., Munoz de Cote, E.: A bayesian approach for learning and tracking switching, non-stationary opponents. In: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pp. 1315–1316 (2016)
Hindriks, K., Tykhonov, D.: Opponent modelling in automated multi-issue negotiation using bayesian learning. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 1, pp. 331–338 (2008)
Ito, T., Zhang, M., Robu, V., Fatima, S., Matsuo, T.: New Trends in Agent-based Complex Automated Negotiations, vol. 383. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24696-8
Liang, Y.Q., Yuan, Y.: Co-evolutionary stability in the alternating-offer negotiation. In: 2008 IEEE Conference on Cybernetics and Intelligent Systems, pp. 1176–1180. IEEE (2008)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016)
Marsa-Maestre, I., Lopez-Carmona, M.A., Ito, T., Zhang, M., Bai, Q., Fujita, K. (eds.): Novel Insights in Agent-based Complex Automated Negotiation. SCI, vol. 535. Springer, Tokyo (2014). https://doi.org/10.1007/978-4-431-54758-7
Matos, N., Sierra, C., Jennings, N.R.: Determining successful negotiation strategies: an evolutionary approach. In: Proceedings International Conference on Multi Agent Systems (Cat. No. 98EX160), pp. 182–189. IEEE (1998)
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Robu, V., Somefun, D., La Poutré, J.A.: Modeling complex multi-issue negotiations using utility graphs. In: Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 280–287 (2005)
Rosman, B., Hawasly, M., Ramamoorthy, S.: Bayesian policy reuse. Mach. Learn. 104(1), 99–127 (2016). https://doi.org/10.1007/s10994-016-5547-y
Sengupta, A., Mohammad, Y., Nakadai, S.: An autonomous negotiating agent framework with reinforcement learning based strategies and adaptive strategy switching mechanism. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’21, pp. 1163–172 (2021)
Williams, C.R., Robu, V., Gerding, E.H., Jennings, N.R.: Negotiating concurrently with unkown opponents in complex, real-time domains. In: ECAI’12, pp. 834–839 (2012)
Zeng, D., Sycara, K.: Bayesian learning in negotiation. Int. J. Hum.-Comput. Stud. 48(1), 125–141 (1998)
Zheng, Y., Meng, Z., Hao, J., Zhang, Z., Yang, T., Fan, C.: A deep bayesian policy reuse approach against non-stationary agents. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 962–972 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, L., Chen, S., Gao, X., Zheng, Y., Hao, J. (2021). Detecting and Learning Against Unknown Opponents for Automated Negotiations. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds) PRICAI 2021: Trends in Artificial Intelligence. PRICAI 2021. Lecture Notes in Computer Science(), vol 13033. Springer, Cham. https://doi.org/10.1007/978-3-030-89370-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-89370-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89369-9
Online ISBN: 978-3-030-89370-5
eBook Packages: Computer ScienceComputer Science (R0)