Haoyu Geng
ABSTRACT
The NP-hard combinatorial Virtual Network Embedding (VNE) Problem refers to finding the node and edge mapping between a virtual net (request) and the physical net (resource). Learning-based methods are recently devised beyond traditional heuristic solvers. However, the efficiency and scalability hinder its applicability as reinforcement learning (RL) is often adopted in an auto-regressive node-by-node mapping manner to handle complex mapping constraints, for each coming request for mapping. Moreover, existing learning-based works often independently consider each online request, limiting the long-term online service performance. In this paper, we present a synergistic Global-And-Local learning approach for the VNE problem (GAL-VNE). At the global level across requests, RL is employed to capture the cross-request relation for better global resource accommodation to improve overall performance. At the local level within each request, we aim to replace the sequential decision-making procedure which relies much on the network size, with a more efficient one-shot solution generation scheme. The main challenge for such a one-shot model is how to encode the constraints under an end-to-end learning and inference paradigm. Accordingly, within the "rank-then-search" paradigm, we propose to first pretrain a graph neural network (GNN)-based node ranker with imitation supervision from an off-the-shelf solver (moderately expensive yet high quality), which is meanwhile regularized by a neighboring smooth prior. Then RL is used to finetune the GNN ranker whose supervision directly refers to the final (undifferentiable) business objectives concerning revenue and cost, etc. Experiments on benchmarks show that our method outperforms classic and learning-based methods in both efficacy and efficiency.
Supplemental Material
- Thomas Anderson, Larry Peterson, Scott Shenker, and Jonathan Turner. 2005. Overcoming the Internet impasse through virtualization. Computer, Vol. 38, 4 (2005), 34--41.Google Scholar
Digital Library
- Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. 2016. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940 (2016).Google Scholar
- Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. 2021. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research, Vol. 290, 2 (2021), 405--421.Google ScholarCross Ref
- Garrett Birkhoff. 1946. Tres observaciones sobre el algebra lineal. Univ. Nac. Tucuman, Ser. A, Vol. 5 (1946), 147--154.Google Scholar
- Andreas Blenk, Patrick Kalmbach, Johannes Zerwas, Michael Jarschel, Stefan Schmid, and Wolfgang Kellerer. 2018. NeuroViNE: A neural preprocessor for your virtual network embedding algorithm. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 405--413.Google ScholarDigital Library
- Juan Felipe Botero, Xavier Hesselbach, Michael Duelli, Daniel Schlosser, Andreas Fischer, and Hermann De Meer. 2012. Energy efficient virtual network embedding. IEEE Communications Letters, Vol. 16, 5 (2012), 756--759.Google ScholarCross Ref
- Xavier Bresson and Thomas Laurent. 2021. The transformer network for the traveling salesman problem. arXiv preprint arXiv:2103.03012 (2021).Google Scholar
- Chen Cai and Yusu Wang. 2020. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 (2020).Google Scholar
- Haotong Cao, Shengchen Wu, Yue Hu, Yun Liu, and Longxiang Yang. 2019. A survey of embedding algorithm for virtual network embedding. China Communications, Vol. 16, 12 (2019), 1--33.Google ScholarCross Ref
- R. Chen, X. Lv, Y. Li, J. Ye, J. Hao, and J. Yan. 2022. The Policy-gradient Placement and Generative Routing Neural Networks for Chip Design. In Neural Information Processing Systems (NeurIPS).Google Scholar
- Xiang Cheng, Sen Su, Zhongbao Zhang, Hanchi Wang, Fangchun Yang, Yan Luo, and Jie Wang. 2011. Virtual network embedding through topology-aware node ranking. ACM SIGCOMM Computer Communication Review, Vol. 41, 2 (2011), 38--47.Google ScholarDigital Library
- Mosharaf Chowdhury, Muntasir Raihan Rahman, and Raouf Boutaba. 2011. Vineyard: Virtual network embedding algorithms with coordinated node and link mapping. IEEE/ACM Transactions on networking, Vol. 20, 1 (2011), 206--219.Google Scholar
- Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. 2019. Differentiable ranking and sorting using optimal transport. Neural Information Processing Systems (NeurIPS), Vol. 32 (2019).Google Scholar
- Lu Duan, Haoyuan Hu, Yu Qian, Yu Gong, Xiaodong Zhang, Yinghui Xu, and Jiangwen Wei. 2018. A multi-task selected learning approach for solving 3d flexible bin packing problem. arXiv preprint arXiv:1804.06896 (2018).Google Scholar
- Ilhem Fajjari, Nadjib Aitsaadi, Guy Pujolle, and Hubert Zimmermann. 2011. VNE-AC: Virtual network embedding algorithm based on ant colony metaheuristic. In ICC.Google Scholar
- Nick Feamster, Lixin Gao, and Jennifer Rexford. 2007. How to lease the Internet in your spare time. ACM SIGCOMM Computer Communication Review, Vol. 37, 1 (2007), 61--64.Google ScholarDigital Library
- Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.Google Scholar
- Long Gong, Yonggang Wen, Zuqing Zhu, and Tony Lee. 2014. Toward profit-seeking virtual network embedding algorithm via global resource capacity. In INFOCOM. 1--9.Google Scholar
- Farzad Habibi, Mahdi Dolati, Ahmad Khonsari, and Majid Ghaderi. 2020. Accelerating virtual network embedding with graph neural networks. In 2020 16th International Conference on Network and Service Management (CNSM). IEEE, 1--9.Google ScholarCross Ref
- Soroush Haeri and Ljiljana Trajković. 2016. VNE-Sim: a virtual network embedding simulator. In Proceedings of the 9th EAI International Conference on Simulation Tools and Techniques. 112--117.Google Scholar
- Soroush Haeri and Ljiljana Trajković. 2017. Virtual network embedding via Monte Carlo tree search. IEEE transactions on cybernetics, Vol. 48, 2 (2017), 510--521.Google Scholar
- Aric Hagberg, Pieter Swart, and Daniel S Chult. 2008. Exploring network structure, dynamics, and function using NetworkX. Technical Report. Los Alamos National Lab.(LANL), Los Alamos, NM (United States).Google Scholar
- William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Neural Information Processing Systems (NIPS). 1025--1035.Google Scholar
- Chuan He, Cong Wang, Yi-Xin Zhong, and Rui-Fan Li. 2008. A survey on learning to rank. In 2008 International Conference on Machine Learning and Cybernetics, Vol. 3. Ieee, 1734--1739.Google Scholar
- Ines Houidi, Wajdi Louati, Walid Ben Ameur, and Djamal Zeghlache. 2011. Virtual network provisioning across multiple substrate networks. Computer Networks, Vol. 55, 4 (2011), 1011--1023.Google ScholarDigital Library
- Haoyuan Hu, Xiaodong Zhang, Xiaowei Yan, Longfei Wang, and Yinghui Xu. 2017. Solving a new 3d bin packing problem with deep reinforcement learning method. arXiv preprint arXiv:1708.05930 (2017).Google Scholar
- Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparametrization with Gumble-Softmax. In International Conference on Learning Representations (ICLR 2017). OpenReview. net.Google Scholar
- Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. 2017. Learning combinatorial optimization algorithms over graphs. In Neural Information Processing Systems (NeurIPS). 6351--6361.Google Scholar
- Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).Google Scholar
- Wouter Kool, Herke van Hoof, and Max Welling. 2019. Attention, Learn to Solve Routing Problems!. In International Conference on Learning Representations (ICLR).Google Scholar
- Meng Li and MeiLian Lu. 2021. A Virtual Network Embedding Algorithm Based On Double-Layer Reinforcement Learning. Comput. J., Vol. 64, 6 (2021), 973--989.Google ScholarCross Ref
- Yang Li, Xinyan Chen, Wenxuan Guo, Xijun Li, Wanqian Luo, Junhua Huang, Hui-Ling Zhen, Mingxuan Yuan, and Junchi Yan. 2023. HardSATGEN: Understanding the Difficulty of Hard SAT Formula Generation and A Strong Structure-Hardness-Aware Baseline. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).Google Scholar
- Jens Lischka and Holger Karl. 2009. A virtual network mapping algorithm based on subgraph isomorphism detection. In Proceedings of the 1st ACM workshop on Virtualized infrastructure systems and architectures. 81--88.Google ScholarDigital Library
- C. Liu, Z. Jiang, R. Wang, L. Huang, P. Lu, and J. Yan. 2023. Revocable Deep Reinforcement Learning with Affinity Regularization for Outlier-Robust Graph Matching. In International Conference on Learning Representations (ICLR).Google Scholar
- Han Lu, Zenan Li, Runzhong Wang, Qibing Ren, Xijun Li, Mingxuan Yuan, Jia Zeng, Xiaokang Yang, and Junchi Yan. 2023. ROCO: A General Framework for Evaluating Robustness of Combinatorial Optimization Solvers on Graphs. In International Conference on Learning Representations (ICLR).Google Scholar
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Neural Information Processing Systems (NeurIPS), Vol. 32 (2019).Google Scholar
- Laurent Perron and Vincent Furnon. 2022-11-25. OR-Tools. Google. https://developers.google.com/optimization/Google Scholar
- Steffen Rendle. 2010. Factorization machines. In Proceedings of the International Conference on Data Mining (ICDM). IEEE, 995--1000.Google ScholarDigital Library
- Michal Rolínek, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vít Musil, and Georg Martius. 2020. Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers. In ECCV. Springer, 407--424.Google Scholar
- Matthias Rost and Stefan Schmid. 2020. On the hardness and inapproximability of virtual network embeddings. IEEE/ACM Transactions on Networking, Vol. 28, 2 (2020), 791--803.Google ScholarDigital Library
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google Scholar
- Richard Sinkhorn. 1964. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, Vol. 35, 2 (1964), 876--879.Google Scholar
- Sen Su, Zhongbao Zhang, Alex X Liu, Xiang Cheng, Yiwen Wang, and Xinchao Zhao. 2014. Energy-aware virtual network embedding. IEEE/ACM Transactions on Networking, Vol. 22, 5 (2014), 1607--1620.Google ScholarDigital Library
- Jonathan S Turner and David E Taylor. 2005. Diversifying the internet. In GLOBECOM'05. IEEE Global Telecommunications Conference, 2005., Vol. 2. IEEE, 6-pp.Google ScholarCross Ref
- Ihsan Ullah, Hyun-Kyo Lim, and Youn-Hee Han. 2021. Ego Network-based Virtual Network Embedding Scheme for Revenue Maximization. In ICAIIC. 155--160.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems (NIPS).Google Scholar
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Neural Information Processing Systems (NIPS), Vol. 28 (2015).Google Scholar
- R. Wang, L. Shen, Y. Chen, X. Yang, D. Tao, and J. Yan. 2023. Towards One-shot Neural Combinatorial Optimization Solvers: Theoretical and Empirical Notes on the Cardinality-Constrained Case. In International Conference on Learning Representations (ICLR).Google Scholar
- Runzhong Wang, Junchi Yan, and Xiaokang Yang. 2020. Combinatorial learning of robust deep graph matching: an embedding based approach. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2020).Google ScholarDigital Library
- Runzhong Wang, Junchi Yan, and Xiaokang Yang. 2021b. Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matching. IEEE transactions on pattern analysis and machine intelligence (TPAMI) (2021).Google Scholar
- Tianfu Wang, Qilin Fan, Xiuhua Li, Xu Zhang, Qingyu Xiong, Shu Fu, and Min Gao. 2021a. Drl-sfcp: Adaptive service function chains placement with deep reinforcement learning. In ICC 2021-IEEE International Conference on Communications. IEEE, 1--6.Google ScholarCross Ref
- Yansheng Wang, Yongxin Tong, Cheng Long, Pan Xu, Ke Xu, and Weifeng Lv. 2019. Adaptive dynamic bipartite graph matching: A reinforcement learning approach. In 2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 1478--1489.Google ScholarCross Ref
- Bernard M Waxman. 1988. Routing of multipoint connections. IEEE journal on selected areas in communications, Vol. 6, 9 (1988), 1617--1622.Google ScholarDigital Library
- J. Yan, S. Yang, and E. Hancock. 2020b. Learning Graph Matching and Related Combinatorial Optimization Problems. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Zhongxia Yan, Jingguo Ge, Yulei Wu, Liangxiong Li, and Tong Li. 2020a. Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks. IEEE Journal on Selected Areas in Communications, Vol. 38, 6 (2020), 1040--1057.Google ScholarCross Ref
- Haipeng Yao, Xu Chen, Maozhen Li, Peiying Zhang, and Luyao Wang. 2018. A novel reinforcement learning algorithm for virtual network embedding. Neurocomputing, Vol. 284 (2018), 1--9.Google ScholarCross Ref
- Haipeng Yao, Sihan Ma, Jingjing Wang, Peiying Zhang, Chunxiao Jiang, and Song Guo. 2020. A continuous-decision virtual network embedding scheme relying on reinforcement learning. IEEE Transactions on Network and Service Management, Vol. 17, 2 (2020), 864--875.Google ScholarDigital Library
- Minlan Yu, Yung Yi, Jennifer Rexford, and Mung Chiang. 2008. Rethinking virtual network embedding: substrate support for path splitting and migration. ACM SIGCOMM Computer Communication Review, Vol. 38, 2 (2008), 17--29.Google ScholarDigital Library
- Jiayi Zhang, Chang Liu, Xijun Li, Hui-Ling Zhen, Mingxuan Yuan, Yawen Li, and Junchi Yan. 2023. A survey for solving mixed integer programming via machine learning. Neurocomputing (2023).Google Scholar
- Peiying Zhang, Chao Wang, Chunxiao Jiang, Neeraj Kumar, and Qinghua Lu. 2021. Resource management and security scheme of ICPSs and IoT based on VNE algorithm. IEEE Internet of Things Journal (2021).Google Scholar
- Peiying Zhang, Chao Wang, Neeraj Kumar, Weishan Zhang, and Lei Liu. 2022. Dynamic virtual network embedding algorithm based on graph convolution neural network and reinforcement learning. IEEE Internet of Things Journal (2022).Google ScholarCross Ref
- Peiying Zhang, Haipeng Yao, and Yunjie Liu. 2017. Virtual network embedding based on computing, network, and storage resource constraints. IEEE Internet of Things Journal, Vol. 5, 5 (2017), 3298--3304.Google ScholarCross Ref
- Sheng Zhang, Zhuzhong Qian, Song Guo, and Sanglu Lu. 2011. FELL: A Flexible Virtual Network Embedding Algorithm with Guaranteed Load Balancing. In 2011 IEEE International Conference on Communications (ICC). 1--5. https://doi.org/10.1109/icc.2011.5962960Google Scholar
- Zhongbao Zhang, Xiang Cheng, Sen Su, Yiwen Wang, Kai Shuang, and Yan Luo. 2013. A unified enhanced particle swarm optimization-based virtual network embedding algorithm. International Journal of Communication Systems, Vol. 26, 8 (2013), 1054--1073.Google ScholarCross Ref
- Chenggui Zhao and Behrooz Parhami. 2019. Virtual network embedding through graph eigenspace alignment. IEEE Transactions on Network and Service Management, Vol. 16, 2 (2019), 632--646.Google ScholarCross Ref
- Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 1059--1068.Google ScholarDigital Library
Index Terms
- GAL-VNE: Solving the VNE Problem with Global Reinforcement Learning and Local One-Shot Neural Prediction
Recommendations
VNE-TD: A virtual network embedding algorithm based on temporal-difference learning
AbstractRecently, network virtualization is considered as a promising solution for the future Internet which can help to overcome the resistance of the current Internet to fundamental changes. The problem of embedding Virtual Networks (VN) in ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent SystemsRecent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Global Reinforcement Learning in Neural Networks
In this letter, we have found a more general formulation of the REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility (REINFORCE) learning principle first suggested by Williams. The new formulation has enabled ...
Comments