Peng Zhang
ABSTRACT
Addressing quality-of-service (QoS) requirements for software applications in undersea acoustic networks is challenging. This is in part due to the strong dependency of the characteristics on the acoustic propagation environment and the lack of effective congestion control protocols for undersea acoustic networks. In this work we propose a deep reinforcement learning (DRL) based algorithm for allocating bandwidth resources across different traffic types. Bandwidth allocations are periodically updated based on traffic prioritization, QoS delay requirements, and queue occupancy levels. A DRL agent learns the optimal bandwidth allocation policy via a Soft Actor-Critic (SAC) algorithm. In order to test the effectiveness of our proposed algorithm, we developed a weighted fair queuing (WFQ) module in ns-3 to enable testing in a high fidelity simulation environment. The WFQ module enables the use of queueing disciplines together with an ns-3 library for underwater acoustic communications and networking. The interaction between the DRL agent and the simulated environment is enabled by the ns3-gym framework. Numerical tests illustrate the performance of the DRL-based policy and a use case for the new WFQ ns-3 module.
- Richelle Adams. 2013. Active Queue Management: A Survey. IEEE Communications Surveys Tutorials 15, 3 (2013), 1425–1476.Google Scholar
Cross Ref
- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv preprint arXiv:1606.01540(2016).Google Scholar
- Steve E. Deering and Robert M. Hinden. 2017. Internet Protocol, Version 6 (IPv6) Specification. RFC 8200.Google Scholar
- Pedro A. Forero, Peng Zhang, and Dusan Radosevic. 2021. Active Queue-Management Policies for Undersea Networking via Deep Reinforcement Learning. In Proc. of MTS/IEEE OCEANS Conference (San Diego, CA USA - Porto, Portugal).Google ScholarCross Ref
- Piotr Gawłowicz and Anatolij Zubow. 2019. ns-3 Meets OpenAI Gym: The Playground for Machine Learning in Networking Research. In Proc. of the 22nd ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM) (Miami Beach, FL USA).Google ScholarDigital Library
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proc. of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 1861–1870.Google Scholar
- John Heidemann, Wei Ye, Jack Wills, Affan A. Syed, and Yuan Li. 2006. Research Challenges and Applications for Underwater Sensor Networking. In Proc. of IEEE Wireless Communications and Networking Conference, 2006. WCNC 2006. (Las Vegas, NV USA), Vol. 1. 228–235.Google ScholarCross Ref
- Pasquale Imputato and Stefano Avallone. 2016. Design and Implementation of the Traffic Control Module in ns-3. In Proceedings of the Workshop on ns-3 (Seattle, WA, USA) (WNS3 ’16). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/2915371.2915382Google ScholarDigital Library
- Pasquale Imputato and Stefano Avallone. 2017. Traffic Differentiation and Multiqueue Networking in ns-3. In Proceedings of the Workshop on ns-3, Porto, Portugal, June 13 - 14, 2017. ACM, 79–86. https://doi.org/10.1145/3067665.3067677Google ScholarDigital Library
- Shengming Jiang. 2018. On Reliable Data Transfer in Underwater Acoustic Networks: A Survey From Networking Perspective. IEEE Communications Surveys Tutorials 20, 2 (2018), 1036–1055.Google ScholarCross Ref
- Robert Martin, Sanguthevar Rajasekaran,, and Zheng Peng. 2017. Aqua-Sim Next generation: An ns-3 Based Underwater Sensor Network Simulator. In Proc. of the International Conference on Underwater Networks & Systems (Halifax, Canada). 1–8.Google ScholarDigital Library
- Kathleen Nichols and Van Jacobson. 2012. Controlling Queue Delay: A Modern AQM is Just One Piece of the Solution to Bufferbloat.Queue 10, 5 (May 2012), 20–34.Google Scholar
- Kathleen Nichols, Van Jacobson, Andrew McGregor, and Jana Iyengar. 2018. Controlled Delay Active Queue Management. RFC 8289.Google Scholar
- Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8.Google Scholar
- Kadangode K. Ramakrishnan, Sally Floyd, and David L. Black. 2001. RFC3168: The Addition of Explicit Congestion Notification (ECN) to IP.Google Scholar
- George F. Riley and Thomas R. Henderson. 2010. The ns-3 Network Simulator. Springer Berlin Heidelberg, Berlin, Heidelberg, 15–34.Google Scholar
- Rayadurgam Srikant and Lei Ying. 2014. Communication Networks: An Optimization Control, and Stochastic Networks Perspective (1st ed.). Cambridge University Press.Google Scholar
Recommendations
Traffic Differentiation and Multiqueue Networking in ns-3
WNS3 '17: Proceedings of the 2017 Workshop on ns-3The Linux networking subsystem provides fundamental abstraction to send and receive packets or to perform other operations. At socket layer, the user can set the socket priority used by the networking stack to prioritise the packets. The kernel sends a ...
Implementation and Evaluation of Proportional Integral Controller Enhanced (PIE) Algorithm in ns-3
WNS3 '16: Proceedings of the 2016 Workshop on ns-3This paper proposes a new ns-3 model and presents the evaluation results for Proportional Integral controller Enhanced (PIE), a recently designed Active Queue Management (AQM) mechanism to address the problem of bufferbloat. The problem of bufferbloat ...
Design and Implementation of the Traffic Control Module in ns-3
WNS3 '16: Proceedings of the 2016 Workshop on ns-3The Linux networking subsystem relies on the Traffic Control infrastructure to process both the incoming and the outgoing packets. One of the most important components of the Traffic Control is the queueing discipline, whose role is to store packets ...
Comments