Skip to main content
SpringerLink
Log in

A source-driven reinforcement learning-based Data reply strategy to reduce communication overhead in Named Data Networks (NDN)

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The Two-Armed Bernoulli Bandit (TABB) problem is an orthodox optimization dilemma in reinforcement learning discipline where a decision-maker or agent is repeatedly faced with a choice of two actions (options). Every time the agent selects an action, it receives a corresponding payoff from an unknown distribution. Thus, the agent must trade-off between exploration of new better action and exploitation of current best action. Content retrieval in Named Data Networks (NDN) commences with a consumer requesting the desired content by sending an Interest that hits multiple content sources over different paths. As the corresponding Interest arrives, the content sources respond by replying the matching Data to the requester. In this work, Data replying problem in NDN is considered a TABB problem, denoted as DTABB. Since numerous sources are available, a content source in DTABB can choose between responding with entire content and partial content once the corresponding Interest strikes. The best source is trained to answer with complete data, while other (sub-optimal) sources learn to react with partial (or payload-free) data. The proposed strategy is formulated from a source’s viewpoint, which uses four prominent reinforcement learning algorithms: greedy, ε-greedy, Upper Confidence Bound (UCB1), and Gradient Bandit to select the optimal action. Eventually, the network picture converges to a point where a single source is exploited for whole data while others send only partial data. Thus, DTABB can substantially reduce the transmission overhead and enjoy a better user experience in terms of delay. DTABB is implemented in ndnSIM, which reveals that the proposed solution can reduce the communication overhead by up to 40% compared to the default strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

Available on-request or on-demand.

References

  1. Zhang, L., et al.: Named data networking. Comput. Commun. Rev. 44(3), 66–73 (2014). https://doi.org/10.1145/2656877.2656887

    Article  Google Scholar 

  2. Cheriton, D.R., Gritter, M.: TRIAD: a new next-generation Internet architecture. http://www-dsg.stanford.edu/triad/. January 2000, pp. 1–20. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.33.5878. Accessed 27 Mar 2020

  3. Koponen, T., et al.: A data-oriented (and beyond) network architecture. Comput. Commun. Rev. 37(4), 181–192 (2007). https://doi.org/10.1145/1282427.1282402

    Article  Google Scholar 

  4. Jacobson, V., Smetters, D.K., Thornton, J.D., Plass, M.F., Briggs, N.H., Braynard, R.L.: Networking named content. In: CoNEXT’09—Proceedings of the 2009 ACM Conference on Emerging Networking Experiments and Technologies, pp. 1–12 (2009). https://doi.org/10.1145/1658939.1658941

  5. Zhang, L.: Named Data Networking (NDN) project—PARC. PARC TR-2010-3, October 2010. https://www.parc.com/technical-publications/named-data-networking-ndn-project. Accessed 27 Mar 2020

  6. Lv, J., Tan, X., Jin, Y., Zhu, J.: DRL-based forwarding strategy in named data networking. In: Chinese Control Conference (CCC), October 2018, vol. 2018–July, pp. 6493–6498, https://doi.org/10.23919/ChiCC.2018.8483989.

  7. Granmo, O.C.: Solving two-armed Bernoulli bandit problems using a Bayesian learning automaton. Int. J. Intell. Comput. Cybern. 3(2), 207–234 (2010). https://doi.org/10.1108/17563781011049179

    Article  MathSciNet  MATH  Google Scholar 

  8. Kelley, T.A.: A note on the Bernoulli two-armed bandit problem. Ann. Stat. 2(5), 1056–1062 (1974). https://doi.org/10.1214/aos/1176342827

    Article  MathSciNet  MATH  Google Scholar 

  9. Cover, T.M.: A note on the two-armed bandit problem with finite memory. Inf. Control 12(5), 371–377 (1968). https://doi.org/10.1016/S0019-9958(68)90382-3

    Article  MathSciNet  MATH  Google Scholar 

  10. Iqbal, S.M.A.: Asaduzzaman: Adaptive forwarding strategies to reduce redundant interests and data in named data networks. J. Netw. Comput. Appl. 106, 33–47 (2018). https://doi.org/10.1016/j.jnca.2018.01.013

    Article  Google Scholar 

  11. Mastorakis, S., Afanasyev, A., Moiseenko, I., Zhang, L.: ndnSIM 2: an updated NDN simulator for NS-3 (2016). https://named-data.net/wp-content/uploads/2016/11/ndn-0028-2-ndnsim-v2.pdf

  12. Wang, L., Bayhan, S., Ott, J., Kangasharju, J., Sathiaseelan, A., Crowcroft, J.: Pro-diluvian: understanding scoped-flooding for content discovery in information-centric networking. In: ICN 2015—Proceedings of the 2nd International Conference on Information-Centric Networking, September 2015, pp. 9–18. https://doi.org/10.1145/2810156.2810162.

  13. Badov, M., Seetharam, A., Kurose, J., Firoiu, V., Nanda, S.: Congestion-aware caching and search in information-centric networks. In: ICN 2014—Proceedings of the 1st International Conference on Information-Centric Networking, September 2014, pp. 37–46. https://doi.org/10.1145/2660129.2660145.

  14. Rossini, G., Rossi, D.: Coupling caching and forwarding. In: Proceedings of the 1st International Conference on Information-Centric Networking—INC ’14, pp. 127–136 (2014). https://doi.org/10.1145/2660129.2660153

  15. Ascigil, O., Sourlas, V., Psaras, I., Pavlou, G.: Opportunistic off-path content discovery in information-centric networks. In: IEEE Workshop on Local and Metropolitan Area Networks, August 2016, vol. 2016. https://doi.org/10.1109/LANMAN.2016.7548860

  16. Amadeo, M., Molinaro, A., Ruggeri, G.: E-CHANET: routing, forwarding and transport in information-centric multihop wireless networks. Comput. Commun. 36(7), 792–803 (2013). https://doi.org/10.1016/j.comcom.2013.01.006

    Article  Google Scholar 

  17. Sutton, R.S., Barto, A.G.: Reinforcement learning, 2nd edn. The MIT Press, London (2015)

    MATH  Google Scholar 

  18. Morales, E.F., Zaragoza, J.H.: An introduction to reinforcement learning. In: Decision Theory Models for Applications in Artificial Intelligence: Concepts and Solutions, pp. 63–80. IGI Global, Hershey (2011). https://doi.org/10.4018/978-1-60960-165-2.ch004.

  19. Avrachenkov, K., Jacko, P.: CCN interest forwarding strategy as Multi-Armed Bandit model with delays. In: NetGCoop 2012—6th International Conference on Network Games, Control and Optimization, pp. 38–43 (2012). https://ieeexplore.ieee.org/document/6486116. Accessed 18 Aug 2020

  20. Carofiglio, G., Gallo, M., Muscariello, L.: Optimal multipath congestion control and request forwarding in information-centric networks: protocol design and experimentation. Comput. Netw. 110, 104–117 (2016). https://doi.org/10.1016/j.comnet.2016.09.012

    Article  Google Scholar 

  21. Udugama, A., Zhang, X., Kuladinithi, K., Goerg, C.: An on-demand multi-path interest forwarding strategy for content retrievals in CCN. In: IEEE Network Operations and Management Symposium (NOMS) (2014). https://doi.org/10.1109/NOMS.2014.6838389

  22. Khan, A.Z., Baqai, S., Dogar, F.R.: QoS aware path selection in content centric networks. In: IEEE International Conference on Communications, pp. 2645–2649 (2012). https://doi.org/10.1109/ICC.2012.6363829.

  23. Qian, H., Ravindran, R., Wang, G.Q., Medhi, D.: Probability-based adaptive forwarding strategy in named data networking. In: Proceedings of the 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), pp. 1094–1101 (2013). https://www.researchgate.net/publication/261399057_Probability-based_adaptive_forwarding_strategy_in_named_data_networking. Accessed 22 Aug 2020.

  24. Yi, C., Afanasyev, A., Moiseenko, I., Wang, L., Zhang, B., Zhang, L.: A case for stateful forwarding plane. Comput. Commun. 36(7), 779–791 (2013). https://doi.org/10.1016/j.comcom.2013.01.005

    Article  Google Scholar 

  25. Zhang, Y., Xu, K., Bai, B., Lei, K.: IFS-RL: an intelligent forwarding strategy based on reinforcement learning in named-data networking. In: NetAI 2018—Proceedings of the 2018 Workshop on Network Meets AI and ML, Part of SIGCOMM 2018, pp. 54–59 (2018). https://doi.org/10.1145/3229543.3229547

  26. Kerrouche, A., Senouci, M.R., Mellouk, A.: QoS-FS: a new forwarding strategy with QoS for routing in Named Data Networking. In: IEEE International Conference on Communications (ICC) 2016. https://doi.org/10.1109/ICC.2016.7511378

  27. Akinwande, O.: A reinforcement learning approach to adaptive forwarding in named data networking. In: 32nd International Symposium, ISCIS 2018, held at the 24th IFIP a Reinforcement Learning Approach to Adaptive Forwarding in Named Data Networking, November 2019 (2018). https://doi.org/10.1007/978-3-030-00840-6

  28. Lei, K., Wang, J., Yuan, J.: An entropy-based probabilistic forwarding strategy in named data networking. In: IEEE International Conference on Communications, September 2015, vol. 2015, pp. 5665–5671. https://doi.org/10.1109/ICC.2015.7249225

  29. Fu, B., Qian, L., Zhu, Y., Wang, L.: Reinforcement learning-based algorithm for efficient and adaptive forwarding in named data networking. In: 2017 IEEE/CIC International Conference on Communications in China, ICCC 2017, April 2018, vol. 2018–January, pp. 1–6. https://doi.org/10.1109/ICCChina.2017.8330354

  30. Bastos, I.V., Moraes, I.M.: A diversity-based search-and-routing approach for named-data networking. Comput. Netw. 157, 11–23 (2019). https://doi.org/10.1016/j.comnet.2019.04.003

    Article  Google Scholar 

  31. Zhang, M., Wang, X., Liu, T., Zhu, J., Wu, Q.: AFSndn: a novel adaptive forwarding strategy in named data networking based on Q-learning. Peer-to-Peer Netw. Appl. 13(4), 1176–1184 (2020). https://doi.org/10.1007/s12083-019-00845-w

    Article  Google Scholar 

  32. Akinwande, O.: Interest forwarding in named data networking using reinforcement learning. Sensors (Switzerland) (2018). https://doi.org/10.3390/s18103354

    Article  Google Scholar 

  33. Chiocchetti, R., Perino, D., Carofiglio, G., Rossi, D., Rossini, G.: INFORM: a dynamic interest forwarding mechanism for information centric networking. In: ICN 2013—Proceedings of the 3rd 2013 ACM SIGCOMM Workshop on Information-Centric Networking, pp. 9–14 (2013). https://doi.org/10.1145/2491224.2491227

  34. Yao, J., Yin, B., Tan, X.: A SMDP-based forwarding scheme in named data networking. Neurocomputing 306, 213–225 (2018). https://doi.org/10.1016/j.neucom.2018.03.057

    Article  Google Scholar 

  35. Yao, J., Yin, B., Lu, X.: A novel joint adaptive forwarding and resource allocation strategy for named data networking based on SMDP. In: A novel joint adaptive forwarding and resource allocation strategy for named data networking based on SMDP (2016)

  36. Bastos, I.V., Moraes, I.M.: A forwarding strategy based on reinforcement learning for content-centric networking. In: International Conference on the Network of the Future (NOF) (2017). https://doi.org/10.1109/NOF.2016.7810121

  37. Lan, D., Tan, X., Lv, L., Jin, Y., Yang, J.: A deep reinforcement learning based congestion control mechanism for NDN. In: ICC 2019—2019 IEEE International Conference on Communications, pp. 1–7 (2019)

  38. Qiu, S., Tan, X., Zhu, J.: Dynamic adaptive streaming control based on deep reinforcement learning in named data networking. In: 2018 37th Chinese Control Conference, pp. 9478–9482 (2018)

  39. Liu, T., Zhang, M., Zhu, J., Zheng, R., Liu, R., Wu, Q.: ACCP: adaptive congestion control protocol in named data networking based on deep learning. Neural Comput. Appl. 31(9), 4675–4683 (2019). https://doi.org/10.1007/s00521-018-3408-2

    Article  Google Scholar 

  40. Kumar, N., Singh, A.K., Srivastava, S.: Feature selection for interest flooding attack in named data networking. Int. J. Comput. Appl. (2019). https://doi.org/10.1080/1206212X.2019.1583820

    Article  Google Scholar 

  41. Muscariello, L.: Supervised machine learning-based routing for named data networking. In: IEEE Conference and Exhibition on Global Telecommunications (GLOBECOM)

  42. Gong, L., Wang, J., Zhang, X., Lei, K.: Intelligent forwarding strategy based on online machine learning in named data networking. In: IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), December 2018, pp. 1288–1294 (2016). https://doi.org/10.1109/TrustCom.2016.0206

  43. Ghali, C., Narayanan, A., Oran, D., Tsudik, G., Wood, C.A.: Secure fragmentation for content-centric networks (extended version) (2014). http://arxiv.org/abs/1405.2861

  44. Afanasyev, A., Shi, J., Wang, L., Zhang, B., Zhang, L.: Packet fragmentation in NDN: why NDN uses hop-by-hop fragmentation. NDN, NDN Memo, Tech. Rep. NDN-0032, pp. 1–5 (2015). http://named-data.net/techreports.html

  45. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13 (1963). https://doi.org/10.2307/2282952

    Article  MathSciNet  MATH  Google Scholar 

  46. Afanasyev, A., Moiseenko, I., Zhang, L.: ndnSIM: NS-3 based Named Data Networking (NDN) simulator. http://ndnsim.net/2.7/, NDN Technical Report NDN-0005 (2012). https://named-data.net/publications/techreports/trndnsim/. Accessed 27 Mar 2020.

  47. Ioannou, A., Weber, S.: A survey of caching policies and forwarding mechanisms in information-centric networking. IEEE Commun. Surveys Tutor 18(4), 2847–2886 (2016). https://doi.org/10.1109/COMST.2016.2565541

    Article  Google Scholar 

  48. Prodhan, A.T., Das, R., Kabir, H., Shoja, G.C.: TTL based routing in opportunistic networks. J. Netw. Comput. Appl. 34(5), 1660–1670 (2011). https://doi.org/10.1016/j.jnca.2011.05.005

    Article  Google Scholar 

  49. Medina, A., Matta, I., Byers, J.: BRITE: A Flexible Generator of Internet Topologies|Guide books (2000). Accessed 27 Mar 2020

  50. Statistical tools for describing experimental data. In: Scientific Methods in Mobile Robotics, pp. 29–84. Springer, London (2006)

Download references

Funding

No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Chittagong University of Engineering & Technology, Chittagong, Bangladesh

    Shahid Md. Asif Iqbal,  Asaduzzaman & Mohammed Moshiul Hoque

  2. Department of Computer Science & Engineering, Premier University, Chittagong, Bangladesh

    Shahid Md. Asif Iqbal

Authors
  1. Shahid Md. Asif Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Asaduzzaman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed Moshiul Hoque
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SMAI—Conceptualization, modeling, methodology, analysis & investigation, software—computer simulation, validation, writing—original draft preparation, writing—review & editing, visualization. Asaduzzaman—conceptualization, modeling, analysis & investigation, supervision, writing—review & editing, visualization. MMH—modeling, analysis, supervision, writing—review & editing, visualization.

Corresponding author

Correspondence to Asaduzzaman.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

The work is a novel work and has not been published elsewhere nor is it currently under review for publication elsewhere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iqbal, S.M.A., Asaduzzaman & Hoque, M.M. A source-driven reinforcement learning-based Data reply strategy to reduce communication overhead in Named Data Networks (NDN). Cluster Comput 25, 647–673 (2022). https://doi.org/10.1007/s10586-021-03443-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-021-03443-9

Keywords

Search

Navigation