Elsevier

Computer Communications

Volume 145, September 2019, Pages 14-28
Computer Communications

A framework for the evaluation of routing protocols in opportunistic networks

https://doi.org/10.1016/j.comcom.2019.06.003Get rights and content

Abstract

The evaluation of routing protocols for opportunistic networks can be seen as a multidimensional problem because it involves several performance aspects. To capture these aspects various evaluation metrics are used, such as the number of delivered packets, the delivery delay and the number of transmissions. Unfortunately, in the context of opportunistic networks, these metrics are often highly correlated and usually conflicting. To make things worse, the characteristics of the network affect the importance of each metric as well as the levels of its correlation with other metrics. In this work, we first propose a set of performance evaluation metrics that are normalized with respect to the optimal performance. This approach tackles several of the above-mentioned shortcomings of traditional evaluation metrics. We then formulate the evaluation of routing protocols as a Multiple-Criteria Decision-Making (MCDM) problem where each routing protocol is an alternative and the performance metrics correspond to a set of criteria. We use this formulation to develop an evaluation framework that objectively ranks the performance of opportunistic routing protocols. To this end, we reshape well-known concepts and algorithms from the MCDM field to address the special requirements that are specific to the opportunistic context. We present detailed simulation results of well-known routing protocols in various opportunistic environments and rank their performance according to the proposed framework. In conclusion, no algorithm was able to achieve the best performance in all or the majority of the network topologies that we studied. This demonstrates the diversity of challenges that routing mechanisms face in such networks.

Introduction

The evaluation and comparison of routing protocols, especially of those designed for opportunistic networks, is a complex process that involves many performance aspects. Typically, researchers capture different performance features by using a set of performance metrics such as the average ratio of successfully delivered packets, the average end-to-end delay, the average number of transmissions, etc. In general, a detailed assessment requires multiple performance metrics to be evaluated jointly rather than individually. The latter task is not straightforward, mainly because the investigated performance features are often correlated or even conflicting. Moreover, the involved trade-offs are ambiguous because the properties of the intermittent network are typically not known. For example, a small number of transmissions, a key performance indicator for energy efficiency, usually coexists with limited delivery capability. When comparing two protocols, it is not always clear what is an acceptable degradation of delivery capability for achieving increased energy efficiency, even if one has a strong preference for the latter.

In this work we aim at developing a generic and easy to implement method for ranking the performance of a set of protocols in a specific network. The ranking should rely on the joint evaluation of all aspects of a protocol’s performance and take into account the performance trade-offs imposed by the network structure. This evaluation approach should be a valuable tool not only for assessing the performance of existing protocols but also for providing useful insight when designing new ones. Towards achieving our goal, we make the observation that we can formulate the evaluation of opportunistic routing protocols as a Multiple-Criteria Decision-Making (MCDM) problem. MCDM methods [1], [2], [3], [4] provide an evaluation of a set of alternatives using a set of criteria. In our approach, we visualize each routing protocol as an alternative and the performance metrics as the set of criteria. Unfortunately, legacy MCDM methods cannot address the full extent of the challenges faced in our scenario. This is mainly because of two reasons. The first relates to the normalization of criteria, i.e., the process of adjusting the values of different criteria to a common scale. Indeed, MCDM methods require such a normalization in order to create a common ground for combining the different criteria. We discuss wide-spread normalization methods in Section 3.1. Unfortunately, normalization significantly affects the outcome of the evaluation. In our scenario, an efficient normalization should consider the best performance allowed by the network. Reasonably, the normalization techniques used in MCDM methods are generic and therefore cannot provide the required “network-awareness”. The second reason for the limited efficiency of conventional MCDM methods in our scenario pertains to the performance metrics used in the opportunistic routing literature. As discussed in detail in Section 4.1, these metrics present a high degree of correlation that significantly limits the information that each metric offers. This correlation is typically non-linear and “network-depended” therefore it cannot be captured by traditional MCDM methods. Moreover, in several cases, the extent of correlation is such that a metric may provide misleading information. For example, a routing protocol may deliver only short-haul packets, i.e. the ones exchanged between non-distant nodes, and therefore exhibit a low average delay. On the other hand, an opponent protocol, delivering short-haul packets with the same delay but also handing over long-haul ones, will present a larger average delay. We further discuss this issue in Sections 4.1 Pitfalls in using performance metrics as evaluation criteria, 4.2 Proposed performance metrics.

We address the prior concerns by taking a two-step approach. First, we examine a new normalization technique through which we redefine traditional performance metrics. Our aim is to provide a network-aware version of those metrics and at the same time reduce their correlation as much as possible. In the second step, we focus on weighting, a key mechanism of MCDM methods. This is a process in which each metric is assigned a number that determines its importance compared to other metrics. We develop a correlation-aware weighting model that is suitable for our context. Such weighting models have been studied in the MCDM literature [4] (we review well-known weighting models in Section 3.2) but only examine linear correlation. Instead, we take a more generic approach that responds to both linear and non-linear relationships. Summarizing, our contributions are:

  • We formulate the evaluation of opportunistic routing protocols as an MCDM problem where the protocols are the alternatives and the performance metrics are the criteria. Based on this formulation, we develop a framework that ranks each protocol’s performance (Section 4).

  • We propose a set of modified performance metrics that stem from well-known ones (Section 4.2). The proposed metrics capture the performance of routing protocols with respect to the optimal one in a specific network. Thus, they provide the means for a network-aware assessment when used either in the context of the proposed MCDM framework or as baseline metrics in future evaluation of opportunistic routing protocols.

  • We propose VIC (Section 4.3), a weighting method that relies on the variability and the dependency (both linear and non-linear) of criteria in order to determine their relative importance. VIC can be applied to any MCDM problem for the assignment of objective weights and it is also compatible with the use of subjective, i.e. user-defined, ones.

  • We perform a detailed performance assessment for a wide range of opportunistic routing protocols using a variety of real-world contact traces that correspond to networks of different scales and structures (Section 5).

In the rest of the paper, we first review opportunistic routing protocols (Section 2.1) and the most popular performance metrics (Section 2.2). In Section 3, we provide background information on traditional MCDM methods. After delineating our contributions (Sections 4–5), we conclude this work in Section 6.

Section snippets

Routing in opportunistic networks

Routing in opportunistic networks follows the store-carry-and-forward approach. That is, a node may store packets for long periods of time and forward/replicate them upon a contact with another node. The main challenge in the design of opportunistic routing protocols is to determine whether a packet should be forwarded/replicated to an encountered node with the two extremes being Epidemic Routing [5] and Direct Delivery [6]. The former follows the most aggressive approach, i.e. replicates a

Multiple-criteria decision-making

In this section we review the Multiple-Criteria Decision-Making (MCDM) literature. In general, MCDM schemes consist of a decision-making algorithm and a weighting method.

A framework for the evaluation of opportunistic routing protocols

As previously discussed, our observation is that it is possible to formulate the evaluation of opportunistic routing protocols as an MCDM problem where each protocol is an alternative and each performance metric is a criterion. Nevertheless, there are several pitfalls in using the traditional performance metrics as criteria. We address such problems and propose solutions in Sections 4.1 Pitfalls in using performance metrics as evaluation criteria, 4.2 Proposed performance metrics. Then, in

Simulation setup

We selected four datasets of varying scale to evaluate the performance of routing protocols for opportunistic networks according to the proposed framework. More specifically, the datasets that we selected are the following: Reality Mining [48], [49], INFOCOM 2005 [50], [51], Lyon [52], [53], and Dartmouth [54], [55], [56]. The Lyon dataset was downloaded from the website of the SocioPatterns collaboration [57], while the rest datasets were downloaded from the website of the CRAWDAD archive [58]

Discussion and conclusions

There are some factors that should be taken into consideration in order to apply the proposed framework. First of all, the decision matrix should contain performance values of a representative set of routing protocols. Even if we are interested in the performance comparison of only a small number of routing protocols, the performance values of algorithms such as Epidemic Routing and Direct Delivery should be included. This is because such algorithms provide useful information for the

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (60)

  • WangY.-M. et al.

    Integration of correlations with standard deviations for determining attribute weights in multiple attribute decision making

    Math. Comput. Modelling

    (2010)
  • Alemi-ArdakaniM. et al.

    On the effect of subjective, objective and combinative weighting in multiple criteria decision making: A case study on impact optimization of composites

    Expert Syst. Appl.

    (2016)
  • ShipleyM.F. et al.

    A decision making model for multi-attribute problems incorporating uncertainty and bias measures

    Comput. Oper. Res.

    (1991)
  • TriantaphyllouE. et al.

    An examination of the effectiveness of multi-dimensional decision-making methods: A decision-making paradox

    Decis. Support Syst.

    (1989)
  • HendersonT. et al.

    The changing usage of a mature campus-wide wireless network

    Comput. Netw.

    (2008)
  • HwangC.-L. et al.
  • VahdatA. et al.

    Epidemic routing for partially connected ad hoc networks

    (2000)
  • GrossglauserM. et al.

    Mobility increases the capacity of ad hoc wireless networks

    IEEE/ACM Trans. Netw.

    (2002)
  • SpyropoulosT. et al.

    Routing for disruption tolerant networks: taxonomy and design

    Wirel. Netw.

    (2010)
  • ZhuY. et al.

    A survey of social-based routing in delay tolerant networks: Positive and negative social effects

    IEEE Commun. Surv. Tutor.

    (2013)
  • H. Dubois-Ferriere, M. Grossglauser, M. Vetterli, Age matters: Efficient route discovery in mobile ad hoc networks...
  • LindgrenA. et al.

    Probabilistic routing protocol for intermittently connected networks RFC 6693

    Internet Res. Task Force (IRTF)

    (2012)
  • V. Erramilli, A. Chaintreau, M. Crovella, C. Diot, Diversity of forwarding paths in pocket switched networks, in:...
  • SpyropoulosT. et al.

    Efficient routing in intermittently connected mobile networks: The single-copy case

    IEEE/ACM Trans. Netw.

    (2008)
  • SpyropoulosT. et al.

    Efficient routing in intermittently connected mobile networks: The multiple-copy case

    IEEE/ACM Trans. Netw.

    (2008)
  • T. Spyropoulos, K. Psounis, C.S. Raghavendra, Spray and wait: An efficient routing scheme for intermittently connected...
  • T. Spyropoulos, K. Psounis, C.S. Raghavendra, Spray and focus: Efficient mobility-assisted routing for heterogeneous...
  • SpyropoulosT. et al.

    Routing in delay-tolerant networks comprising heterogeneous node populations

    IEEE Trans. Mob. Comput.

    (2009)
  • S.C. Nelson, M. Bakht, R. Kravets, Encounter-based routing in DTNs, in: Proceedings of the 28th IEEE International...
  • DalyE.M. et al.

    Social network analysis for information flow in disconnected delay-tolerant manets

    IEEE Trans. Mob. Comput.

    (2009)
  • Cited by (0)

    View full text