1 Introduction

During the past decade, the academic community has extensively studied peer-to-peer (P2P) networked virtual environments (NVEs). This research activity has been driven by the phenomenal success of other P2P applications, typified by file-sharing applications (e.g., Bit-Torrent), and video streaming systems (e.g., PPLive). However P2P architectures have not yet found their ways in NVEs, partly because the development of fully-distributed NVEs has turned out to be especially challenging. In this paper, we review over ten years of research related to the design of fully distributed infrastructure-less NVEs. Our goal is to give a thorough description of the main approaches that have been explored so far, and to emphasize their main advantages and their weaknesses. We think that it is the right time to survey in a fair manner these approaches because technological shifts, in particular the development of cheap computing and storage facilities in data-centers, have made P2P NVE systems less attractive. We discuss the perspectives around these works for the next years. We believe that distributed algorithms for the management of large-scale NVEs are still needed, in a context that is just slightly different. We discuss these perspectives at the end of the paper in a more personal opinion.

1.1 Fundamentals about NVEs

Networked Virtual Environments (NVEs) can be classified into three major categories: multiplayer online games, military simulations, and social virtual worlds.

  • Among the most popular NVEs, multiplayer online games are computerized games in which a high number of players play in the same game environment at the same time. Multiplayer online games can be further classified into First Person Shooter, Real-Time Strategy and Role-Playing games. Below are some typical examples:

  • First Person Shooter: Doom [1], Quake Live [2], Marathon [3], Blake Stone: Aliens of Gold [4], Star Wars: Dark Forces [5].

  • Real-Time Strategy: Homeworld [6], Starcraft II: Wings of Liberty [7], Company of Heroes [8].

  • Role-Playing: Ultima Online [9], EverQuest [10], Final Fantasy XI [11], EVE online [12], World of Warcraft [13].

  • NVEs originated from military simulations designed for training purposes. Typical examples are SIMNET [14], DIS [15], HLA [16], FALCON 4.0 [17] and America’s Army [18].

  • Social virtual worlds are online communities living in computerized virtual environments and aiming at socializing with other users via mutual interactions within the virtual world. Typical examples are Second Life [19], HiPiHi [20], The Sims Online [21], There [22] and Active Worlds [23].

A typical virtual world is made up of immutable objects (e.g., terrain geometry, textures), mutable objects (e.g., food, weapons, tools), characters controlled by players (i.e., avatars) and non-player characters controlled by automated algorithms [24]. The goal of a NVE system is to provide to its participants a shared sense of space, time, and presence through virtual representations, called avatars. This idea has derived into a set of non-functional requirements. In particular, users observing the same virtual scene need to have the same view on data at one time (i.e., consistency). A NVE should respond to users’ events at a fast rate (i.e., responsiveness), since high latencies degrade user experience; for instance, first person shooter games cannot tolerate latencies above 200 milliseconds [25]. In addition, users should be prevented from cheating, and users’ data should be protected from being changed (intentionally or unintentionally) by other users (i.e., security). Content within NVEs should also remain available across different user sessions, so in-world objects’/avatars’ data should be stored on a persistent storage device (i.e., persistence). NVEs should be able to survive failures in order to continuously provide the expected service (i.e., availability). As NVEs are becoming increasingly popular, they are expected to support a high number of simultaneous participants (i.e., scalability), and consequently deal with potentially highly populated regions in the virtual world (i.e., load-balancing).

To address these requirements, NVEs have traditionally been developed on client/server architectures. The server (or a cluster of servers) maintains the world data, calculates state changes and disseminates state updates to the different users. A user informs the server about the events that he/she creates and is informed by the server about state changes of players and objects in the virtual world. Since a player is only interested in the portion of the virtual world that is located near the player’s avatar, the server usually only sends information about events that occurred within the Area of Interest (AoI).

1.2 Peer-to-peer NVEs

The scientific community has advanced peer-to-peer (P2P) systems as an alternative design choice for building NVEs. We identify three main motivations behind this choice. First is the perspective of cheap scalability. Centralized systems can indeed lead to communication and computational overhead on the servers. To support an ever growing user population, the only solution is to increase the number of servers. To get rid of expensive infrastructure has become a key driver for scientists. The second motivation is related to avoid possible points of failure. Fully distributed architectures exhibit extra-ordinary self-healing and self-stabilizing properties. Since maintaining the redundancy of such large-scale applications is hard, scientists proposed P2P solutions. Finally, the third motivation refers to the libertarian vision. Like in social networks, users are concerned by the fact that a central authority owns the VE, tracks all the activities of users, potentially uses logs for profit, and might use censorships on the activities within the virtual world. A P2P infrastructure appears as a way to build an authority-less NVE, where users are free to act and have control over several important aspects of their participation to the NVE.

In P2P systems, the overall system load is distributed among all participating users/peers, which organize themselves so that they all have an equal role and act as both clients and servers. Since in a virtual world a player is only interested in its AOI, a key issue in a NVE is to answer the following question: how can each player be informed of the state changes of other players and objects within its AOI while moving around the world? In other words, how can a state update message be sent to relevant players?

To add to complexity, P2P NVEs bring some additional constraints that are inherent to the fact that the software is running on the machines of end-users. Given the relatively limited (upload) bandwidth and processing power of users’ machines, P2P NVE designers tend to limit the number of connections as well as the number of exchanged messages between participating peers. Moreover, network latency and message losses make maintaining strict consistency extremely challenging. Finally, direct connections between interacting players should be preferred because the transmission of a message through several intermediate peers leads to high message dissemination delays.

1.3 The idea behind this survey

As illustrated by the number of references in the present document, numerous P2P-based NVE systems have been proposed. In this paper, we focus on recent efforts to build P2P overlay networks for the full support of NVEs. This means that, for instance, special mechanisms that deal with physical network latency are out of the scope of this survey even if they refer to both P2P and NVEs. In other words, we do not review the works that improve one specific non-functional requirement in coordination with a traditional client-server system. Instead, we are interested in systems that build their own software infrastructure by leveraging exclusively on the machines of end users. At a time where large-scale distributed systems are more than ever deployed, we believe that leveraging on lessons learned and experiences gained from a decade of intensive efforts and scientific works through a comprehensive state-of-the-art is necessary. Most of these previous works did not envision to be used in what is now known as the cloud. Nonetheless, they are of high interest in the perspective of deployment of NVEs in large-scale distributed infrastructures.

We identify six key approaches to construct the overlay network: the world partitioning, DHT, multicast, fully-connected, neighbor-list exchange and mutual notification approaches. Given the plethora of existing efforts in these directions, the present paper is not intended to be exhaustive of all existing related works, but rather to study, through representative examples of each approach, the general organization of the overlay and the message schemes that such overlay implies. More precisely, we describe (i) the mechanisms that allow the overall system to stay connected, (ii) the main schemes for message exchange, and (iii) the distribution of data. We summarize at the end of each Section the main advantages and drawbacks of the approach. We discuss their comparative interest for the future in the Discussion part.

The remainder of this paper is organized as follows. Section 2 provides a brief overview of the different P2P NVE approaches. Sections 3, 4, 5, 6, 7 and 8 provide a detailed discussion of each approach, its advantages and drawbacks, and examine several representative P2P NVE systems based on the presented approaches. Finally, Section 9 provides a critical analysis of the presented approaches and Section 10 concludes this work.

2 P2P NVE approaches: A brief overview

The different P2P NVE approaches are briefly introduced in the following.

World partitioning approach

partitions the virtual world into regions, which group a set of players potentially interested in each other. The manager of a region manages peers and objects inside its region. The main issues are efficient sized partitioning of the world and the management of player crowding.

DHT approach

uses Distributed hash tables (DHTs), which provide peer connectivity through a dynamic logical overlay, as well as efficient lookup through a hash table-like functionality and load balancing of the overall system’s load. The open issues are the fair distribution of the system load among peers, and latency due to the number of hops associated with every message over the DHT key space.

Multicast approach

is used to disseminate update messages to relevant peers (i.e., multicast group) in the virtual world. A multicast group is formed for each world region or entity (e.g. avatar, object) to inform group members about corresponding state changes. P2P-based NVE systems propose both IP multicast and application-layer multicast (ALM). However, IP multicast is not widely deployed due to group and network management issues. In ALM, the management of multicast groups is still challenging, due to a high number of group members as well as highly dynamic players in the virtual world.

Fully-connected approach

connects all peers to each other in the VE while using message filtering techniques to limit message transmissions. Players thus frequently send messages to only a small group of players, while infrequently to all other players in the world. Fully-connected systems do not scale as each player needs to communicate with all other players.

Neighbor-list exchange approach

connects peers closest to each other in the virtual world. Directly connected peers regularly inform each other about not only themselves but also their own neighbors through neighbor-list exchange. However, global network connectivity is not guaranteed and constant neighbor-list exchanges may introduce a high message overhead.

Mutual notification approach

connects nearby peers in the VE usually based on well-known geometrical structures (e.g., Voronoi diagram). Global network connectivity and neighbor discovery are realized only when necessary through mutual notification among direct neighbors. However, the underlying overlay might induce high maintenance cost as it is subject to high connection change rate due to the continuous movement of players.

3 World partitioning approach

The partitioning of the virtual world into regions was the first used method to construct a P2P NVE system [26–29]. Each region is assigned to a manager peer which is a peer with sufficient resources in terms of computing power, bandwidth and storage capacity, responsible for the management of the region. A region groups a small number of entities potentially in relation to each other (e.g., players potentially interested in each other and objects in the region). However, NVEs can be supported by the world partitioning approach under the assumption that the resources of each manager are sufficient to manage its region.

The manager peer manages the data of objects of its region. The data of an avatar on the other hand can be kept by either the manager of the region inside which the avatar resides [30] or the peer controlling the avatar [24]. The manager of a region determines the relations between peers based on players’ overlapping AOIs. Regular peers residing in a region inform their manager about the events they create (e.g., position changes). The manager in return informs regular peers about state changes due to events occurring in its region. Regular peers directly exchange messages to reduce the load on the manager [31], while neighboring managers exchange their regions information in order to provide a continuous view to their regular peers [31, 32].

3.1 Different shaped partitioning

The virtual world can be partitioned into different shaped and sized regions. Rectangular [24, 33–36], tree-based [32, 36–38], hexagonal [31, 39–42], and polygonal (e.g., Voronoi) [30, 35, 43–45] regions are the most popular partitioning forms.

MOPAR [31] partitions the virtual world into hexagonal cells (Fig. 1). Each hexagonal cell is assigned to a home node through a DHT. A home node is used for assigning the cell to a master node which is responsible for the cell management (i.e., a DHT is used as a rendezvous point for master node registration). The first node entering into a cell becomes the master node of the cell, otherwise it becomes a slave node. The master of a cell keeps information about slave nodes residing inside the cell. Slave nodes inform their master about their position changes in the VE, and are notified of new neighbors by their master. Slaves directly inform each other about their movements to decrease the load on their master node. Masters exchange their slave lists to be informed about the slaves residing in the neighboring cells and thus to completely cover the AOI of their slaves in order to provide them with a continuous view of the virtual world. However, although slave list exchange is performed only between masters, its frequency is still a concern. When a master leaves its cell, one of the slaves residing in the cell becomes the new master, to which the cell data is transferred. The continuous data transfer to newly assigned cell masters can create a significant overhead in the system. Besides, the hexagonal zoning does not allow the repartitioning of a cell into smaller regions when the cell master becomes overloaded due to slave crowd in the cell. Moreover, in case of crowding, excessive bandwidth requirements beyond slaves’ capabilities can occur due to direct connectivity between them.

Fig. 1
figure 1

Hexagonal partitioning - Mopar [31]

Full size image

3.2 Static and dynamic partitioning

Besides different shaped zoning, NVEs based on world partitioning can follow two types of world division strategies with different partitioning dynamics: static and dynamic. In static partitioning, the virtual world is partitioned into fixed-size regions (i.e., prepartitioning of the virtual world into regions); thus, the regions do not change with the evolution of the VE [24, 31, 41, 46, 47]. In dynamic partitioning, on the other hand, regions can evolve over time due to application needs. A region can be repartitioned into smaller regions each of which is assigned to a new manager [34, 36–38, 45, 48–50]. In the following, COVER [37] and Badumna [35] are presented as P2P NVE system examples following different types of world partitioning strategies.

COVER [37] proposes a dynamic world decomposition through a global quadtree to partition the world by recursively subdividing it into four quadrants (i.e., regions) depending on the number and position of players (Fig. 2). Each region is assigned to a super-node, the closest player to the geometric center of the region. Each player is directly connected to all other players in its AOI. A player is called as a covered player if its AOI is completely covered by its neighbors’ AOI, and as an uncovered player if its AOI is not completely covered. To provide awareness, each player sends its movement update messages to all its AOI neighbors, which in turn propagate the messages to their own AOI neighbors. Players receiving a message can thus determine their AOI neighborhood relationship with the player sending the message. A super-node is responsible for the management of the uncovered players in its region. Uncovered players inform not only their AOI neighbors but also their super-node about their position changes in the VE. A super-node can thus provide awareness to uncovered players located within its region. However, players’ movement can produce frequent changes in the underlying quadtree structure, making the management of the quadtree structure difficult. When crowding occurs in a small area inside a region, successive divisions are needed to be able to partition the crowded area, which results in several nearly empty regions. In addition, in case of crowding, direct connectivity to all AOI neighbors can result in excessive bandwidth requirements beyond players’ capabilities.

Fig. 2
figure 2

COVER [37]

Full size image

Badumna [35, 51] provides both static partitioning through fixed-size cells and dynamic partitioning through dynamic bounded regions, which can change size and position as required for the management of a dense region (Fig. 3). Badumna is based on both world partitioning and DHT approaches. Both cells and bounded regions are assigned to peers through the underlying DHT. Entities query the DHT to find out about other entities in their AOI. A bounded region containing several entities queries the DHT as a single entity (i.e., one query serving for all entities in the region), thus reducing the traffic on the DHT. In addition, a gossip protocol is proposed for the management of a highly dense area (i.e., a large number of players clustered in a small area) in order not to overload the peer responsible of this area. In a highly dense area, players reduce the frequency of their interaction with their manager peer; instead, players rely on their neighbors to inform each other about other players in the region based on the gossip protocol. However, players’ frequent position changes in the VE make the management of bounded regions difficult. In addition, the search for interest purposes through queries on the DHT introduces unacceptable latency due to the number of hops required by the underlying DHT structure.

Fig. 3
figure 3

Badumna [35]

Full size image

3.3 Discussion

Table 1 presents a list of the different NVE systems presented in this section. The world partitioning approach is widely used to construct P2P NVE systems in a simple manner, where the global world management is divided into the management of small regions containing entities possibly in relation based on interest. However, the partitioning of the virtual world, which provides a fair distribution of the VE load among managers, remains a challenging issue. Note that a large number of players can congregate in a region for interest purposes and crowding can thus dynamically occur in different regions due to players’ frequent position changes in the VE.

Table 1 The presented world partitioning based NVE systems
Full size table

To address this issue, in static world partitioning-based NVEs, communication is generally constructed based on a secondary mechanism in order to reduce the load on managers, such as message dissemination in a region through load balancing trees in case of crowding (e.g., Distributed event delivery [33]), application layer multicast based on the underlying DHT structure (SimMud [24]), direct communication between regular peers (e.g., MOPAR [31]), etc. Dynamic partitioning is proposed as an alternative solution, in which overloaded regions due to high density are repartitioned into smaller regions managed by new manager peers. However, region reassignments and management of dynamic entities can create a significant overhead on the system. A dynamic partitioning approach capable of providing static assignment of entities to managers is thus the most suitable world partitioning approach to the NVE requirements, which can also be supported by a secondary communication mechanism as in static world partitioning.

4 DHT approach

The use of distributed hash tables (DHTs) [52–54] is another method commonly used to support P2P NVEs. DHTs provide an efficient way of indexing data items by fairly balancing the system load among all participating peers. DHT systems implement a hash function that maps any given item to a specific peer in the system. Items and peers are mapped to a key space, which is partitioned among all peers. Each peer is responsible of a specific key space partition. DHTs provide dynamic maintenance of peers, efficient lookup and load balancing of both resources and query messages.

P2P NVE systems partitioning the virtual world into regions can use a DHT to assign each region to a manager peer [24, 31, 35, 36, 38, 44, 46, 49, 55]. On the other hand, Colyseus [56] and Loader [57] use a DHT for distributing entities, instead of regions, to peers.

To ensure a fair distribution for load balancing purposes, DHT systems use a uniform hash function that randomly maps items and peers to a common key space [53, 54]. However, hashing destroys both the physical and logical locality relation among peers. Pastry [52] takes into account physical network locality, which minimizes the distance that the messages have to travel. However, peers residing inside the same region in the virtual world are still placed far-apart in the DHT key space. Thus, peers probably interacting with each other are connected via multiple hops over the DHT, which may result in high routing latency. To address this issue, NL-DHT [58], Loader [57], Walkad [38] take into consideration the logical locality (i.e., distance in the virtual world) by mapping nearby peers in the virtual world to nearby keys on the DHT space. In addition, Colyseus [56], OPeN [55], Walkad [38] propose range queries on the DHT to find the set of entities located in peers’ area of interest.

To support communication between peers inside a region, SimMud [24], PastryMMOG [40], Loader [57] and PBT [36] use Scribe [59], the application layer multicast built on the underlying DHT overlay, Pastry [52]. However, to avoid unnecessary network delay due to possible number of hops on the DHT, [31, 32, 34–36, 46, 49, 56] use DHTs only as a rendezvous point for peers to discover relevant entities, and propose a communication model independent from the DHT routing infrastructure (e.g., direct, multicast, etc.) for update message dissemination.

4.1 Representative systems

This section examines several P2P NVE systems based on a DHT, namely Colyseus [56], SimMud [24] and PBT [36]. Colyseus assigns entities to peers, while SimMud and PBT assign regions. Update messages are disseminated to relevant peers through a publish-subscribe fashion in Colyseus, and through Scribe [59] in SimMud and PBT.

4.1.1 Colyseus

Colyseus [56] proposes a NVE system based on Mercury [60], which is used as both a traditional randomized DHT and a range-queriable DHT to distribute objects to peers and to discover objects within the AOI of a peer. Each object has one primary copy assigned to a peer through the DHT, and may have several replicas on different peers in the system to enable quick local access to the object. The primary owner of an object is responsible for the serialization and synchronization of the updates to the object performed on any peer in the system. Peers interested in an object subscribe to the object owner through the DHT. The owner of an object periodically publishes the object data to the subscribers.

The components of Colyseus nodes are presented in Fig. 4a, while an example of publication and subscription in Mercury is given in Fig. 4b. H x and H y are the range-queriable DHT hubs corresponding to the X and Y coordinates, respectively, of a two dimensional world of 320 × 320 units. The publication is sent to both H x and H y , where it is stored at peers b and e respectively. However, a subscription is passed to exactly one of the hubs corresponding to the required attributes. The subscription thus enters H x at peer d and is processed at peers b and c.

Fig. 4
figure 4

Colyseus’ system design

Full size image

Through interest prediction, a peer prefetches the replica of the objects likely to be needed in the near future (e.g., objects entering in the player AOI due to the player movement in the world), and subscribes to the owner of these objects through the DHT. However, a short-lived object (e.g., missile) might not be consistently prefetched. To address this issue, Colyseus proposes a proactive replication algorithm which attaches a short-lived object to a peer before the object is requested by the peer based on the fact that most short-lived objects originate at locations very close to their creator; thus, peers interested in the creator will probably be interested in the newly created objects.

Colyseus performs better scalability and load balancing with a range-queriable DHT compared to a randomized DHT. However, in a dynamic world based on a range-queriable DHT, the continuous movement of objects introduces a high message overhead due to the frequent DHT update requirements. In addition, since the neighborhood relations between peers are built independently from the logical locality relations among them in the VE, peers residing inside the same area in the VE are connected via multiple hops over the DHT, which may result in high latency. Finally, in a world with a large number of peers, looking up objects with O(log N) search hops may create unacceptable object discovery latency.

4.1.2 SimMud

SimMud [24] is a distributed NVE system following the static world partitioning, DHT and multicast approaches. SimMud’s architecture is based on Pastry [52] and its corresponding application layer multicast, Scribe [59]. The virtual world is partitioned into fixed-size regions containing players and objects (Fig. 5). Using Pastry, each region is randomly mapped to a coordinator peer that manages all objects in the region and serves as the root of the multicast tree to which all players residing in the region subscribe. State changes occurring in a region are periodically multicast by the coordinator to all region members. Coordinators connect to each other through Pastry to enable player migration between regions. Due to disjoint region partitioning, SimMud does not provide players a continuous view of the virtual world. Since a region’s area may exceed players’ AOI, players receive irrelevant messages beyond their AOI. If many players crowd in a region, the coordinator can quickly become a bottleneck and suffers from performance and robustness issues. In most cases, messages are delivered in less than 6 hops on Pastry’s address space. However, due to Scribe multicast routing, the number of hops might go beyond 50 hops, resulting in a delay up to several seconds, since a multicast tree is not constructed based on the AOI concept, and so may include peers not residing inside the region.

Fig. 5
figure 5

SimMud [24]

Full size image

4.1.3 PBT

PBT [36] is a P2P NVE system following both the static and dynamic world partitioning, DHT and multicast approaches. The whole world is first partitioned into static regions, each of which is assigned to a region master through the underlying DHT (Fig. 6). Players interested in a region subscribe to the region through its region master, to be informed about state changes occurring inside the region. Update dissemination inside a region is maintained based on Scribe [59]. In addition, in order to manage a static region with a high number of entities or subscribers, a Prediction Binary Tree (PBT) is built on top of the region, which becomes the root of the PBT. When necessary, a region is dynamically partitioned into subregions, which correspond to the leaves of the PBT managed by the region master. Each subregion is assigned to a subregion worker through the DHT. A new multicast group is constructed for each subregion. A subregion worker manages the entities inside the subregion and multicasts update messages to subscriber peers. Compared to SimMud [24], PBT additionally provides continuous view of the virtual world and dynamic partitioning of crowded regions. However, players’ frequent position change makes it extremely difficult to efficiently manage a PBT in order to provide an adequate management of world regions. In addition, PBT is subject to message transmission latency, as SimMud, due to the underlying DHT structure and application layer multicast.

Fig. 6
figure 6

PBT’s system architecture [36]

Full size image

4.2 Discussion

The DHT-based NVE systems presented in this section are listed in Table 2. A DHT assigns items to peers in a simple way; thus, VE items can efficiently be distributed among peers via a DHT. However, the relation between items and peers can be destroyed by the hash function due to random mapping of items and peers over the DHT key space. On the other hand, DHT-based NVE systems addressing this issue by mapping items on the DHT key space based on the logical locality can suffer from the high dynamism of the VE in terms of position changes in the virtual world. In addition, frequent entity discovery and message dissemination based on a DHT can cause unacceptable delay for NVE applications. Therefore, a DHT can be used for static assignment of entities to peers while taking into account the logical relation in the virtual world; however message dissemination based on an approach other than a DHT might produce more efficient results in the context of NVEs.

Table 2 A list of the representative DHT based NVE systems
Full size table

5 Multicast approach

In a VE, when the state of an entity is modified due to an event, related update messages are disseminated to players interested in the entity [61]. Multicast is a method used to send an update message to relevant players, namely, a multicast group. A world region or an entity is assigned a multicast group to inform group members about corresponding state changes. Players subscribe to the multicast group of regions or entities intersecting with their AOI to receive relevant update messages. P2P-based NVE systems exploit both IP multicast and application layer multicast (ALM) for update message dissemination. In the following, we examine different types of multicast schemes proposed by NVE systems.

5.1 IP multicast

Both region and entity-based IP multicasts are proposed by NVE systems to support message dissemination in VEs. In region-based IP multicast, the virtual world is partitioned into regions, each of which is assigned a multicast group through a multicast address (e.g., NPSNET [26], SPLINE [28], SCORE [62], DIVE [63]). When an event takes place inside a region, the related message is multicast to all players residing in the region. On the other hand, entity-based IP multicast focuses on the entity, instead of the region, and assigns each entity (e.g., avatar) a group to multicast its state information to other interested players (e.g., VELVET [64], Massively Multiplayer Audio Communication [65]).

5.2 Application layer multicast

Application layer multicast (ALM) has been proposed to provide multicasting functionality as an application service instead of a network service, based on overlay networks. In ALM, a multicast tree is constructed through unicast links between overlay network peers forming the multicast group [66]. NVE systems propose both physical and logical locality aware ALM algorithms to support message dissemination.

5.2.1 Physical locality aware ALM

ALM algorithms typically construct their multicast tree based on the characteristics of unicast links between peers (e.g., end-to-end latency) in order to minimize the additional routing overhead compared to IP multicast [67]. Physical locality aware ALM thus realizes update message dissemination based on nearby peers in the physical network in terms of latency. NVE systems based on physical locality aware ALM are detailed in the following.

Region-based physical locality aware ALM

This approach has been used in [24, 32, 33, 36, 39, 40, 42, 68–70]. These systems follow the region-based multicast strategy and propose an ALM scheme to disseminate update messages to players inside a region in the virtual world. SimMud [24] and PastryMMOG [40] are based on Pastry [52] and use its corresponding ALM, Scribe [59]. The virtual world is partitioned into regions, each assigned a multicast group with a unique ID. Regions are distributed to peers through Pastry. The peer with an ID closest to a region ID serves as the manager of the region and the root of its multicast tree. The manager of a region manages the residing entities and multicasts messages related to events occurring in the region, to the players residing inside the region.

Various systems [33, 39, 42, 68–70] have proposed a different region-based ALM algorithm, which aims to provide an efficient group management, update transmission delay and load balancing. MM-VISA [42, 71, 72] organizes and manages entities by partitioning the virtual world into hexagonal regions, each of which is assigned to a coordinator, responsible for multicast-based communication between players inside a region. Coordinators are connected to each other to support interregional communication due to AOI overlaps of players with different neighbor regions. To provide a better multicast group management, entities are clustered into classes according to their movement behavior in the virtual world (e.g., stationary, slow, normal and fast moving objects). Multicast trees in a region are constructed based on one cluster; in other words, the members of a multicast group belong to the same cluster of the region. While constructing a multicast tree, the cumulative end-to-end delay between a coordinator and peers is taken into consideration, which results in multicast trees with limited height. When all tree paths in a region reach the maximum height limit, a new multicast channel is opened for newly joined/arrived players to the region. Figure 7 represents the ALM scheme and message propagation in MM-VISA architecture, where C stands for coordinator, and O-1, O-2, O-3, and O-4, for classes having different behaviors.

Fig. 7
figure 7

ALM scheme and message propagation in MM-VISA [42]

Full size image

Entity-based physical locality aware ALM

In entity-based ALM, each entity’s update messages are disseminated based on ALM multicast to players interested in the entity. P2P propagation [73] is a client/server system using P2P multicast for message dissemination to reduce the server load. P2P propagation follows the entity-based multicast strategy, while proposing its own ALM algorithm. Each entity is assigned a multicast group. The set of peers interested in an entity is determined by the server. For each multicast group, the peer having the minimum delay to all other peers inside the group is selected as relay to forward a message destined to the group, from the server to the group members. The update message of an entity thus goes from the server through the relay of the multicast group to reach the members (Fig. 8). However, when crowding occurs, a relay can be overloaded due to message delivery to all group members.

Fig. 8
figure 8

P2P propagation [73]

Full size image

5.2.2 Logical locality aware ALM

Players’ interest in the virtual world can follow a different order than the peers’ layout on the latency-based multicast tree. The message transmission delay between nearby players in the virtual world can be more significant than the delay between distant players, since distant players have more tolerance for weak synchronization than nearby players. Logical locality aware ALM is thus proposed to prioritize update message dissemination to players that are near the event. NVE systems based on logical locality aware ALM are detailed in the following.

Region-based logical locality aware ALM

To disseminate update messages first to the most interested players (i.e., nearby players to the event), N-Trees [32] and PIMTRE [74] construct a multicast tree based on the interest relation between peers in the multicast group. A peer residing in a region transmits update messages due to an event, first to nearby peers residing in the region, then to distant peers residing in the neighbor regions within the event’s scope. Therefore, nearby peers in the world can receive messages faster.

N-Trees [32] divide the virtual world into nested rectangular regions based on a quadtree (i.e., 2-Tree) in order to organize players by their interest (Fig. 9). A node in an N-Tree is either a leaf node (i.e., a node with no children) or a node with 2N children. Each node in the N-tree is managed by a selected peer, called leader. Each leaf node in the N-Tree corresponds to a world region. Each successive level below the root node in the N-tree represents a further subdivision of the virtual world by 2N. Players residing inside a region form a group and join the corresponding leaf node. When a player generates an event in a region, the player disseminates the event message to other players, including the leader, residing in the same leaf node corresponding to the region. If the event scope exceeds the region boundaries, the message is then propagated to the regions (i.e., nodes) intersecting with the event scope through their leaders. When the number of players in a region, namely the population of a leaf, exceeds a given threshold, the leaf in the N-Tree is subdivided into N children and the players are placed into the appropriate leaves. The subdivision of a crowded region into subregions decreases the bandwidth requirements of a player since the number of players to which a player should send its update messages is reduced. However, leaders managing the corresponding leaves and nodes may be overloaded due to interregional message transmission. In addition, players’ movement can produce frequent changes in the underlying N-tree structure, making the management of the N-tree structure difficult.

Fig. 9
figure 9

N-Trees [32]

Full size image

Entity-based logical locality aware ALM

Apolo [75], AOICast [76] and pSense [77] follow a similar approach to N-Trees and PIMTRE, while using the entity-based multicast strategy. Players multicast their update messages to other players inside their AOI based on the locality relation between players in the virtual world.

Apolo [75] is the first P2P NVE system achieving message through AOI-based multicast trees. Apolo connects each peer to its closest peer in each of the four quadrants in the virtual world (Fig. 10). A peer p informs other peers inside its AOI about its state change through a multicast tree built on top of the overlay (Fig. 10b). The source peer generating the event, namely p, becomes the root of the multicast tree. For a peer q in p’s AOI, the closest peer to p among q’s closest quadrant neighbors becomes q’s tree parent. The event update message is first received by peers closest to the source peer p, and then relayed to other farther peers. However, the connectivity of each peer to exactly four peers can lead to multicast trees with long paths, thus increasing the number of hops (i.e., latency) needed for sending update messages to AOI neighbors.

Fig. 10
figure 10

Entity-based logical locality aware ALM - Apolo [75]

Full size image

5.3 Discussion

A list of the multicast-based NVE systems given in this section are listed in Table 3. Multicast provides an efficient way of communication between peers by distributing message dissemination load among all relevant peers. However, entity-based multicast might create different trees with the same multicast group members (e.g., multicast tree structures for entities that are nearby in the virtual world), thus overloading peers participating to multicast trees. On the other hand, in region-based multicast, the region size has an important role in the efficiency of message delivery. In case of large regions, since the region may exceed players’ AOI, players receive irrelevant messages beyond their AOI. With small regions, since players’ AOI may cover several regions, the same players become participants of several multicast trees, thus increasing computation overhead on peers managing the multicast trees (i.e., tree roots), and players participating in the multicast.

Table 3 A list of the multicast-based NVE systems
Full size table

IP multicast offers efficient message delivery inside a group, however it is not widely deployed due to issues related to group management, multicast address allocation and network management [78, 79]. On the other hand, in ALM, peers’ upload bandwidth is a critical resource that should be optimized. The depth of the ALM tree is increased as group size increases, leading to long latency which may not be appropriate for NVE requirements [80]. In addition, message dissemination may be delayed in ALM algorithms that do not construct multicast trees based on the AOI concept; for instance, Scribe [59] can create trees with long paths which include peers transmitting event messages while not residing in event scopes. As such, the properties of both the underlying physical network as well as the logical network built on top of the physical network should be considered to provide an efficient multicast algorithm.

6 Fully-connected approach

In the fully-connected approach, all players in the virtual world are directly connected to each other. Since everyone is connected to every-one else, message filtering techniques are used to limit message transmissions between certain peers to make the system feasible. Players frequently send messages to only a small group of players, while infrequently to all other players in the world. P2P NVE systems based on this approach focuses mainly on the management of avatars where avatars’ data are stored on the corresponding user machines. A world object, if any, can be treated as an avatar and its data are thus managed on the machine of the user creating the object.

6.1 Representative systems

Two fully-connected P2P NVE systems, Frontier set [81] and Donnybrook [82] are examined in the following.

Frontier set

Frontier set [81, 83] is inspired from update-free region (UFR) in the geometric message filtering protocols [84, 85] and potentially visible set (PVS) [86]. The UFR concept is used to determine the message exchange protocol between players, while the PVS concept is used to group players interested in each other and thus frequently exchanging messages.

In the geometric message filtering protocols [84, 85], each player of any pair of players defines its respective update-free region (UFR) inside which the player does not inform the other about its position change. When a player leaves its UFR, the pair of players become relevant to each other. From that time on, update messages are thus mutually and regularly transmitted between the two players. When they become irrelevant again, new UFRs are computed. The players do not send any update message until they become relevant again.

Potentially visible set (PVS) [86] is based on the fact that from most viewpoints in the world, only a small portion is visible due to occlusion by opaque surfaces (e.g., walls). The world is partitioned into rectangular cells, the boundaries of which coincide with opaque surfaces (Fig. 11a). Non-opaque portals are identified on cell boundaries, and used to form an adjacency graph connecting the world cells. Each cell can thus determine other visible cells from this cell, based on existing unblocked sightlines between cells. In Fig. 11b, cells B, C, and E are visible to cell A since there exists a sightline from A to each of them.

Fig. 11
figure 11

Potential visibility set

Full size image

Frontier set [81, 83] defines frontier to determine the visibility between cell pairs based on the PVS concept and the message exchange frequency between players residing in different cells based on UFR concept. Given two cells A and B, a frontier consists of two cell sets, F AB and F AB , such that no cell in F AB is visible to a cell in F AB and vice-versa. There exists no frontier between two cells visible to each other. The players residing inside the mutually visible cells exchange messages. No message is exchanged between players residing inside the cells invisible to each other. Therefore, players residing inside cells without any frontiers are visible to each other and thus frequently exchange messages until they become invisible due to their movement. Players invisible to each other do not exchange messages until they cross any frontier and thus become visible again. Fig. 12 shows the frontier of the cell pairs corresponding to players’ positions in the world. No message is exchanged between players in Figs. 12a, b, and d. However, players exchange messages while residing inside visible cells (Fig. 12c).

Fig. 12
figure 12

Frontier sets [83]. Points represent players. a F AI = {A, B, C, F}, F IA = {G, H, I}. b F EH = {A, B, C, E, F}, F HE = {H, I}. c No frontier set since cell D can see cell H. d F DI = {A, B, C, D, E, F}, F ID = {I}. No message exchange between players in a, b, and d. Mutual message exchange in c

Full size image

Although only players visible to each other exchange messages, computing requirements can exceed peers’ capabilities in a VE with a high number of players frequently changing position, since each player should check the frontier set with each other player in the VE, and calculate new frontier sets if necessary.

Donnybrook

Donnybrook [82] is a fully-connected NVE system based on the fact that a person can only focus on a constant number of entities at any moment [87, 88]. Donnybrook thus limits the number of players in which a player is most interested. Players frequently send their state update messages to the players which are interested in them. Besides, players send infrequent state information to other players (i.e., players with less interest in them). A player p is thus frequently informed about the state of players in which p is interested, and infrequently informed about the state of players in which p is less/not interested. To extrapolate the behavior of a remote player q, from which p receives infrequent information, an intelligent bot runs on p’s local machine. This bot approximates q’s behavior through q’s infrequently sent information. In case of a difference between q’s approximated and actual state, the bot approaches smoothly to the actual state. Since p is not interested in q, p is unlikely to notice the difference.

6.2 Discussion

The presented NVE systems based on the fully-connected approach are given in Table 4. In the fully-connected approach, all players in the virtual world are directly connected to each other. The fully-connected approach thus does not scale well with the number of players in the VE due to bandwidth and computing overhead on each peer. In addition, if a large number of peers are interested in (and thus want to directly interact with) each other, the necessary message exchanges can quickly exceed peers’ capabilities.

Table 4 A list of the representative fully-connected NVE systems
Full size table

7 Neighbor-list exchange approach

In P2P systems, using neighbors of neighbors knowledge is a well-known strategy to acquire required information and to make a better routing decision [89]. In neighbor-list exchange-based P2P NVEs, peers exchange their knowledge of the virtual world in order to be better informed of the changes occurring in their surroundings. P2P NVE systems following this approach only consider avatars, the data of which are managed on corresponding user machines.

7.1 Representative systems

This section examines two P2P NVE systems based on the neighbor-list exchange approach, P2P message exchange scheme [90] and Message interchange protocol [91].

7.1.1 P2P Message Exchange Scheme

P2P message exchange[90] is a fully-distributed system in which the VE is supported through regular neighbor-list exchange between peers. Each peer maintains a fixed number of direct connections to the closest neighbors in the virtual world. A peer p’s direct neighbors are called active entities (AE). p is regularly informed by its AE about not only themselves but also their own neighbors. The neighbors of p’s AE are called latent entities (LE). p regularly updates its AE list based on the information received from its AE. Figure 13 shows the neighborhood relationship and message flow for a peer O, in which O’s active entities are A1, A2, A3, A4, and latent entities are B1, B2, …,B8.

Fig. 13
figure 13

P2P message exchange scheme [90]

Full size image

In a world with uncrowded regions, connections to only a fixed number of nearest peers can divide the network into isolated subgroups. [92] is an extension of [90] that proposes a probabilistic solution to the network partitioning problem. To ensure global network connectivity through information exchange between isolated subgroups, peers connect to randomly selected peers at arbitrary time intervals through a bootstrap server and exchange information with those peers.

To provide an efficient solution to the network partitioning problem even in the presence of a high number of entities, [93] replaces the random message exchange scheme proposed in [92] with a DHT-based scheme to reorganize isolated subgroups through message exchange between them. The virtual world is partitioned into regions, each of which is assigned to a manager peer through a DHT built separately from the P2P message exchange protocol. The manager peer of a region keeps information about peers inside its region. Peers thus inform the manager of regions inside which they reside about their position changes. When a peer p belonging to an isolated subgroup inquires the manager of a region, the manager provides p with the list of peers residing inside the region. Then, p asks all peers in the list their current location in the world and updates its AE list based on distance. The network is thus reorganized and subgroups are kept connected. However, this solution requires the management of a DHT structure based on peers’ positions in the virtual world in addition to the P2P message exchange protocol, and thus increases the load on peers. Furthermore, as in exchanged neighbor-list information, the peer list information provided by the manager of a region to an inquiring peer can contain redundant information, especially in case of player crowding in the region, which can create excessive message exchanges.

7.1.2 Message interchange protocol

Message interchange protocol [91] connects each peer to at least four peers, the closest in each of the four quadrants in the virtual world (Fig. 14) in a similar way to Apolo [75] (cf. Section 5.2.2). However, in Apolo, each peer connects to exactly four peers (i.e., the closest peers in each of the four quadrants), which can introduce additional hops to send a message to an AOI neighbor, thus increasing latency.

Fig. 14
figure 14

Message interchange protocol [91]. e connects to at least the closest peers in each of the four quadrants

Full size image

In Message interchange protocol, a peer p keeps information about other peers in three tables: Observable Table (OT), Global Table (GT), and Group Leader Table (GLT). The OT contains information about peers interested in p and to which p regularly sends its update messages. The GT contains information about other peers residing in p’s surroundings, however not directly interested in p; this table is used for accurate AOI neighbor discovery. Peers interchange their GT within a time period depending on the distance between them in the virtual world. Nearby peers interchange their GT more frequently than faraway peers. Besides, the GLT keeps track of the closest peers in each of the four quadrants to ensure global connectivity. Peers periodically updates their GLT and peers having empty OT and GT interchange their GLT to keep the network connected. Message interchange protocol is, therefore, basically based on the exchange of peers’ OT, GT and GLT with their neighbors, while the OT, GT and GLT are used for state update dissemination, AOI neighbor discovery and global network connectivity, respectively.

Compared to P2P message exchange scheme, Message interchange protocol provides better AOI and global network connectivity managements. However, similarly to P2P message exchange, exchanged lists may contain redundant information. Player crowding causes a high number of peers (with long lists) exchanging their list information with each other, the bandwidth requirements of which may exceed peers’ capabilities.

7.2 Discussion

Table 5 shows the neighbor-list exchange based NVE systems presented in this section. In the neighbor-list exchange approach, each peer has limited number of connections independently from the total number of users in the VE. The direct connectivity between peers in relation minimizes the delay of message propagation in the VE. However, as peer connectivity supported by only neighbor-list exchange is not sufficient to ensure global network connectivity, an auxiliary mechanism is thus needed to avoid network partitioning. In addition, constant neighbor-list exchanges, which may contain redundant information for destination peers, introduce a high message overhead.

Table 5 The presented neighbor-list exchange based NVE systems
Full size table

8 Mutual notification approach

In P2P NVE systems based on the mutual notification approach, peers maintain direct connections to nearby peers surrounding them in the VE, usually based on well-known geometrical structures such as Voronoi in [94], Yao graph in [77] or quadtree in [95]. In the event of peers’ position change in the VE, global network connectivity and neighbor discovery are achieved through mutual cooperation and notification among direct neighbors.

8.1 Representative systems

This section examines several P2P NVE systems based on the mutual notification approach, namely, pSense [77], Solipsis‘03 [96], VON [94], Relaxed Triangulation [97] and Red-Black Delaunay [98].

8.1.1 pSense

pSense [77] provides an AOI-based peer connectivity. Each peer keeps information about other peers in its AOI (Fig. 15). To ensure global network connectivity and neighbor discovery, each peer also connects to sensor peers, the closest peers outside its AOI in each of the six sectors of equal angle, similarly to Yao graphs [99]. When a peer p moves in the world, p informs its AOI and sensor neighbors about its position change. If a sensor peer detects a peer q entering in p’s AOI, it informs p about q which becomes p’s new AOI neighbor. If the total number of neighbors exceeds p’s upload bandwidth capacity, p sends its update message to only a randomly selected subset of its neighbors. The neighbors receiving p’s update message then forward the message to their own neighbors residing in p’s AOI. However, in case of player crowding, many duplicates can be produced during message forwarding. Also, peers receiving a message may not have enough bandwidth to forward the message to all their neighbors, leading to a second round of randomly selected (instead of all) neighbors, and thus increasing the delay with which some peers will receive messages.

Fig. 15
figure 15

pSense [77]

Full size image

8.1.2 Solipsis‘03

Solipsis‘03 [96, 101] is a P2P NVE system that proposes a new connectivity structure based on the angular relation between peers’ positions in the virtual world. The virtual world is designed as a two dimensional torus to provide an unbounded surface. Each peer maintains direct connections with the peers residing in its AOI. Neighbor discovery is achieved through the notification of neighbors. The neighbors of a peer should always form a convex hull (Fig. 16), which guarantees global network connectivity. If a peer detects a convex hull violation through recursive collaboration with its direct neighbors, the peer establishes new connections until the convex hull condition is satisfied (Fig. 16). In the event of position change, new neighbors are discovered through mutual cooperation and notification among direct neighbors. When a peer wants to move to a specific position in the virtual world upon login or by teleportation, the peer’s request is transmitted from the requesting peer to the peer closest to the specified position (Fig. 16).

Fig. 16
figure 16

Solipsis‘03 [96]

Full size image

However, Solipsis‘03 might lead to inconsistencies where some neighboring peers go undetected [100]. Figure 17 presents such a situation where e and e f are not connected although e f is residing inside e’s AOI.

Fig. 17
figure 17

Solipsis‘03 - Undetected neighbor [100]

Full size image

8.1.3 VON

VON [94, 102] proposes a distributed NVE system similar to Solipsis‘03 [96]. VON mainly replaces the geometrical structure providing peer connectivity in Solipsis‘03 with a Voronoi diagram [103] to resolve the undetected neighbor problem in Solipsis. The virtual world is partitioned using Voronoi based on peers’ positions in the virtual world. Each peer maintains a Voronoi diagram of its AOI neighbors, and keeps a direct connection to all of them. Figure 18 presents the Voronoi-based partitioning of the virtual world according to the participants’ position and the neighborhood relations between them. Each peer keeps connections with its enclosing neighbors whose Voronoi regions have a common edge with its own Voronoi region (even if they reside outside its AOI). The outermost peers inside the AOI of a peer are called boundary neighbors, and are responsible for neighbor discovery. VON thus optimizes the mutual notification between peers compared to Solipsis‘03 by reducing the number of peers participating in the neighbor discovery from all AOI neighbors to only boundary neighbors. VON provides limited number of connections between peers independently from the total number of users in the VE, and guarantees global network connectivity since peers are always connected to their enclosing neighbors even though these neighbors reside outside peers’ AOI.

Fig. 18
figure 18

Neighbors in VON [94]

Full size image

In the event of player crowding, however, mechanisms that connect a peer to all its AOI neighbors might lead to situations where almost all peers become connected to each other due to overlapping AOIs. To deal with this issue, AOICast [76], built on VON, proposes to construct a spanning tree over all AOI neighbors of a peer, thus reducing bandwidth usage at the cost of a higher latency.

VON however considers peer movements under the assumption that when a peer moves in the virtual world, all other peers surrounding the moving peer remain stationary. This assumption is needed so that the surrounding peers can be used as reliable landmarks to update the underlying overlay structure according to the peer’s movement. In addition, the layout of peers in the VE might overload some peers by forcing them to make direct connections beyond their capacity. For instance, when in the VE a high number of peers are lying on a circle inside the center of which a peer p resides, then the peers on the circle become p’s enclosing neighbors and p should manage the direct connections to all its enclosing neighbors which may exceed p’s bandwidth capabilities [104] (Fig. 19). Also, in a highly dynamic environment, particularly in the context of P2P NVEs, the underlying Voronoi structure produces high maintenance cost as it is subject to high connection change rate due to continuous players’ movement. As players tend to move in the VE, the network topology should continuously adapt to players’ movements such that the overlay structure is kept valid. Maintenance cost can become particularly high when crowding occurs, and a highly dynamic interaction with the surrounding takes place.

Fig. 19
figure 19

VON - Peer connectivity in the worst case [104]

Full size image

8.1.4 Relaxed triangulation

To deal with high maintenance cost related to the underlying overlay in the mutual notification approach, Relaxed Triangulation (RT) [97] proposes a new triangulation algorithm to connect peers based on their positions in the virtual world. Relaxed Triangulation (RT) for a set of vertices V in a 2-D plane is defined as a triangulation T(V ) such that no vertex of V is inside any triangle in T(V ). In order to minimize the maintenance cost resulting from high connection change rate due to user movements, RT maximizes the flip-free area of a node, namely the area inside which the node’s movement does not trigger any edge flip operation. To this end, RT defines the flip-free area of a node as the region composed of all the triangles of which the node is a vertex in RT. A flip operation thus occurs only when the node passes across the base line of one of its triangles. The gray area in Figs. 20a and b shows point 0’s flip-free area in Delaunay and in RT, respectively. RT provides overlay network connectivity to support P2P NVEs while decreasing maintenance overhead by reducing the number of connection changes due to users’ movement. However, similar to VON, RT considers peer movements under the assumption that when a peer moves in the virtual world, all other peers surrounding the moving peer remain stationary.

Fig. 20
figure 20

Node 0’s flip-free area [97]

Full size image

8.1.5 Red-Black Delaunay

Red-Black Delaunay [98] proposes a distributed NVE system based on a Delaunay graph [105] of peers’ positions in the virtual world. Red-Black Delaunay constructs the same overlay structure as VON [94]. Red-Black Delaunay focuses on the construction of a valid Delaunay graph considering several concurrent peers’ movements and clearly defines the groups of moving peers and of the stationary peers used to update the overlay structure. Peers in the virtual world are randomly grouped into two sets, called red and black. For each set, a separate Delaunay subgraph is constructed based on the position of its peers in the virtual word (Fig. 21). A peer is directly connected not only to its same colored Delaunay neighbors, but also to a group of different colored peers, called parents, which are responsible of updating the neighbor connections of the peer during its movement. A peer p in a subgraph becomes the parent of a peer q in the other subgraph, if p’s Voronoi region in p’s corresponding subgraph intersects with q’s Voronoi region in q’s corresponding subgraph. If the peers in a subgraph move in the virtual world in times T, 2T, 3T,…, then the peers in the other subgraph move in times T/2, 3T/ 2,5T/2,…. Therefore, alternately, when the peers in a subgraph change position in the virtual world, the peers in the other subgraph remain stationary and update the subgraph of the moving peers by advising them about connection changes in their subgraph. The overall Delaunay graph including all peers in the virtual word is then reconstructed based on the red and black subgraphs.

Fig. 21
figure 21

Red-Black Delaunay [98]

Full size image

Red-Black Delaunay constructs a valid Delaunay structure that deals with peer movements based on the sequential synchronized movement of peers in a subgraph (with respect to peers in the other subgraph) and update of the moving subgraph by the stationary peers in the other subgraph. The feasibility of the proposed scheme strictly depends on the sequential movement and update phases of each subgraph. In addition, in the event of crowding of the same-colored peers, the parent peers are likely to become overloaded due to the update requirements of their subgraph.

8.2 Discussion

The mutual notification-based NVE systems presented in this section are given in Table 6. Mutual notification can be considered as a more specific approach of neighbor-list exchange. In the neighbor-list exchange approach, peers exchange their knowledge of the VE through regular neighbor-list information exchanges regardless of the necessity of this information on the receiver side. On the other hand, in mutual notification, peers cooperate with each other on a specific task (e.g., neighbor discovery, global network connectivity) only when necessary. Since neighbor-list information might contain irrelevant and redundant data, mutual notification provides a more bandwidth efficient solution. High scalability is achieved since peers maintain a limited number of connections with other peers based on the underlying geometrical structure independently from the total number of peers in the VE. The direct connectivity between peers in relation minimizes the delay of message propagation in the VE. However, the underlying structure might cause high maintenance cost as it is subject to high connection change rate due to the continuous movement of players. The management of several concurrent peers’ movements remains an open issue. Moreover, the layout of peers in the VE might overload some peers by forcing them to make direct connections beyond their capacity.

Table 6 The representative mutual notification based NVE systems
Full size table

9 Discussion and future perspectives

Table 7 summarizes the different approaches used to construct some of the P2P NVE systems presented in this paper, while Table 8 summarizes the characteristics of the different approaches discussed in this paper for the construction of P2P NVE systems. Overall, the world partitioning and fully-connected approaches may suffer from bandwidth consumption issues, while the DHT approach may introduce high latency, and the neighbor-list exchange approach does not guarantee global network connectivity. The mutual notification approach supported by the multicast approach for message diffusion and the world partitioning approach for the management of less dynamic entities (in terms of position changes) can provide global network connectivity, low latency due to direct connectivity between peers in relation and distributed bandwidth load among peers. However, given the high dynamism of VEs in terms of position changes, rigid overlay structures (typified by the Voronoi diagram) introduce a high overlay maintenance cost.

Table 7 P2P NVE systems’ characteristics
Full size table
Table 8 P2P NVE approaches’ characteristics
Full size table

From the above, and despite their several advantages, P2P designs for NVEs still suffer from several weaknesses that are hindering their adoption by the NVE market. However, we believe that the main reason behind the non-adoption of P2P technologies for the support of NVEs is the difficulty in providing a high level of availability and trust. In an open system where nodes join and leave at will, dealing with failures, disconnections, and churn requires a great degree of redundancy (and its associated consistency maintenance cost) to increase service availability. The trust issue is even more challenging in open environments. In client-server architectures, every user trusts at least the server. However, such trust may not exist in a P2P setting, making security issues very difficult to solve without centralized control. Although several works have been lately conducted to provide solutions capable of establishing a higher level of availability and trust in P2P NVEs [106–110], these efforts are still at their early stages.

However, beyond the very specific characteristics, advantages, and drawbacks of P2P NVE approaches, it is prudent to take a step back and observe from a global perspective the place that a decade of intensive scientific research on this topic can have in today’s technological landscape. Given current skyrocketing interest in cloud computing as a means of dynamic provisioning of (virtually infinite) resources, one might wonder whether the P2P technology still has a role to play in today’s computing setting. In other words, the following important question arises: Is there one distributed computing paradigm that will define future NVE architectures, or will there be a place for multiple paradigms, including P2P?

We strongly believe that no one distributed computing approach will be able to meet the requirements of NVE applications. Just like its client-server and P2P counterparts, cloud computing presents several limitations in the context of NVEs with strong real-time interactivity requirements. Currently, two major cloud gaming models are emerging, with each having its advantages and drawbacks. In the first model, the entire game software is hosted in the cloud, where heavy processing and graphics rendering are executed and sent back as video streams to users accessing the game through thin clients [111, 112]. This gaming model opens NVEs to a much wider user population by making the capabilities of users’ machines unimportant. However, streaming high quality interactive video to clients, especially at a massive scale, is confronted with bandwidth limitations and cost as major bottlenecks. Also, user quality of experience relies solely on Internet connection quality, including upload/download speeds, connection stability, as well as latency. Also, traditional delay compensation techniques, such as dead reckoning, used on the client side are no longer possible when the complete game software is run remotely in the cloud. This is a critical issue in an Internet environment where latencies are not predictable. The second cloud gaming model avoids the above-mentioned issues by hosting in the cloud only the game server software running the game logic and handling players’ interactions, while keeping graphics rendering on clients’ machines. This, however, makes high-end computers or particular hardware necessary to process the client software for the game.

Beyond the advantages of one over the other, both cloud gaming models face the following major challenge: how to best exploit the elastic character of cloud computing to answer the strong scalability requirements of NVEs.

Current solutions deal with the scalability issue through rigid parallelization techniques of players’ interactions. The main technique is sharding, which involves creating several completely separate instances, or shards, of the virtual world, and assigning each player to a single shard based on some criteria, such as players’ geographical regions or skill levels. However, this goes against the idea of a “single unified virtual world” that all users connect to, with the potential of interacting with millions of other users worldwide instead of being limited to users of the same world instance.

In light of the above discussion, we strongly believe that alternative parallelization techniques of players’ interactions are required to best exploit the elastic capabilities that cloud infrastructures can offer. In particular, the game server software should be engineered in a way that allows the server to elastically stretch and shrink its use of cloud resources. In this respect, P2P technologies are extremely relevant in their extraordinary capacities for self-organization and self-healing in dynamic environments where peers (and their associated resources) continuously join and leave the network. We note in particular that:

  • The world partitioning approach has a huge potential for implementation in data-centers. Many of the algorithms that have been described in Section 3 could significantly enhance the rigid sharding process which is implemented today. Flexible area management and load-balancing among area managers are among the most attractive algorithms.

  • The DHT-based approach has high potential given that its main drawback in the context of P2P systems –high latency– is no longer an issue in data-centers. Similarly to Facebook’s Cassandra system that is based on a customized DHT, we envision promising research activities in the customization of the previous works described in Section 4 in the context of data-centers.

  • The mutual notification and neighbor-list exchange approaches are in our opinion less attractive. Though, an architecture where each avatar runs on a virtual machine in a data-center makes sense. Most of the previous works described in Section 8 focus on hypotheses that are not accurate in data-centers, namely the absence of central authority and peer churn, and are thus not directly applicable to data-centers.

Another argument in favor of a solution based on both cloud and P2P computing relates to the game streaming model (i.e., the first cloud gaming model discussed above), where latency and bandwidth consumption are the key bottlenecks. Given that P2P solutions have shown to be very effective for content streaming, we believe that an interesting solution would be to augment the cloud resources with a P2P streaming solution where the streaming load can be offloaded to the P2P network, thus leveraging peers’ bandwidth and reducing latency.

Finally, we would like to emphasize that the libertarian motivations for building authority-less NVEs is still viable. Even if this libertarian motivation is less attractive from an economical perspective, we believe that there is a need to perform research with this objective in mind. The recent proliferation of projects related to peer-to-peer social networks (for example [113]), and the fear that current dominant web giants are provoking, emphasizes the critical need for research efforts on this area.

10 Conclusion

This paper analyzes the key design approaches to construct a P2P NVE system. We discuss each approach, its advantages and drawbacks, and examine several representative P2P NVE systems based on the presented approaches. Despite all the nice characteristics of the P2P approach, we believe that a solution that provides all the necessary requirements of NVE systems cannot be achieved following a pure P2P design choice. However, we argue that future NVE architectures will build on the strengths of several distributed computing paradigms to overcome the shortcomings of adopting only one. In particular, a very promising research direction would be to look into hybrid solutions where the self-organizing and self-healing nature of P2P approaches can be exploited on top of cloud resources to best exploit the elastic capabilities of cloud computing.