1 Introduction

In recent years, there is a rapid development of Internet of Things (IoT). It is predicted that the number of IoT equipment will reach over 75 billion by 2025 [1]. Such a large number of connected equipment will generate a massive amount of data. To deal with the placement problem of data, the traditional way is to transmit all of them to data centers for storage and processing [2]. However, data centers in the cloud often fail to process and store the data generated by a massive number (billions) of distributed IoT devices [3] and could cause the problem of high latency, network congestion, poor quality of Service, etc. [3]. To deal with the problem mentioned above, Cisco proposes the concept of Fog Computing [4], which engages the Fog nodes for data storage and processing. Fog nodes are the type of network equipment which is close to the users, e.g. routers, switches, set top boxes, proxy servers, Base Stations(BS), etc. [5]. Nowadays, more and more applications (e.g. intelligent surveillance, smart cities, wireless sensor networks, etc) require to store, process and obtain the data with extremely low latency and the introduction of Fog Computing could achieve this requirement. With the proper utilization of the computing and storage capacity of Fog nodes, the overall data transmission latency will drop down significantly because typically Fog nodes are closer to the IoT equipment than data centers. Hence, how to properly place the massive amount of data generated by IoT equipment on fog nodes which have different storage capacity in order to minimize the overall transmission latency for Fog-enabled data services is the main research topic in this work.

In the area of Cloud Computing, [6, 7] have proposed some data placement strategies. However, those strategies both tend to place data replicas on the Cloud. Compared with data centers, Fog nodes are more widely distributed in network topology, larger in number and more limited in storage capacity. As a result, the strategies proposed in [6, 7] cannot be applied directly in the field of Fog Computing. To the best of our knowledge, only in [8, 9], some data placement strategies in the field of Fog Computing were proposed. However, those strategies only focus on the data placement problem of placing one single data replica to the proper Fog node. Under the situation that there are several data consumers whose topology positions are different subscribing the same data, one single data replica cannot meet the latency requirement of all the consumers. Hence, on the basis of [8, 9], we improve the data placement model to support multiple data replicas placement and the overall latency is significantly reduced compared to the single data replica strategies. The multi-replica model proposed in this paper is a MILP model which has been proved to be NP-hard for solving. It will take an unbearable amount of time to solve the model by using mathematical programming optimizer like CPLEX [10]. In order to solve the model efficiently, we design a few heuristic rules and propose a latency-aware strategy to dramatically reduce the searching space for solving the target model. We have conducted the extensive experiments. The results show that the solving time of the model is significantly reduced and the scheduling decision can be made at real time.

In summary, this paper makes the following contributions:

  1. 1.

    Under the multi-replica storage architecture for Fog Computing, we propose a latency-aware model, iFogStorM, which can support the multiple data replicas placement.

  2. 2.

    We analyze the complexity of iFogStorM and further develop a heuristic data placement strategy, MultiCopyStorage, to solve the model efficiently even when the problem scale is large.

  3. 3.

    We implement and integrate our latency-aware strategy into iFogSim [2], and evaluate the performance of MultiCopyStorage with extensive tests. The results show that compared to the existing strategies, our strategy can achieve much better performance in terms of reducing data transmission latency and execution time.

The rest of this paper is organized as follows. Section 2 discusses the related work. Section 3 formulates the multiple data replicas placement problem and constructs the latency-aware model, iFogStorM. Section 4 proposes the heuristic algorithm to solve the model. After evaluating the performance of our strategy in Section 5, we summarize our conclusions and points out the future work in Section 6.

2 Related Work

Many research efforts were done to propose some module placement strategies. In [11], Mahmud et al. propose a latency-aware heuristic module placement strategy. According to the subscription relationship between modules and the tolerable delay, the modules should be placed to the North end (near the cloud) as far as possible without violating the maximum tolerable delay. Furthermore, they propose a module forwarding strategy in order to optimize number of computationally active Fog nodes and reduce energy consumption. In [12], Taneja et al. abstract the subscription relationship between modules into a directed acyclic graph (DAG), and propose a placement strategy to maximize the utilization of computing resources of fog nodes. In [13], a method based on fuzzy theory [14] for fractional evaluation of modules and fog nodes is proposed, and then a matching model between modules and fog nodes is established. Finally, the model is solved by SCIP [15], an integer programming solver. In [16], Skarlat et al. propose the concept of fog colonies, which abstracts multiple fog nodes into a cluster, and then propose a module placement model considering module response time, CPU, memory and storage resources required by the module. On the premise of meeting the corresponding constraints, the number of modules placed in fog nodes was maximized. Sun et al. [17] also combine multiple fog nodes into clusters and proposes a two-step scheduling scheme: 1) resource scheduling among various fog clusters. 2) resource scheduling among fog nodes in the same fog cluster. For step 2, a multi-objective placement model is proposed to meet the minimum completion time of the module, and two objectives are proposed: 1) minimize the service latency of the application and 2) maximize the stability of the application. And a NSGA-II [18] based strategies applied to the multi-objective optimization of the model.

However, due to the heterogeneity of fog nodes, the computing capacity and storage space of fog nodes are often asymmetric, and the research mentioned above does not consider the storage of data. Considering the case that data is stored in a fog node with storage space and then forwarded to the consumption service (module) which subscribes to the data, we formulate the data placement problem.

In order to reduce the overall latency of data transmission, many solutions have been proposed recently. In [19], Wang et al. propose a new architecture for computation and storage offloading which can reduce the communication cost and latency. In [8], Naas et al. propose a data placement model of data production, storage and consumption. The data can be produced by sensors or modules, and the produced data is sent to a storage node (host) in the network topology for storage and then forwarded to consumption services (modules) which subscribe to the data. To achieve the model proposed in [8], CPLEX MILP [10] is used for accurate solution, and a heuristic graph partitioning-based strategy, iFogStorZ, is proposed to reduce the complexity of the problem. iFogStorZ defines Regional Point of Presences (RPOPs) as points of partitioning but it does not consider the data flow between different parts after partitioning. In [9], based on the model designed in [8], a new graph partitioning method, iFogStorG, is proposed, which balances the complexity of various parts after graph partitioning and minimizes the data flow across parts. This strategy improves the accuracy of model solution to a certain extent. However, after graph partitioning, all parts are still using CPLEX MILP [10] for problem solving. Under large-scale scheduling, the solving speed is not ideal for real-time scheduling. In addition, both iFogStorZ and iFogStorG are based on the model proposed in [8]. The data produced by the data producer is stored on only one single storage node (host). When the corresponding number of data consumers is large and geographically distributed, a single replica cannot meet the latency requirements of all consumers.

In addition, many scholars have proposed some data placement strategies for multiple replicas in distributed environments such as big data computing. In [20], based on the network topology and the storage load of each node, a multiple replicas data placement strategy is proposed on the Hadoop platform, which ensures load balancing and data transmission performance. In [21], Wu et al. propose a multiple data replicas placement strategy under the tree network, which significantly reduced the cost of the entire tree network. In [22], Lizhen et al. propose a genetic algorithm based data replica placement strategy for scientific applications in Clouds. The experiment result shows that when the number of replicas changes from 1 to 2, the data transfer time decreases by nearly 50%. In [23], Rajaretnam et al. propose a dynamic multi-copy placement strategy for data grid. According to the request frequency of data, the number and the locations of data replicas are dynamically adjusted. However, to the best of our knowledge, no other research has proposed multiple data replica placement strategies in the Fog Computing environment.

Therefore, we propose the multiple data replicas placement model, iFogStorM, in view of the deficiency of the model above. A data producer can send data to multiple Fog nodes for storage at the same time, and consumers who subscribe to the data can choose any one of the storage nodes to get the data. In order to achieve the model, we propose a greedy algorithm based strategy, MultiCopyStorage, which can obtain the nearly optimal solution under the speed requirement of real-time scheduling.

3 System Modeling

As shown in Fig. 1, we assume that our system architecture is composed of a certain number of Fog nodes (consist of Region Point of Presences (RPOPs), Local Point of Presences (LPOPs) and Gateways), data centers, IoT equipment such as sensors and a set of services. A Region Point of Presence (RPOP) covers a geographical zone in ISP infrastructures and a Local Point of Presence (LPOP) is assigned to a Remote Point of Presence (RPOP) in the hierarchical system. Services can be deployed on any of the data centers, Fog nodes or IoT equipment. Generally, the service on IoT equipment will generate the raw data, which will be processed by the services on Fog nodes. Services can be divided into two types, data producers, denoted by DP = {dp1,  … , dpi,  … , dpn}, and data consumers, denoted by DC = {dc1,  … , dck,  … , dcq} (note that a service can be both producer and consumer). Every Fog node which has storage capacity can be the data hosts, denoted by DH = {dh1,  … , dhj,  … , dhm}. All the data produced by data producers must be stored on at least one of the data hosts and it takes a certain amount of time to transmit those data to the data hosts (in single replica placement model proposed in [8], the produced data is only stored on one data host). The data produced by data producer dpi is denoted by datai.Note that if the services are the producers but not deployed on sensors or other IoT equipment which can produce the data themselves, they must have the consumption relation with other services (i.e., these services are consumer services too and consume the data transmitted from other services). Each consumer has a subscription relationship with the data generated by the producer, which is denoted by pcik. A simplified multi-replica data flow example is shown in Fig. 1. Note that every service deployed on IoT equipment is a producer (we do not mark them so in the figure for the sake of clarity).

Figure 1
figure 1

Architecture of multi-replica Fog storage system.

Under the above architecture, the model named iFogStorM is proposed, aiming to solve the multi-replica data placement problem (see the main notation in Table 1). Before presenting the model in detail, the following constraint conditions are first given.

Table 1 Notation Dictionary.
  1. 1)

    cijk denotes that the data consumption relation, i.e., if dck obtains (consumes) datai from dhj then cijk is set to 1 else cijk is set to 0. There is at least one replica of datai being placed on the host nodes:

$$ {\sum}_j^m{\sum}_k^q{c}_{ijk}\ge 1\kern0.5em \forall i $$
(1)
  1. 2)

    xij denotes that a replica of datai placed on dhj.In the case where there exists a consumer obtaining datai from dhj, there must be a replica of datai placed on dhj:

$$ {x}_{ij}=\Big\{{\displaystyle \begin{array}{cc}1,& {\sum}_k^q{c}_{ij k}\ge 1\\ {}0,& {\sum}_k^q{c}_{ij k}=0\end{array}} $$
(2)

And (2) is equal to (3), (4), (5)

$$ {x}_{ij}\ge {c}_{ij k}\kern0.5em \forall i,\forall j,\forall k $$
(3)
$$ {x}_{ij}\le {\sum}_k^q{c}_{ij k}\kern0.5em \forall i,\forall j $$
(4)
$$ {x}_{ij}\epsilon \left\{0,1\right\}\kern0.5em \forall i,\forall j $$
(5)
  1. 3)

    storj denotes the storage capacity of dhj and si denotes the size of datai.The total storage usage of all the data replicas placed on dhj must be smaller than the storage capacity:

$$ {\sum}_i^n{s_i}^{\ast }{x}_{ij}\le sto{r}_j\forall j $$
(6)
  1. 4)

    pcik denotes the subscription relationship between dck and datai.When pcik = 1, which means that there is a subscription relationship between dck and datai, there must be at least one replicas of datai placing in the fog nodes and dck choose one of the fog nodes which has replica of datai to obtain the data .Thus, in this case, one of the cijk when j ∈ m is equal to 1 .When pcik = 0, dck needn’t to obtain datai, so in this case, all of the cijk when j ∈ m is set to 0.

$$ {\sum}_j^m{c}_{ijk}=p{c}_{ik}\kern0.5em \forall i,\forall k $$
(7)
  1. 5)

    lrjk denotes the time required for dck to obtain b from dhj and lwji denotes the time required for dpi to transmit b to dhj. trijk denotes the time required for dck to obtain datai from dhj and tsji denotes the time required for dpi to transmit datai to dhj.The relationships between them are:

$$ t{r}_{ijk}=\frac{1}{b}\cdot {s}_i\cdot l{r}_{jk} $$
(8)
$$ t{s}_{ji}=\frac{1}{b}\cdot {s}_i\cdot l{w}_{ji} $$
(9)

We define the overall latency as the sum of the required time of storing data on the data host and the time of transferring them to the data consumers. Note that the more replicas are placed on fog nodes, more required time is needed to store data while probably (not definitely) the less time of transferring them to the consumers is needed because the superfluous replicas which are not data providers of any data consumers would not be beneficial to decrease the time of data consumers to achieve the data they need. Since there is a negative correlation between the data transmitting time from data producers to data hosts and that from data hosts to data consumers, there is a trade-off between them and that’s the reason we set overall latency as the scheduling objective. Moreover, the number of replicas of a piece of data is affected by the number of consumers that subscribe the specified data (which is denoted by cp) and in general, the bigger cp is, the more replicas would be needed to achieve our goal of minimizing the overall latency. The required time is calculated according to the minimum latency between nodes as shown in (8) and (9). The problem can be modeled as follows:

$$ \mathit{\operatorname{Minimize}}\sum \limits_i^n\sum \limits_j^m\left(\sum \limits_k^q\left({c}_{ij k}\cdot t{r}_{ij k}\right)+{x}_{ij}\cdot t{s}_{ji}\right) $$
(10)
$$ s.t.\kern0.5em \sum \limits_j^m\sum \limits_k^q{c}_{ijk}\ge 1\kern0.5em \forall i $$
(10a)
$$ {x}_{ij}\ge {c}_{ij k}\kern0.5em \forall i,\forall j,\forall k $$
(10b)
$$ {x}_{ij}\le \sum \limits_k^q{c}_{ij k}\kern0.5em \forall i,\forall j $$
(10c)
$$ \sum \limits_i^n{s_i}^{\ast }{x}_{ij}\le sto{r}_j\forall j $$
(10d)
$$ \sum \limits_j^m{c}_{ijk}=p{c}_{ik}\kern0.5em \forall i,\forall k $$
(10e)
$$ {c}_{ijk}\epsilon \left\{0,1\right\}\kern0.5em \forall i,\forall j,\forall k $$
(10f)
$$ {x}_{ij}\epsilon \left\{0,1\right\}\kern0.5em \forall i,\forall j $$
(10g)

4 MultiCopyStorage: A Greedy Data Placement Strategy

4.1 Overview of the Greedy Data Placement Strategy

Similar to the data allocation (or resource management) model proposed in [24,25,26], iFogStorM can’t be solved in polynomial time and it is a MILP model. Thus solving the formulated model is NP-hard. It will take an unbearable amount of time to solve the model by using the mathematical programming optimizer such as CPLEX [10]. In order to solve the model efficiently, in this paper we design a few heuristic rules and propose a latency-aware strategy to dramatically reduce the searching space for solving the target model. In particular, the greedy strategy MultiCopyStorage, which is based on iFogStorM model, is proposed in this section. MultiCopyStorage is a greedy strategy for solving the problem of data placement in the Fog Computing area, which can greatly improve the solving speed while maintaining the quality of the solution when the problem scale is large.

In the iFogStorM model, after determining the placement of all the replicas, it is also necessary to determine which host the consumer obtains the data from. In the MultiCopyStorage strategy, a heuristic rule is added: all consumers get the corresponding data from the host that stores the subscribed data replica with the lowest latency, so as to reduce the scope of searching for good solutions. It is a greedy algorithm in essence. In turn, every piece of data is placed in the local optimal placement solution with the lowest overall latency. After obtaining all the local placement solutions of every piece of data, the final solution is obtained. The procedure of finding the local optimal solution for every piece of data is a recursive procedure. Starting with the case of one replica, the recursion is carried out level by level until the latency cannot be reduced even if the number of replicas increases. Moreover, a few heuristic pruning strategies are used to control the number of recursion levels.

In summary, the following heuristic rules are applied when solving the model:

  1. 1.

    The consumers select the host which holds the replica of data that they subscribe to and offers the least latency as the node to obtain the subscribed data from.

  2. 2.

    When the number of replicas is higher than the number of consumers for that data, it deemed impossible to have the optimal solution with the minimum overall latency because of the extra write latency of the replicas that not being used by any of the consumers.

  3. 3.

    In every recursion level, only the nodes that reduce the overall latency after adding to the temporary solution are considered as the candidate nodes for the next recursion level.

4.2 The Algorithm Details

In this subsection, we present the greedy algorithm, MultiCopyStorage, and discuss it in detail.

MultiCopyStorage is shown in Algorithm 1. In the algorithm, for each data that needs to be placed, its candidate nodes are initialized to be all host nodes (line 4), and then the recursive procedure (line 6) is performed in order. After each recursive solution, the idle capacity of the host where the data is placed is updated (lines 7-9), and then the placement result of this data is added to the final result (line 10).

The recursive process uses the RecursiveSolve() function to solve the problem. When the number of recursive layers (which indicates the number of replicas) becomes more than the number of consumers subscribed to each data, adding more replicas of the data will only increase the write latency and will not reduce the read latency. Therefore, returning the current placement and latency cost (lines 1-3) is a better choice. In lines 7-17, we try to choose among candidate hosts the best node to place a new replica of the data on. To achieve this, we calculate the latency of different placement situations (line 10), add the host to the set of candidate hosts for next recursive level (line 12), and record the placement solution with the minimum latency (line 13-14). If there is at least one placement solution which offers lower latency than the currently best solution, the placement solution with the minimum latency is recursively passed to the next level (lines 18-21). The best placement solution and its minimum latency at the current level are updated after the recursion is completed (lines 22-25).

In order to clarify our algorithm, we illustrate the RecursiveSolve() procedure which is the core-part of the algorithm in detail. At first, the candidates are initialized to all the data hosts in the system architecture. After initialization, we iterate all the candidates and calculate the score of temporary solutions when adding the corresponding candidate to the data hosts of the to-be scheduled data. Then we record the solution whose score is the smallest and add the solutions which are lower than the minimum score of the previous recursion level to the next candidates set. Note that adding a replica would cause the increase of transmitting time from data producers to data hosts (let’s call it write latency), so there must be several consumers which originally get data from another replica in the previous recursion level changing their original data provider. They decide to obtain data from the newly added replica because it is closer. And the increase of write latency can be offset by reducing read latency for several consumers (time for transmitting from data hosts to data consumers). In other words, the new added replica “steal” the consumers from other replicas. And for those replicas fail to “steal” enough consumers from other replicas in the current recursion level, they can’t be able to offset the write latency in the next recursion level too. Thus, some hosts which have a possibility to help obtaining a lower score are put into the next candidates set. And those not in next candidates set would not be considered in the next recursion level. After the iteration of candidates set, we greedily take the temporary solution with minimum score as currently best solution and go into the deeper recursion level in order to find out that if the score could be lower when the number of replicas is bigger.

figure ffigure f

The latency is calculated by the function CalculateScore(). It calculates the write latency in lines 4-6 and the read latency in lines 7-15. When calculating the read latency, the algorithm traverses the consumers, calculates the read latency from the consumer to the nearest host, where the data replica is placed and the corresponding time is added to the overall latency.

Assuming that cp denotes the number of consumers that subscribe to the same data, and n, m and q respectively represent the number of producers, storage nodes and consumers. The time complexity of the function, CalculateScore(), which calculates the overall latency of every solution is cp ∙ m. The function RecurceiveSolve() calls calculateScore() m times in each recursive level. The max number of recursive levels is cp. Therefore, the time complexity of this method is cp2 ∙ m2.Finally, the main function, MultiCopyStorage (), must call RecurceiveSolve () for each data. The total number of calls is n. Thus, the time complexity of the algorithm is n ∙ cp2 ∙ m2.

With the exhaustive search method, the time complexity is non-polynomial. Assuming that in the extreme case, the number of replicas of every piece of data is from 1 to m and those replicas can be placed on every Fog node. Then, there are \( n\bullet \sum \limits_l^m{C}_m^l \) different placement solutions, and \( n\bullet \sum \limits_l^m{C}_m^l \) is approximately equal to n ∙ (2m − 1). As a result, the exhaustive search method would not be an efficient method and the time it takes would exponentially increase with the growth of m.

5 Simulation and Performance Evaluation

5.1 Experimental Settings

In order to verify the validity of MultiCopyStorage strategy, the extended iFogSim [2] is used for the simulation experiments. iFogSim is a simulation platform for simulating fog computing environment, which is extended in [27] by Naas et al. to support the implementation of data placement strategies in the fog computing environment. The fog nodes consist of gateways(GW), Local PoPs(LPOP) and Regional PoP(RPOP), forming the simulated network topology together with data centers and sensors.

1000 GWs, 50 LPOP, 10 RPOP and 5 data centers were used in the simulation environments. The packet size produced by the services deployed on IoT equipment and by other services are set to SI and SO, respectively. Each GW is connected with Ns sensors, which sends a packet every tp milliseconds. Consumers that consume data from sensors send a packet after receiving and processing Np packets while other consumers send one packet once they have finished processing one packet from their data providers.

Table 2 shows the default values of the experimental parameters.

Table 2 Experimental Parameters.

Table 3 shows the network latency between different Fog nodes while Table 4 shows the capacity of Fog nodes. The evaluation experiments are conducted on a 16-core CPU and 32GB memory machine.

Table 3 Network Latency.
Table 4 Data Host Storage Capacity.

Two types of different workload are used in the evaluation:

  1. 1.

    Zoning workload: the consumers are situated in the same geographical zone as the producers which produce the data they subscribe to. A zone is a covered by at least one RPOP.

  2. 2.

    Distributed workload: every service, wherever they are located, may be the consumer of the data.

The simulation time is set to be Tsim. The number of consumers that subscribe to the same data is cp. In order to evaluate the overall latency, we conduct the experiments to compare the MultiCopyStorage strategy with CloudStorage [8], ClosestNode [8], iFogStor [8], iFogStorZ [8] and iFogStorG [9] when cp = 1,5,15,30,50. The CloudStorage strategy [8] is used to store all data replicas on the cloud computing center while ClosestNode [8] to place all data replicas on the nearest storage node. In order to test the running speed of the strategy, we test the running time of iFogStor [8],iFogStorZ [8] and iFogStorG [8] under cp = 5, 30, 50 and GWs = [500,3000] and obtain the average value. We have also utilized other optimization toolkits (e.g. Lingo [28]) to test the running speed of iFogStor. We figure out that the running speed of Lingo is almost the same as Cplex [10] to solve the MILP model and they can both precisely obtain the optimal solution. Since Cplex provides a usable java interface while Lingo doesn’t, in the latter experiments, we only utilize Cplex as the optimizer for iFogStor [8],iFogStorZ [8] and iFogStorG [8].

5.2 Experimental Results

In the experiments regarding the overall latency, the latency of data transmission is added to the current value of overall latency after each experiment run is completed. The solving time is obtained by taking the difference between the start time and the end time of each experimental run. The experiment results are shown in Figs. 2, 3, 4, 5.

Figure 2
figure 2

Overall latency of Different strategies on Distributed workload.

Figure 3
figure 3

Overall latency of Different strategies on Distributed workload.

Figure 4
figure 4

Overall latency of Different strategies on Zoning workload.

Figure 5
figure 5

Overall latency of Different strategies on Zoning workload.

  1. 1)

    Overall latency: Fig. 2 shows that the latency achieved by MultiCopyStorage with distributed workload is 63.7% lower on average than that by the CloudStorage strategy, and 36.8% lower than the ClosestNode strategy. Figure 4 shows that the latency of the MultiCopyStorage strategy with zoning workload is lower by 69.1% and 40.0% than that of CloudStorage and ClosestNode, respectively. It can be seen that compared with CloudStorage, both ClosestNode and MultiCopyStorage reduce the overall latency significantly. The reason for this is because in CloudStorage, the data need to be transferred back and forth between the producers and data centers. Since the producers and the consumers often have a non-negligible number of network hops to the data centers, it is a better decision to place the data on the Fog nodes, which are closer to producers and consumers.

Figure 3 shows the overall latency of different strategies on distributed workload. When cp = 15, 30 and 50, the latency of MultiCopyStorage drops by 1.94%,6.23% and 9.16%, respectively compared with iFogStor and drops by 7.63%, 11.81% and 14.57%, respectively, compared with iFogStorG(5). One can observe that comparing with the single replica based models (e.g. iFogStor, iFogStorZ, iFogStorG), the overall latency of our strategy with distributed workload decreases as cp increases. This is due to the fact that MultiCopyStorage places replicas of data on multiple hosts. When the number of consumers that subscribe to the same data is large, placing multiple replicas on different hosts increases the corresponding writing latency slightly. However, the consumers can read the data with much lower reading latency and consequently reduce the overall latency compared to the single replica strategy. With Zoning workload, as shown in Fig. 5, the benefit of MultiCopyStorage declines slightly. When cp = 15, 30 and 50, the overall latency reduces by 1.07%, 1.14% and 1.21%, respectively, comparing with iFogStor. We can observe that the latency of MultiCopyStorage decreases slightly compared to other strategies. This is because when the consumers are confined to a particular area, a single replica is enough to minimize the overall latency effectively. When the consumers are distributed, placing multiple replicas across the network topology manifests more prominent impact.

  1. 2)

    Average replicas number: Fig. 6 shows the average number of replicas of each piece of data achieved by MultiCopyStorage. It can be observed that the number of replicas climbs with the increase of cp. Generally, the number of replicas is bigger on Distributed workload than on Zoning workload since the consumers of each data are distributed on distributed workload and the more replicas are required to lessen the transmit time of extracting the data. Moreover, one cannot discuss a replication strategy without considering the overhead of replicating the data since more replicas in the system accounts for greater overhead in terms of additional network traffic and data storage cost. Nevertheless, the average number of replicas shown in Fig. 6 is acceptable since it is indirectly constrained by our objective function. With the constraint of replicas number, the overhead could be controlled.

Figure 6
figure 6

Average number of replicas of MultiCopyStorage when GWs = 1000.

  1. 3)

    Solving time: As shown in Fig. 7, compared with MultiCopyStorage, the solving time of iFogStorZ(5),iFogStorG(5) and iFogStorG(10) strategy increases significantly with the increase of GWs. Compared with iFogStorZ(5),iFogStorG(5) and iFogStorG(10), the solving time of MultiCopyStorage decreases by 83.69%,85.75% and 68.18%, respectively, when GW = 3000 and cp = 5. Meanwhile, Fig. 9 illustrates that when cp = 50 and GW = 3000, the solving time of MultiCopyStorage is lower than iFogStorZ(5), iFogStorG(5) and iFogStorG(10) by 39.01%, 50.69% and 39.38% respectively. As can be seen from Figs. 7, 8 and 9, with the increase of cp, it takes MultiCopyStorage more time to solve the problem.

Figure 7
figure 7

Solving time of different strategies when cp=5.

Figure 8
figure 8

Solving time of different strategies when cp=30.

Figure 9
figure 9

Solving time of different strategies when cp=50.

In conclusion, although iFogStor, based on a linear model, can always get the optimal solution of the single replica placement problem, when cp is large and the consumers are widely distributed the single replica strategy fails to enable the distributed consumers to obtain the data with a low latency. In contrast, MultiCopyStorage can reduce the overall latency to some extent. As cp increases, the reduction in overall latency becomes more prominent, although more time is needed to solve the problem. In addition, although the graph partitioning strategies e.g. iFogStorZ and iFogStorG reduce the complexity of the linear model to a certain extent, each partition still uses the CPLEX [10] to solve the problem, which is very time-consuming compared with the MultiCopyStorage strategy. It can be seen from the experiment results that MultiCopyStorage is superior to other single replica strategies in both solution quality and efficiency. This is because of the fact that the MultiCopyStorage strategy is based on the multiple data replicas placement model, which considers the drawback of single replica model. Furthermore, some heuristic rules are applied when solving the problem, which greatly reduces the solving time.

6 Conclusions and Future Work

In this paper, iFogStorM, a multiple data replicas placement model for Fog Computing is proposed to reduce the overall latency of data storage. Finding the optimal solution of this model is a NP-hard problem. Therefore, based on iFogStorM, we propose a greedy strategy called MultiCopyStorage, which applies a number of heuristic rules and greatly reduces the scope of searching for good solutions. In order to verify the proposed strategy, we extended iFogSim [2] to implement the real-time simulation of the multiple data replicas placement strategy, and conducted the experiments to compare our strategy with the latest single replica strategies. The experiment results show that MultiCopyStorage reduces the overall latency by 6% on average compared with FogStorage [8], and by 10% compared with the heuristic strategy iFogStorG [9]. Because of its high efficiency, MultiCopyStorage can be operated at a high rate and offers good data storage solutions at real time in the fog computing environment.

As for the data placement problem in Fog Computing, this paper only considers the resource capacity as the limiting factor. Basically, the more replicas involve greater overhead in terms of additional network traffic, data storage cost, CPU and memory consumption due to transmission. In our study, the number of replicas is indirectly constrained by minimizing the overall latency but with the larger cp (the number of consumers that subscribe to the same data), the number of replicas increases and the overhead would climb too. Hence, there should be a trade-off between the performance in reducing the overall latency and the number of replicas. We will further investigate this problem in future work. In addition, the fixed latency is used for every network link when calculating the overall latency, which does not consider the latency changes due to network congestion. In addition, we do not consider the stability of fog nodes. Thus, a trust evaluation mechanism [29] may be introduced to prevent the cases like device faults, network congestion. Furthermore, since cyber threats are growing up with the development of cloud technology, some Fog-based storage security scheme should be proposed to ensure the integrity, availability and confidentiality of the data preserving in fog nodes [30, 31]. In the future, we will incorporate these factors and further improve the existing model.