An Efficient Dynamic Allocation Mechanism for Security in Networks of Interdependent Strategic Agents

Abstract

Motivated by security issues in networks, we study the problem of incentive mechanism design for dynamic resource allocation in a multi-agent networked system. Each strategic agent has a private security state which can be safe or unsafe and is only known to him. At every time, each agent faces security threats from outside as well as from his unsafe neighbors. Therefore, the agents’ states are correlated and have interdependent stochastic dynamics. Agents have interdependent valuations, as each agent’s instantaneous utility depends on his own security state as well as his neighbors’ security states. There is a network manager that can allocate a security resource to one agent at each time so as to protect the network against attacks and maximize the overall social welfare. We propose a dynamic incentive mechanism that implements the efficient allocation and is ex-ante (in expectation) individually rational and budget balanced. We present a reputation-based payment that mitigates any risk that the agents or the network manager may face to get a negative utility or to run a budget deficit, respectively, for some realizations of the network stochastic evolution. Therefore, our results provide a dynamic incentive mechanism that implements efficient allocations in networked systems with strategic agents that have correlated types and interdependent valuations, and is approximate ex-post individually rational and budget balanced.

Appendices

Proof of Lemma 1:

Consider an arbitrary agent $i \in {\mathcal {N}}$ and time $t \in {\mathcal {T}}$. When all agents except i adopt truthful strategies for the future, i.e., ${\varvec{r}}^{-i}_{\tau }=\varvec{\theta }^{-i}_{\tau }$, $\tau \ge t$, the discounted value of the expected incentive agent i gets (which is the negative of the monetary payment he makes) at time $t+1$ is

$$\begin{aligned} \delta {\mathbb {E}}\left\{ -p^i_{t+1}\left( m^i_t, {\varvec{r}}^{-i}_t,a_t\right) \right\}= & {} \delta {\mathbb {E}}\left\{ -p^i_{t+1}\left( m^i_t, \varvec{\theta }^{-i}_t,a_t\right) \right\} \nonumber \\= & {} -\delta \left[ {\mathcal {P}}\left( m^i_t=0 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) p^i_{t+1}\left( 0, \varvec{\theta }^{-i}_t,a_t\right) \nonumber \right. \\&\left. +\,{\mathcal {P}}\left( m^i_t=1 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) p^i_{t+1}\left( 1, \varvec{\theta }^{-i}_t,a_t\right) \right] , \end{aligned}$$

(24)

where in the last equality, the expectation is computed by conditioning on the value of the cross-inference signal $m^i_t$; Agent i pays $p^i_{t+1}(0, \varvec{\theta }^{-i}_t,a_t)$ if he is assessed as an unsafe agent at time t, and pays $p^i_{t+1}(1, \varvec{\theta }^{-i}_t,a_t)$ if he is considered to be safe. Substituting the tax incentives (15–16) in (24) we have

$$\begin{aligned}&\delta {\mathbb {E}}\left\{ -p^i_{t+1}\left( m^i_t, \varvec{\theta }^{-i}_t,a_t\right) \right\} \nonumber \\&\quad ={\mathcal {P}}\left( m^i_t=0 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) \left[ \sum _{j \ne i}{v^j\left( 0,\varvec{\theta }^{-i}_t,a_t\right) }\right. \nonumber \\&\quad \quad \left. -\frac{{\mathcal {P}}\left( m^i_t=1 |0, \varvec{\theta }^{-i}_t,a_t\right) }{{\mathcal {P}}\left( m^i_t=0 |0, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) } \right. \left. \alpha \sum _{\begin{array}{c} j \in O^i: \\ \theta ^j_t=1 \text { or } a_t=j \end{array}}{ \frac{l_{ij}}{\sum _{k \in {\mathcal {N}}^j}{l_{kj}}}}\right] \nonumber \\&\quad \quad +{\mathcal {P}}\left( m^i_t=1 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) \left[ \sum _{j \ne i}{v^j\left( 1,\varvec{\theta }^{-i}_t,a_t\right) }\right. \nonumber \\&\quad \quad \left. +\,\frac{{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) }{{\mathcal {P}}\left( m^i_t=0 |0, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) }\right. \left. \alpha \sum _{\begin{array}{c} j \in O^i: \\ \theta ^j_t=1 \text { or } a_t=j \end{array}}{\frac{l_{ij}}{\sum _{k \in {\mathcal {N}}^j}{l_{kj}}}}\right] .\nonumber \\ \end{aligned}$$

(25)

Using (3–4), we can show that

$$\begin{aligned} \alpha \sum _{\begin{array}{c} j \in O^i: \\ \theta ^j_t=1 \text { or } a_t=j \end{array}}{ \frac{l_{ij}}{\sum _{k \in {\mathcal {N}}^j}{l_{kj}}}}=\sum _{j \ne i}{v^j\left( 1,\varvec{\theta }^{-i}_t,a_t\right) }-\sum _{j \ne i}{v^j\left( 0,\varvec{\theta }^{-i}_t,a_t\right) }, \end{aligned}$$

(26)

where the right-hand side is the effect of agent i’s safety on the total valuations of other agents. Substituting (26) in (25) and rearranging, we obtain

$$\begin{aligned}&\delta {\mathbb {E}}\left\{ -p^i_{t+1}\left( m^i_t, \varvec{\theta }^{-i}_t,a_t\right) \right\} \nonumber \\&\quad =\frac{{\mathcal {P}}\left( m^i_t=0 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) }{{\mathcal {P}}\left( m^i_t=0 |0, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) }\sum _{j \ne i}{v^j\left( 0,\varvec{\theta }^{-i}_t,a_t\right) }\nonumber \\&\qquad +\,\frac{{\mathcal {P}}\left( m^i_t=0 |0, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |\theta ^i_t, \varvec{\theta }^{-i}_t,a_t\right) }{{\mathcal {P}}\left( m^i_t=0 |0, \varvec{\theta }^{-i}_t,a_t\right) -{\mathcal {P}}\left( m^i_t=0 |1, \varvec{\theta }^{-i}_t,a_t\right) } \sum _{j \ne i}{v^j\left( 1,\varvec{\theta }^{-i}_t,a_t\right) }\nonumber \\&\quad ={\varvec{1}}_{\left\{ \theta ^i_t=0\right\} }\sum _{j \ne i}{v^j(0,\varvec{\theta }^{-i}_t,a_t)}+ {\varvec{1}}_{\left\{ \theta ^i_t=1\right\} }\sum _{j \ne i}{v^j(1,\varvec{\theta }^{-i}_t,a_t)}= \sum _{j \ne i}{v^j(\theta ^i_t,\varvec{\theta }^{-i}_t,a_t)}, \end{aligned}$$

(27)

which completes the proof. $\square$

Proof of Proposition 1:

We want to show that the DCI mechanism is incentive compatible. To prove this, we can disregard participation fees (17) taken from the agents at the beginning of the mechanism, since these participation fees are independent of the agents’ future reports, and hence do not affect the agents’ strategies when they join the mechanism. To prove that the truth-telling is the best strategy that each agent i can choose from each time t onward, using the one shot deviation principle [4], we only need to show that for each i, t and $\varvec{\theta }_t$, telling the truth by agent i at time t maximizes his expected continuation payoff, i.e., $r^i_t=\theta ^i_t$ is a solution to the following optimization problem

$$\begin{aligned}&\max _{r^i_t} \left[ v^i\big (\varvec{\theta }_t,a^*_t\big (r^i_t,\varvec{\theta }^{-i}_t\big )\big )-p^i_t\left( m^i_{t-1}, {\varvec{r}}^{-i}_{t-1},a_{t-1}\right) \right. \nonumber \\&\left. \quad +\,\delta {\mathbb {E}}\left\{ v^i\big (\varvec{\theta }_{t+1},a^*_{t+1}\big (\varvec{\theta }_{t+1}\big )\big )-p^i_{t+1}\big (m^i_t, \varvec{\theta }^{-i}_{t},a^*_t\big (r^i_t,\varvec{\theta }^{-i}_t\big )\big )\right\} \right. \nonumber \\&\left. \quad +\,{\mathbb {E}}\left\{ \sum \limits _{\tau =t+2}^{\infty }{\delta ^{\tau -t} \big (v^i\big (\varvec{\theta }_\tau ,a^*_\tau \big (\varvec{\theta }_\tau \big )\big )-p^i_\tau \big (m^i_{\tau -1}, \varvec{\theta }^{-i}_{\tau -1},a^*_{\tau -1}\big (\varvec{\theta }_{\tau -1}\big )\big )\big )}\right\} \right] . \end{aligned}$$

(28)

Using Lemma 1 we can show that the problem (28) is equivalent to the following optimization problem:

$$\begin{aligned} \max _{r^i_t}&\left[ \sum \limits _{j=1}^n {v^j\big (\varvec{\theta }_t,a^*_t\big (r^i_t,\varvec{\theta }^{-i}_t\big )\big )}\right. \nonumber \\&+\left. {\mathbb {E}} \left\{ \sum \limits _{\tau =t+1}^\infty {\delta ^{\tau -t} \sum \limits _{j=1}^n {v^j\big (\varvec{\theta }_\tau ,a^*_\tau \big (\varvec{\theta }_\tau \big )\big )}}\right\} -p^i_t\big (m^i_{t-1}, {\varvec{r}}^{-i}_{t-1},a_{t-1}\big )\right] . \end{aligned}$$

(29)

Since the third term in the objective function of (29), i.e., $p^i_t(m^i_{t-1}, {\varvec{r}}^{-i}_{t-1},a_{t-1})$, is independent of $r^i_t$, the optimal report of agent i is also a solution to the optimization problem below which maximizes the social surplus from time t onward, i.e.,

$$\begin{aligned} \max _{r^i_t}{ \left[ \sum \limits _{j=1}^n {v^j\big (\varvec{\theta }_t,a^*_t\big (r^i_t,\varvec{\theta }^{-i}_t\big )\big )}+{\mathbb {E}} \left\{ \sum \limits _{\tau =t+1}^\infty {\delta ^{\tau -t} \sum \limits _{j=1}^n {v^j\big (\varvec{\theta }_\tau ,a^*_\tau \big (\varvec{\theta }_\tau \big )\big )}}\right\} \right] }. \end{aligned}$$

(30)

Therefore, agent i’s objective is aligned with the social welfare. The allocation policy $a^*$ used by the network manager is the policy that maximizes the social welfare under the centralized information; therefore, the best strategy for agent i, whose objective is shown to be aligned with the social welfare, is to trust the network manager and to provide her with truthful information. Consequently, $r^i_t=\theta ^i_t$ is a solution to problem (30) and this establishes the incentive compatibility of the DCI mechanism. $\square$

Proof of Proposition 4:

We want to show that the DCI mechanism is individually rational, i.e., each agent prefers the outcome of the mechanism to that attained by opting out. According to (17), the participation fee each agent pays to join the mechanism is equal to the expected value of the subsidies he gets at truth-telling equilibrium; therefore,

$$\begin{aligned} {\mathbb {E}}\left\{ (1-\delta )\sum _{\tau =0}^\infty {\delta ^\tau p^i_\tau \big (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau \big )}\right\} =0,\quad \forall i \in {\mathcal {N}}. \end{aligned}$$

(31)

However, each agent’s valuation when he participates in the mechanism is greater than or equal to the utility he gets when he opts out. This is because when agent i does not participate in the mechanism the network manager allocates no security measure to any agent which makes the security situation worse for all agents. Therefore,

$$\begin{aligned}&{\mathbb {E}}\left\{ (1-\delta )\sum _{\tau =0}^\infty {\delta ^\tau \left[ v^i\big (\varvec{\theta }_\tau ,a_\tau \big (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau \big )\big )-p^i_\tau \big (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau \big )\right] }\right\} \nonumber \\&\quad ={\mathbb {E}}\left\{ (1-\delta )\sum _{\tau =0}^\infty {\delta ^\tau v^i\big (\varvec{\theta }_\tau ,a_\tau \big (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau \big )\big )}\right\} \ge {\mathbb {E}}\{U_0^i\},\quad \forall i \in {\mathcal {N}}. \end{aligned}$$

(32)

This proves individual rationality of the DCI mechanism. $\square$

Proof of Theorem 1:

In Propositions 1–4 we prove that the DCI mechanism maximizes the social welfare function W and is incentive-compatible, budget balanced, and individually rational. These results proves that the DCI mechanism solves the network manager’s dynamic incentive design problem (13). $\square$

Proof of Lemma 2:

Let $V^i=(1-\delta )\sum _{\tau =0}^\infty {\delta ^\tau v^i(\varvec{\theta }_\tau ,a_\tau (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau ))}$ and $P^i=(1-\delta )\sum _{\tau =0}^\infty {\delta ^\tau p^i_\tau (\varvec{\theta }_{\tau }^{-i}, \theta _{\tau }^i,h_\tau )}$ denote agent i’s total valuation and total payment, respectively. Using these notations, ex-post requirements (19)-(20) can be rewritten as

$$\begin{aligned} {\mathbb {P}}\Big \{V^i-P^i \le U_0^i -\eta \Big \} \le \zeta ,\quad \forall i \in {\mathcal {N}}, \end{aligned}$$

(33)

and

$$\begin{aligned} {\mathbb {P}}\left\{ |{\sum _{i\in {\mathcal {N}}}{P^i}|} \ge \eta \right\} \le \zeta . \end{aligned}$$

(34)

In this problem, for any realization of the stochastic events, the valuation of the agent when he participates in the mechanism is at least equal to his utility when opts out, i.e., $V^i \ge U_0^i$. This is because, if agent i unilaterally opts out, the network manager stops allocating security measures to the network agents; therefore, the epidemics spread uncontrollably and all of the agents lose. Therefore, a condition sufficient to satisfy (33) is

$$\begin{aligned} {\mathbb {P}}\{|{P^i}|\ge \eta \} \le \zeta ,\quad \forall i \in {\mathcal {N}}. \end{aligned}$$

(35)

Suppose that the conditions of Lemma 2 are satisfied. Due to condition (1), each $P^i$, $i \in {\mathcal {N}}$, and hence $\sum _{i\in {\mathcal {N}}}{P^i}$ are zero-mean random variables. Therefore, using the Chebyshev inequality, we have

$$\begin{aligned} {\mathbb {P}}\{|{P^i}|\ge \eta \} \le \frac{Var(P^i)}{\eta ^2},\quad \forall i \in {\mathcal {N}}, \end{aligned}$$

(36)

and

$$\begin{aligned} {\mathbb {P}}\left\{ \Big |{\sum _{i\in {\mathcal {N}}}{P^i}\Big |} \ge \eta \right\} \le \frac{Var\left( \sum _{i\in {\mathcal {N}}}{P^i}\right) }{\eta ^2} \le \frac{\left( \sum _{i\in {\mathcal {N}}}{\sqrt{Var(P^i)}}\right) ^2}{\eta ^2}, \end{aligned}$$

(37)

where the last inequality in (37) follows from Cauchy–Schwarz inequality.

According to condition (2), the mechanism can make the variance of the agents’ payments smaller than any bound $\epsilon >0$. Let $\epsilon =\zeta \eta ^2/n^2$. Substituting $Var(P^i) \le \zeta \eta ^2/n^2$, $i \in {\mathcal {N}}$, in (36–37), we conclude inequalities (35) and (34), respectively, and establish that the mechanism is approximate ex-post IR and BB. $\square$

Proof of Theorem 2:

We divide the proof into two cases:

Case 1$h<1$. In this case we show that Markov chain $M^*$ derived from the evolution of the system under the optimal allocation policy is irreducible and aperiodic, hence is ergodic. We prove this via several steps.

Step 1 We show that in the Markov chain $M^*$ every state is accessible from the clean state $\varvec{\theta }={\varvec{1}}$ where all agents are safe. A state $\hat{\varvec{\theta }}$ is said to be accessible from a state $\varvec{\theta }$ if a system started in state $\varvec{\theta }$ has a nonzero probability of transitioning into state $\hat{\varvec{\theta }}$ in one or more moves. If the system is in the clean state, since all the probabilities $d_i$, $i \in {\mathcal {N}}$, $l_{ij}$, $i \in {\mathcal {N}}$, $j \in {\mathcal {N}}^i$, and h are strictly between 0 and 1, for each agent i both the events that i gets attacked at the next time or remains safe have nonzero probabilities. Therefore, any state $\hat{\varvec{\theta }}$ is accessible from $\varvec{\theta }={\varvec{1}}$ in just one move.

Step 2 We show that the clean state $\varvec{\theta }={\varvec{1}}$ is accessible from any state $\hat{\varvec{\theta }} \in \Theta ^n$. We prove this by induction on the number of unsafe agents in the original/starting state $\hat{\varvec{\theta }}$, i.e., $K(\hat{\varvec{\theta }})=n-\sum _{i \in {\mathcal {N}}}{{\hat{\theta }}^i}$.

Base case Let $K(\hat{\varvec{\theta }})=0$. The only state in the network that has no unsafe agent is the clean state; i.e., $\hat{\varvec{\theta }}={\varvec{1}}$. Starting from the clean state the claim directly follows from Step 1.

Induction step Suppose that the clean state is accessible from any state with at most k unsafe agents. Now, let $K(\hat{\varvec{\theta }})=k+1$. In a state $\hat{\varvec{\theta }}$ with $k+1$ unsafe agents, by the assumption made in the statement of the theorem the network manager applies the security measure to one of the unsafe agents, say agent i. Since the success probability h of the security measure is nonzero, and the probabilities of new external or internal attacks to agents are strictly less than one, there is a positive probability that the system transitions to a state $\tilde{\varvec{\theta }}$ where agent i is safe and no new agent gets attacked. In state $\tilde{\varvec{\theta }}$ the number of unsafe agents is k; therefore, by the induction hypothesis, the clean state is accessible from $\tilde{\varvec{\theta }}$. Since the accessibility relation is transitive, the clean state $\varvec{\theta }={\varvec{1}}$ is accessible from the starting state $\hat{\varvec{\theta }}$.

Conclusion By the principle of induction, we have proved that the clean state is accessible from every state.

Step 3 We show that the Markov chain $M^*$ is irreducible. Since the accessibility relation is transitive, it follows from Steps 1 and 2 that in $M^*$ every state is accessible from every other state. Therefore, the Markov chain $M^*$ is irreducible.

Step 4 We show that the Markov chain $M^*$ is aperiodic. For irreducible Markov chains, either all states are periodic or all are aperiodic. Therefore, since in Step 3 we proved that the Markov chain $M^*$ is irreducible, it is enough to show that one of its states is aperiodic. A sufficient condition for a state to be aperiodic is that it has a self-loop, i.e., starting from that state the probability that the next state is the same as the current state is nonzero. According to Step 1, in the Markov chain $M^*$ the clean state has a self-loop; therefore, it is an aperiodic state, and this proves the aperiodicity of the Markov chain $M^*$.

It follows from Steps 3 and 4 that the Markov chain $M^*$ is both irreducible and aperiodic; furthermore, $M^*$ is finite-state, every state in it is positive recurrent. Therefore, $M^*$ is ergodic.

Case 2$h = 1$. In this case we show that the Markov chain $M^*$ consists of one class of ergodic states and possibly one transient state, hence it is an ergodic uni-chain. We prove this in two steps.

Step I We show that in the Markov chain $M^*$ every state $\hat{\varvec{\theta }} \ne {\varvec{0}}$ is accessible from the clean state $\varvec{\theta }={\varvec{1}}$. Let $i=a^*({\varvec{1}})$ denote the agent who receives the security measure when the network is in the clean state. Since the probability of success when applying the security measure is $h=1$, starting from the clean state, agent i will keep his safety until the next time slot. However, since the probabilities of external and internal attacks are strictly between 0 and 1, other agents except i could be either safe or unsafe in the next time. Therefore, all states with ${\hat{\theta }}^i=1$ are accessible from the clean state in just one move. Now consider a state $\hat{\varvec{\theta }} \ne {\varvec{0}}$ with ${\hat{\theta }}^i=0$. Since $\hat{\varvec{\theta }}\ne {\varvec{0}}$ there is at least one agent $j \ne i$ such that ${\hat{\theta }}^j=1$. We show that $\hat{\varvec{\theta }}$ is accessible from the clean state in two moves. Consider a state $\bar{\varvec{\theta }}$ where ${\bar{\theta }}^j=0$, and ${\bar{\theta }}^k=1$, for all $k \ne j$, that is agent j is unsafe and all other agents are safe. As a result of the argument of step 1, $\bar{\varvec{\theta }}$ is accessible from the clean state $\varvec{\theta }={\varvec{1}}$ in one move. By the assumption made in the statement of theorem, in state $\bar{\varvec{\theta }}$ the optimal policy applies the security measure to agent j, i.e., $a^*(\bar{\varvec{\theta }})=j$. As a result, agent j switches to the safe state at the next time; all other agents may or may not get attacked, and each event occurs with a positive probability. Thus, all states where agent j is safe, which includes $\hat{\varvec{\theta }}$, are accessible from $\bar{\varvec{\theta }}$ in one move. Since $\bar{\varvec{\theta }}$ is itself accessible from the clean state in one move, we can conclude that $\hat{\varvec{\theta }}$ is accessible from the clean state in two moves. Therefore, every state $\hat{\varvec{\theta }} \ne {\varvec{0}}$ is accessible from the clean state in one or two moves.

Step II We show that the Markov chain $M^*$ is an ergodic uni-chain. Using the same proof as in Step 2 of Case 1, we can show that the claim of Step 2 is still true when $h=1$, i.e., the clean state $\varvec{\theta }={\varvec{1}}$ is accessible from every state. Therefore, due to the transitivity of the accessibility relation, all states except ${\varvec{0}}$ are accessible from each other. If state ${\varvec{0}}$ is also accessible from the clean state, using the same proof as in Steps 3 and 4 of Case 1, we can show that the Markov chain is ergodic. However, if state ${\varvec{0}}$ is not accessible from the clean state, it is a transient state. For finite-state Markov chains, a state is transient if it is not accessible from at least one of the states to which it has access. In this case, following the proofs of Steps 3 and 4 of Case 1, we can show that the set of states except ${\varvec{0}}$ is an ergodic class. Therefore, the Markov chain is an ergodic uni-chain. $\square$

Proof of Theorem 3:

Using the Cauchy–Schwarz inequality we have,

$$\begin{aligned}&Var({\hat{p}}^i(\delta ,K))=(1-\delta )^2 Var\left( \sum _{t=0}^\infty {\delta ^t {\hat{p}}^i_{t}(\delta ,K)}\right) \nonumber \\&\le (1-\delta )^2 \left( \sum _{t=0}^\infty {\delta ^t \sqrt{Var({\hat{p}}^i_{t}(\delta ,K))}}\right) ^2. \end{aligned}$$

(38)

Equation (38) gives an upper bound for the variance of agent i’s total payment based on the variances of his payments ${\hat{p}}^i_{t}(\delta ,K)$ at each time t. Using (22), we can derive time-t variance of agent i’s payment as

$$\begin{aligned}&Var\left( {\hat{p}}^i_{t}(\delta ,K)\right) \nonumber \\&\quad =\frac{1}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\frac{1}{\delta ^{2k}}}\sigma _{i,t-k}^2 + \frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\frac{\rho ^i_{t-k,t-k'}}{\delta ^{k+k'}}}} \sigma _{i,t-k} \sigma _{i,t-k'},\quad \quad \end{aligned}$$

(39)

where $\sigma _{i,t}=\sqrt{Var(p^i_{t})}$, $t\in {\mathcal {T}}$, denotes the standard deviation of agent i’s tax at time t, and $\rho ^i_{t,t'}=Corr(p^i_{t},p^i_{t'})$, $t,t'\in {\mathcal {T}}$, is the correlation coefficient between agent i’s taxes at time t and $t'$.

We want to show that when the discount factor $\delta$ is large enough, the network manager can choose an appropriate K so as to make $Var({\hat{p}}^i_{t}(\delta ,K))$ and hence $Var({\hat{p}}^i(\delta ,K))$, sufficiently small. First we consider the extreme case where $\delta =1$. In this case, we show that the manager can achieve his goal by choosing the agreement period K large enough, as we demonstrate that $Var({\hat{p}}^i(\delta ,K))$ goes to zero when K goes to infinity. Setting $\delta =1$ in (39), we obtain

$$\begin{aligned} Var({\hat{p}}^i_{t}(1,K)) \le \frac{\min {(K,t)}}{K^2} \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} + \frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\rho ^i_{t-k,t-k'}}} \sigma _{i,t-k} \sigma _{i,t-k'}. \end{aligned}$$

(40)

When K goes to infinity, the first term of (40) goes to zero. We upper bound the second term by taking the absolute value and using the triangle inequality as follows,

$$\begin{aligned}&\frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\rho ^i_{t-k,t-k'}}} \sigma _{i,t-k} \sigma _{i,t-k'}\nonumber \\&\quad \le \left| \frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\rho ^i_{t-k,t-k'}}} \sigma _{i,t-k} \sigma _{i,t-k'}\right| \nonumber \\&\quad \le \frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }}\sigma _{i,t-k} \sigma _{i,t-k'} \nonumber \\&\quad \le \frac{2}{K^2} \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }}. \end{aligned}$$

(41)

It is clear that for $t \le K$, the upper bound derived in (41) approaches zero when K goes to infinity. However, to establish the same result for $t >K$, we need the following lemma.

Lemma 3

For the fixed allocation policy $\pi ^*$, we have

$$\begin{aligned} \lim _{l \rightarrow \infty }{\rho ^i_{t,t+l}}=0,\quad \forall i \in {\mathcal {N}}. \end{aligned}$$

(42)

Proof

By definition, we have $\rho ^i_{t,t+l}=Corr(p^i_{t},p^i_{t+l})$, for all $t\in {\mathcal {T}}$. Having fixed the allocation policy, $p^i_{t}$ is a function of the network states at time t and $t-1$, i.e.,

$$\begin{aligned} p^i_{t}=f(\varvec{\theta }_{t-1},\varvec{\theta }_{t}). \end{aligned}$$

(43)

Therefore, we have

$$\begin{aligned} \rho ^i_{t,t+l}=Corr(f(\varvec{\theta }_{t-1},\varvec{\theta }_{t}),f(\varvec{\theta }_{t+l-1},\varvec{\theta }_{t+l})). \end{aligned}$$

(44)

According to Assumption 1, the allocation policy $\pi ^*$ leads to an ergodic uni-chain. Therefore, independently of the initial states of the network, when time tends to infinity, the distribution of the state of the network approaches the limiting distribution of the Markov chain. Consequently, when l goes to infinity, $\varvec{\theta }_{t+l-1}$ and $\varvec{\theta }_{t+l}$ become independent of $\varvec{\theta }_{t-1}$ and $\varvec{\theta }_{t}$. Functions of independent random variables are also independent and hence are uncorrelated; therefore,

$$\begin{aligned} \lim _{l \rightarrow \infty }{\rho ^i_{t,t+l}}=0. \end{aligned}$$

(45)

$\square$

Using the definition of limit, Lemma 3 says that for any $\epsilon >0$, there is an $l_0 (\epsilon )$ such that $l \ge l_0 (\epsilon )$ implies $\left| \rho ^i_{t,t+l}\right| <\epsilon$. Since we want to find the limit of the upper bound derived in (41) when K approaches infinity, for each $\epsilon >0$ we can assume that $K > l_0 (\epsilon )$. Then, for $t > K$ we have

$$\begin{aligned}&\frac{2}{K^2} \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }} \nonumber \\&\quad =\frac{2}{K^2} \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} \left[ \sum _{k=l_0 (\epsilon )}^{K-1} {\sum _{k'=0}^{k-l_0 (\epsilon )} {\left| \rho ^i_{t-k,t-k'}\right| }}+\sum _{k=0}^{l_0 (\epsilon )-1} {\sum _{k'=0}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }}\nonumber \right. \\&\left. \qquad +\, \sum _{k=l_0(\epsilon )}^{K-1} {\sum _{k'=k-l_0 (\epsilon )+1}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }}\right] . \end{aligned}$$

(46)

According to the definition of $l_0 (\epsilon )$, $\left| \rho ^i_{t,t+l}\right| <\epsilon$, for all $l \ge l_0 (\epsilon )$. Moreover, due to the definition of correlation coefficient, $\left| \rho ^i_{t,t+l}\right| \le 1$ for all $l < l_0 (\epsilon )$. Using these bounds in (46) and computing the series, we obtain

$$\begin{aligned}&\frac{2}{K^2} \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\left| \rho ^i_{t-k,t-k'}\right| }}\nonumber \\&\quad \le 2 \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2} \left[ \frac{\epsilon (K-l_0 (\epsilon ))(K-l_0 (\epsilon )+1)}{2K^2} + \frac{l_0(\epsilon ) (l_0 (\epsilon )-1)}{2K^2}\right. \nonumber \\&\quad \left. + \frac{(K-l_0 (\epsilon )) (l_0 (\epsilon )-1)}{K^2}\right] . \end{aligned}$$

(47)

The right-hand side of (47) goes to $\epsilon \max _{0 \le k \le t-1}{\sigma _{i,t-k}^2}$ when K goes to infinity. Since $\epsilon$ could be any arbitrary positive number, it implies that

$$\begin{aligned} \lim _{K \rightarrow \infty }{\frac{2}{K^2} \sum _{k=0}^{\min {(K-1,t-1)}} {\sum _{k'=0}^{k-1} {\rho ^i_{t-k,t-k'}}} \sigma _{i,t-k} \sigma _{i,t-k'}}=0. \end{aligned}$$

(48)

Since a variance is always non-negative, from (40), (41), (47), and (48) it follows that

$$\begin{aligned} \lim _{K \rightarrow \infty }{Var({\hat{p}}^i_{t}(1,K))}=0. \end{aligned}$$

(49)

Using (49), we conclude from (38) that

$$\begin{aligned} \lim _{K \rightarrow \infty }{Var({\hat{p}}^i(1,K))}=0. \end{aligned}$$

(50)

Summing (50) over all $i\in {\mathcal {N}}$, we get

$$\begin{aligned} \lim _{K \rightarrow \infty }{\sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(1,K))}}=0, \end{aligned}$$

(51)

which, due to the definition of limit, is equivalent to

$$\begin{aligned} \forall \epsilon _1>0, \exists K_0(\epsilon _1)>0,\,\,\, s.t. \,\,\, K \ge K_0(\epsilon _1) \Rightarrow \sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(1,K))}<\epsilon _1. \end{aligned}$$

(52)

Therefore, when $\delta$ equals one, for each $\epsilon _1 >0$, the network manager can find appropriate Ks so as to make the total variance of all agents’ payments smaller than $\epsilon _1$. Such Ks obviously make the individual variance of each agent’s payment less than $\epsilon _1$.

Now, we consider the case where $\delta <1$. In this case, the first term on the right-hand side of (39) is a geometric series with common ratio $1/\delta ^2 >1$, hence it diverges as K goes to infinity. This is expected because when the value of money is discounted over time agents must pay larger installments in the future to compensate a tax in the past. Thus, if the agreement period is too long, the installments of the agents become large and the variance of the payments tends to infinity. Therefore, in this case in order to make the risk of agents arbitrarily small, the network manager must choose an appropriate agreement period K, far away from zero and infinity, so as to strike a balance between the positive effects of reputation and the negative effects of large installments. This goal is achievable due to the continuity of the function $\sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(\delta ,K))}$ with respect to $\delta$.

For each $K >0$, $\sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(\delta ,K))}$ is a continuous function of $\delta$. Therefore, by the definition of continuity, we have

$$\begin{aligned}&\forall \epsilon _2>0, \exists \delta _0(K,\epsilon _2)<1,\,\,\, s.t. \,\,\, \delta > \delta _0(K,\epsilon _2)\nonumber \\&\Rightarrow \left| \sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(\delta ,K))}-\sum _{i\in {\mathcal {N}}}{Var({\hat{p}}^i(1,K))}\right| <\epsilon _2. \end{aligned}$$

(53)

For each $\epsilon >0$, consider $\epsilon _1=\epsilon _2=\epsilon /2$. Then, from (52) we get

$$\begin{aligned} \sum _{i\in {\mathcal {N}}}{Var\left( {\hat{p}}^i\left( 1,K_0(\epsilon /2)\right) \right) }<\epsilon /2. \end{aligned}$$

(54)

Considering $K=K_0(\epsilon /2)$, it follows from (53) that

$$\begin{aligned}&\delta > \delta _0(K_0(\epsilon /2),\epsilon /2) \Rightarrow \sum _{i\in {\mathcal {N}}}{Var\left( {\hat{p}}^i\left( \delta ,K_0(\epsilon /2)\right) \right) }<\epsilon \nonumber \\&\quad \Rightarrow Var\left( {\hat{p}}^i\left( \delta ,K_0(\epsilon /2)\right) \right) <\epsilon , \forall i \in {\mathcal {N}}, \end{aligned}$$

(55)

where the last step follows because of the non-negativity of variance. Therefore, for each $\epsilon >0$, there exists a threshold $\delta _0(K_0(\epsilon /2),\epsilon /2)$, such that for all $\delta > \delta _0(K_0(\epsilon /2),\epsilon /2)$, the network manager can make the variance of each agent’s payment smaller than $\epsilon$ by choosing $K=K_0(\epsilon /2)$. This completes the proof of Theorem 3. $\square$

Proof of Theorem 5:

Due to symmetry, we can capture the state of the system by a new state variable $s=\sum \limits _{i=1}^{n} {\theta ^i}$ which only counts the number of safe agents. We can also define a new action variable $b \in \left\{ 0,1 \right\}$, where $b=0$ ($b=1$) represents the act of healing an unsafe (safe) agent. It is easy to show that the optimal value function $V^*(s)$ is increasing in s, meaning that a state with a greater number of safe agents is strictly preferred to a state with smaller number of safe agents.

To prove the theorem, we should show that if the condition holds, ${\pi }^*(s)=0$, for all $s <n$, which is trivial when $s=0$, and is equivalent to showing

$$\begin{aligned} r(s,0)+\delta \sum _{s^{\prime }}{P(s^{\prime }|s, 0) V^*(s^{\prime })} \ge r(s,1)+\delta \sum _{s^{\prime }}{P(s^{\prime }|s, 1) V^*(s^{\prime })}, \end{aligned}$$

(56)

when $s \in [1,n-1]$. We prove (56) in two steps.

Step 1 We show that $r(s,0) \ge r(s,1)$, for all $s \in [1,n-1]$. Due to (4), we have

$$\begin{aligned} r(s,0)=s\left( 1+\alpha \frac{s-1}{n-1}\right) + \alpha \frac{s}{n-1} \ge s\left( 1+\alpha \frac{s-1}{n-1}\right) =r(s,1), \end{aligned}$$

(57)

where the inequality holds because $\alpha$ is positive.

Step 2 We prove that P(.|s, 0) has first order stochastic dominance (FSD) over P(.|s, 1), for all $s \in [1,n-1]$. To this end, we show that $P(s^{\prime } \ge k|s, 0) \ge P(s^{\prime } \ge k|s, 1)$, for all k. Let $n_0$ and $n_1$ denote one of the unsafe and one of the safe agents, respectively. Therefore, due to symmetry, choosing each action variable $b \in \{ 0,1\}$ is equivalent to applying security measure to agent $n_b$. Now we have

$$\begin{aligned} P(s^{\prime } \ge k|s, b)= & {} P\left( \sum _{i} {\theta ^ {^{\prime }i}} \ge k|s, a=n_b\right) =P\left( \sum _{i \ne n_0,n_1} {\theta ^{^{\prime }i}} \ge k|s\right) \nonumber \\&+ P\left( \sum _{i \ne n_0,n_1} {\theta ^{^{\prime }i}} = k-1 |s\right) P(\theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} \ge 1 |s, a=n_b)\nonumber \\&\quad +\, P\left( \sum _{i \ne n_0,n_1} {\theta ^{^{\prime }i}} = k-2 |s\right) P\left( \theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} =2 |s, a=n_b\right) ,\quad \quad \end{aligned}$$

(58)

for $b=0,1$. The terms that depend on b can be computed as follows:

$$\begin{aligned} P(\theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} =2 |s, a=n_1)&=0, \end{aligned}$$

(59a)

$$\begin{aligned} P(\theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} =2 |s, a=n_0)&=h (1-d) (1-d(1-h)) (1-l)^{2(n-s)-1}, \end{aligned}$$

(59b)

$$\begin{aligned} P(\theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} \ge 1 |s, a=n_1)&=(1-d(1-h)) (1-l)^{n-s}, \end{aligned}$$

(59c)

$$\begin{aligned} P(\theta ^{^{\prime }n_0}+\theta ^{^{\prime }n_1} \ge 1 |s, a=n_0)&=\nonumber \\ 1-(1-h (1-d(1-h))&(1-l)^{n-s-1})(1-(1-d) (1-l)^{n-s}). \end{aligned}$$

(59d)

These probabilities (59a–59d) are derived from (2) by setting $\mathcal {N}^i=\mathcal {N}-{i}$ which is true for completely connected networks. It can be shown by some simple algebra that, if $d (1-h) \le l$, (59b) and (59d) are greater than or equal to (59a) and (59c), respectively, and these inequalities complete the proof of Step 2.

Since $V^*(s)$ is increasing in s, we conclude from the above steps that (56) holds for all $s \in [1,n-1]$; therefore, the assertion of Theorem 5 is established.