Pricing strategies in mobile crowdsensing: an enhanced MAPPO approach using a behavior network

Abstract

Mobile crowdsensing involves assigning multiple tasks around points of interest to mobile users (MUs) for execution. Developing an optimal task allocation strategy is crucial for the entire system, as it directly impacts the benefits of stakeholders. Leveraging recent advancements in multi-agent reinforcement learning (MARL), which have demonstrated unique advantages in simulating complex interactions among multiple agents, we propose a pricing strategy based on an improved behavior network and multi-agent proximal policy optimization (MAPPO) algorithm. Specifically, we formulate the problem as a multi-leader multi-follower Stackelberg game, and then apply MAPPO, a MARL technique which employs centralized training and decentralized execution, to solve this game. To better capture complex sequential input information and achieve superior behavior strategies, we integrate an attention mechanism with a gated recurrent unit (GRU) network into the actor network, forming a MARL algorithm with an improved behavior network, termed GRU-and-Attention-based MAPPO (GA-MAPPO). Simulation results demonstrate that the proposed GA-MAPPO algorithm is effective compared with baseline approaches. It can learn an optimal pricing strategy that maximizes the benefits of Task Initiators (TIs) and guides TIs in pricing MUs effectively.

Appendix A

Proof of theorem 1

The corresponding Lagrangian form of Problem 1 is

$$\begin{aligned} L_n=\sum _m(p_m^nt_m^n-a_m^nt_m^{n2}-b_m^nt_m^n)-\lambda _{n0}(\sum _mt_m^n-\kappa _n)+\sum _m\lambda _{nm}t_m^n, \end{aligned}$$

(A1)

where ${\lambda _{n0}}$ and ${\lambda _{nm}}$ are the Lagrangian multipliers.

The Karush-Kuhn-Tucker (KKT) conditions are given as follows.

$$\begin{aligned} & \frac{{\partial {L_n}}}{{\partial t_m^n}} = 0,\forall m \in \mathcal{M}, \end{aligned}$$

(A2)

$$\begin{aligned} & {\lambda _{n0}}(\mathop \sum \limits _m t_m^n - {\kappa _n}) = 0,{\lambda _{nm}}t_m^n = 0, \end{aligned}$$

(A3)

$$\begin{aligned} & \lambda _{n0},\lambda _{nm}\ge 0,t_m^n\ge 0,\sum _mt_m^n\le \kappa _n. \end{aligned}$$

(A4)

Without loss of generality, we consider $t_m^n> 0$ and we can obtain that ${\lambda _{nm}} = 0$. Eqn. (A2) can be converted to

$$\begin{aligned} p_m^n - 2a_m^nt_m^n - b_m^n - {\lambda _{n0}} + {\lambda _{nm}} = 0,\forall m \in \mathcal{M}. \end{aligned}$$

(A5)

Then, Eqn. (A5) can be decomposed into two cases as follows.

• (1) Case I: ${\lambda _{n0}} = 0$, according to Eqn. (A5), we have

  $$\begin{aligned} t_m^n = \frac{{p_m^n - b_m^n}}{{2a_m^n}}. \end{aligned}$$

  (A6)

• (2) Case II: ${\lambda _{n0}}> 0$, according to Eqn. (A5), we have $t_m^n = \frac{{p_m^n - b_m^n - {\lambda _{n0}}}}{{2a_m^n}}$. Substituting $t_m^n$ into Eqn. (A3), we have ${\lambda _{n0}} = \frac{{\left( {\sum \nolimits _m {\left( {p_m^n - b_m^n} \right) \left( {\prod \nolimits _{m1 \ne m} {a_{m1}^n} } \right) } } \right) - 2{\kappa _n}\left( {\prod \nolimits _m {a_m^n} } \right) }}{{\sum \nolimits _m {\left( {\prod \nolimits _{m1 \ne m} {a_{m1}^n} } \right) } }}$, and therefore $t_m^n$ can be obtained as follows.

  $$t_{m}^{n} = {\text{ }}\left\{ {\begin{array}{*{20}l} {\frac{{p_{m}^{n} - b_{m}^{n} }}{{2a_{m}^{n} }},} \hfill & {{\text{if}}\;\sum\nolimits_{{m = 1}}^{M} {\frac{{p_{m}^{n} - b_{m}^{n} }}{{2a_{m}^{n} }} \le \kappa _{n} ,} } \hfill \\ {\frac{{p_{m}^{n} - b_{m}^{n} - \lambda _{{n0}} }}{{2a_{m}^{n} }},} \hfill & {{\text{else}}\;{\text{if}}\;\lambda _{{n0}} {\text{> }}0,} \hfill \\ {0,} \hfill & {{\text{otherwise}}{\text{.}}} \hfill \\ \end{array} } \right.$$

  (A7)

However, the positive or negative of $t_m^n$ is not constrained in fact and it is a must. Then, we supplement the above equation. Considering that $\sum _mt_m^n\le \kappa _n$, we have ${C_1} = \sum \nolimits _m {\min \left( {0,t_m^n} \right) }$ and ${C_2} = \sum \nolimits _m {\max \left( {0,t_m^n} \right) }$, which are the cumulative sum of sensing time that are less than or greater than 0. We process the $t_m^n$ greater than zero with a specified ratio ${{{C_1}} / {{C_2}}}$ and set the $t_m^n$ less than 0 to 0, which means the consistent distribution of losses and initial benefit.

In summary, the optimal solution to Problem 1 is given as follows.

$$\left( {t_{m}^{n} } \right)^{*} = {\text{ }}\left\{ {\begin{array}{*{20}l} {t_{m}^{n} \left( {1 - \frac{{\left( { - C_{1} } \right)}}{{C_{2} }}} \right),} \hfill & {{\text{if}}\;C_{1} < 0\;{\text{and}}\;t_{m}^{n} {\text{ }}> 0,} \hfill \\ {0,} \hfill & {{\text{else}}\;{\text{if}}\;C_{1} < 0\;{\text{and}}\;t_{m}^{n} {\text{ }} < 0,} \hfill \\ {t_{m}^{n} ,} \hfill & {{\text{otherwise,}}} \hfill \\ \end{array} } \right.$$

(A8)

where,

$$t_{m}^{n} = {\text{ }}\left\{ {\begin{array}{*{20}l} {\frac{{p_{m}^{n} - b_{m}^{n} }}{{2a_{m}^{n} }},} \hfill & {{\text{if}}\;\sum\nolimits_{{m = 1}}^{M} {\frac{{p_{m}^{n} - b_{m}^{n} }}{{2a_{m}^{n} }} \le \kappa _{n} ,} } \hfill \\ {\frac{{p_{m}^{n} - b_{m}^{n} - \lambda _{{n0}} }}{{2a_{m}^{n} }},} \hfill & {{\text{else}}\;{\text{if}}\;\lambda _{{n0}} {\text{ }}> 0,} \hfill \\ {0,} \hfill & {{\text{otherwise, }}} \hfill \\ \end{array} } \right.$$

(A9)

$$\begin{aligned} {\lambda _{n0}} = \frac{\left( {\sum \nolimits _m {\left( {p_m^n - b_m^n} \right) \left( {\prod \nolimits _{m1 \ne m} {a_{m1}^n} } \right) } } \right) - 2{\kappa _n}\left( {\prod \nolimits _m {a_m^n} } \right) }{\sum \nolimits _m {\left( {\prod \nolimits _{m1 \ne m} {a_{m1}^n} } \right) } }, \end{aligned}$$

(A10)

$$\begin{aligned} {C_1} = \sum \nolimits _m {\min \left( {0,t_m^n} \right) }, \end{aligned}$$

(A11)

$$\begin{aligned} {C_2} = \sum \nolimits _m {\max \left( {0,t_m^n} \right) }. \end{aligned}$$

(A12)

$\square$