A deep reinforcement learning-based approach for pricing in the competing auction-based cloud market

Abstract

In the cloud market, there exist multiple cloud providers adopting auction-based mechanisms to offer cloud resources to users. These auction-based cloud providers need to compete against each other to maximize the profits by setting the cloud resource prices effectively. In this paper, we analyze how an auction-based cloud provider sets the auction price effectively when competing against other cloud providers in the evolutionary market where the amount of participated cloud users is changing. The pricing strategy is affected by many factors, such as the auction price of its opponents, the prices charged to users in the previous round, the bidding behavior of cloud users, and so on. Therefore, we model this problem as a partially observable Markov game and adopt a gradient-based multi-agent deep reinforcement learning algorithm to generate the competing pricing strategy. We also run extensive experiments to evaluate our pricing strategy against other five benchmark pricing strategies in the auction-based cloud market. The experimental results show that our generated pricing strategy can beat other pricing strategies in terms of long-term profits and the amount of participated users, and it can also learn cloud users’ marginal values and their choice of cloud providers effectively.

Appendix

In this section, we describe how to derive the probability of the cloud user choosing a cloud provider given its expected utility. Specifically, the probability of cloud user j choosing to be served by provider i at stage t is denoted as $P_{j, i}^{t}$:

$$\begin{aligned} \begin{aligned} P_{j, i}^{t}&={\text {Prob}}\left( u_{j, i}^{t}>u_{j, i^{\prime }}^{t},\forall i^{\prime } \ne i\right) \\&={\text {Prob}}\left( v_{j, i}^{t}+\eta _{j, i}>v_{j, i^{\prime }}^{t}+\eta _{j, i^{\prime }}, \forall i^{\prime } \ne i\right) \\&={\text {Prob}}\left( \eta _{j, i^{\prime }}<v_{j, i}^{t}-v_{j, i^{\prime }}^{t}+\eta _{j, i}, \forall i^{\prime } \ne i\right) \end{aligned} \end{aligned}$$

(20)

From Eq. 20, we can see that cloud user j will choose to be served by provider i only if the user’s utility is maximized. According to Eq. 7, we then get an expression for the choice probability:

$$\begin{aligned} P_{j,i}^{t}=e^{-e^{(v_{j,i}^{t}-v_{j,i^{\prime }}^{t}+\eta _{j,i})}} \end{aligned}$$

(21)

Since $\eta _{j,i}$ is independent, the cumulative distribution over all $i^{\prime }\ne i$ is the product of the individual cumulative distributions:

$$\begin{aligned} P_{j, i}^{t} \vert \eta _{j, i}=\prod _{i \ne i^{\prime }} e^{-e^{v_{j, i}^{t}-v_{j, i^{\prime }}^{t}+\eta _{j, i}}} \end{aligned}$$

(22)

Now, the choice probability is the integral of $P_{j,i}^{t}\vert \eta _{j,i}$ over all values of $\eta _{j,i}$ weighted by its density:

$$\begin{aligned} \small P_{j, i}^{t}=\int \left( \prod _{i \ne i^{\prime }} e^{-e^{-(p_{i,t}^\mathrm{avg}-p_{i^{\prime },t}^\mathrm{avg}-\eta _{j, i})}}\right) e^{-\eta _{j, i}} e^{-e^{-\eta _{j, i}}} \hbox {d} \eta _{j, i} \end{aligned}$$

(23)

The closed-form expression is

$$\begin{aligned} P_{j, i}^{t}=\frac{e^{v_{j, i}^{t}}}{\sum _{i^{\prime }} e^{v_{j, i^{\prime }}^{t}}} \end{aligned}$$

(24)

which is the probability of user j choosing provider i at stage t.