Optimal time-based strategy for automated negotiation

Abstract

Recent years are showing increased adoption of AI technology to automate business and production processes thanks to the recent successes of machine learning techniques. This leads to increased interest in automated negotiation as a method for achieving win-win agreements among self-interested agents. Research in automated negotiation can be traced back to the Nash bargaining game in the mid 20th century. Nevertheless, finding an optimal negotiation strategy against an unknown opponent with an unknown utility function is still an open area of research. The most recent result in this area is the Greedy Concession Algorithm (GCA) which can be shown to be optimal under specific constraints on both the negotiation protocol (non-repeating offers), opponent (static acceptance-model) and search space (deterministic time-based strategies). In this paper, we extend this line of work by providing an algorithmically faster version of GCA called Quick GCA which reduces the time-complexity of the search process from O(K²T) to O(KT) where K is the size of the outcome-space and T is the number of negotiation rounds allowed. Moreover, we show that GCA/QGCA can be applied in a more general setting; Namely with repeating-offers protocols and to search the more general probabilistic time-based strategies. Finally, we heuristically extend QGCA to more general opponents with general time-dependent acceptance-model and negotiation settings (real-time limited negotiations) in three steps called QGCA⁺, PBS, and PA that iteratively and greedily modify the policy proposed by QGCA⁺ applied to an approximate static acceptance model. The paper evaluates the proposed approach empirically against state of the art negotiation strategies (winners of all relevant ANAC competition winners) and shows that it outperforms them in a wide variety of negotiation scenarios.

Appendices

Appendix A: Proofs for operations on policies

Here we provide a simple proof for the results found in Tables 1 and 2. In the following proofs, there are several cases in which a division by $1 -{a_{k}^{k}}$ is needed. This seems to pose a difficulty when ${A_{k}^{k}} = 1$ but in all of these cases, it is possible to rewrite the expression avoiding this division using P and S values albeit with a much longer expression and cumbersome notation. We will avoid this complexity here and keep the version using the devision.

1.1 A.1 Proof for $\pi ^{\omega }_{i}$

$$ \begin{array}{@{}rcl@{}} \mathcal{E}\mathcal{U}(\pi) &=& \sum\limits_{j=1}^{T+1} u_{j} {{a}_{j}^{j}} P_{j} = \sum\limits_{j=1}^{i-1} u_{j} {{a}_{j}^{j}} P_{j} + u_{i} {{a}_{i}^{i}} P_{i} \\&&+ \sum\limits_{j=i+1}^{T+1} u_{j} {{a}_{j}^{j}} P_{j} \end{array} $$

Notice that, by definition, a(ω,T + 1) = 1 and π_T+ 1 = ϕ (See Section 3.5) which takes into account nonzero reservation values.

$$ \begin{array}{@{}rcl@{}} \mathcal{E}\mathcal{U}(\pi_{i}^{\omega}) &=& \sum\limits_{j=1}^{i-1} u_{j} {{a}_{j}^{j}} P_{j} + u(\omega) a(\omega, i) P_{i} \\ && + \sum\limits_{j=i+1}^{T+1} {u_{j} {{a}_{j}^{j}} P_{i} (1-a(\omega, i)) \prod\limits_{l=i+1}^{j-1} 1-{{a}_{l}^{l}}} \end{array} $$

Subtracting and after some manipulations we get:

$$ \begin{array}{@{}rcl@{}} &&\mathcal{E}\mathcal{U}(\pi_{i}^{\omega}) - \mathcal{E}\mathcal{U}(\pi) \\&=& P_{i} (u(\omega)a(\omega, i) - u_{i} {{a}_{i}^{i}})\\ && - (S_{T +1}- S_{i}) + (S_{T+1} - S_{i}) \frac{1-a(\omega, i)}{1-{{a}_{i}^{i}}} \\ &=& P_{i} (u(\omega)a(\omega, i)-u(\pi_{i}) {{a}_{i}^{i}})\\ && + (S_{T+1} - S_{i}) \frac{{{a}_{i}^{i}} - a(\omega, i)}{1-{{a}_{i}^{i}}} \end{array} $$

This completes the proof for TDAM (Table 1). The proof for SAM (Table 2) follows from replacing a(x,i) with a(x).

1.2 A.2 Proof for $\pi _{k\leftrightarrow s}$

$$ \begin{array}{@{}rcl@{}} &&\mathcal{E}\mathcal{U}(\pi_{k\leftrightarrow s}) \\&=& \sum\limits_{i=1}^{k-1} {u_{i} {{a}_{i}^{i}} P_{i}} + u_{s} {{a}_{s}^{k}} P_{k} + \sum\limits_{i=k+1}^{s-1} {u_{i} {{a}_{i}^{i}} P_{i} \frac{1-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}}}\\ && + u_{k} {{a}_{k}^{s}} P_{s} \frac{1-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} + \sum\limits_{i=k+1}^{T+1} u_{i}{{a}_{i}^{i}} P_{i} \\ && - \sum\limits_{i=1}^{T+1} {u_{i} {{a}_{i}^{i}} P_{i}} \\ &=& P_{k} (u_{s}{{a}_{s}^{k}} - u_{k}{{a}_{k}^{k}}) + P_{s} \left( u_{k}{{a}_{k}^{s}} \frac{1-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} - u_{s}{{a}_{s}^{s}}\right)\\ && + \left( \sum\limits_{i=k+a}^{s-a} u_{i}{{a}_{i}^{i}} P_{i}\right) \frac{1-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} \end{array} $$

$$ \begin{array}{@{}rcl@{}} &=& P_{k} (u_{s}{{a}_{s}^{k}}-u_{k}{{a}_{k}^{k}}) + P_{s} \left( u_{k}{{a}_{k}^{s}} \frac{1-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} - u_{s}{{a}_{s}^{s}}\right)\\ && + (S_{s-1} - S_{k}) \frac{1-{{a}_{s}^{k}}}{a-{{a}_{k}^{k}}} \\ &=& u_{k} \left( {{a}_{k}^{s}} P_{s} \frac{1-{{a}_{s}^{k}}}{a-{{a}_{k}^{k}}} - {{a}_{k}^{k}} P_{k}\right) + u_{k} \left( {{a}_{s}^{k}} P_{k}-{{a}_{s}^{s}} P_{s}\right)\\ && + \frac{{{a}_{k}^{k}} - {{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} (S_{s-1}-S_{k+1}) \end{array} $$

This completes the proof for TDAM (Table 1). The proof for SAM (Table 2) follows from replacing ${a_{x}^{i}}$ with a_x.

1.3 A.3 Proof for $\pi _{k\rightarrow 1}$

This is a special case of $\pi _{k\leftrightarrow s}$ with s = k + 1.

$$ \begin{array}{@{}rcl@{}} \mathcal{E}\mathcal{U}(\pi_{k\leftrightarrow s}) \!&=&\! u_{k}\left( {{a}_{k}^{s}} P_{s} \frac{1 - {{a}_{s}^{k}}}{a-{{a}_{k}^{k}}} - {{a}_{k}^{k}} P_{k}\right) + u_{k} ({{a}_{s}^{k}} P_{k} - {{a}_{s}^{s}} P_{s})\\ && \!+ \frac{{{a}_{k}^{k}}-{{a}_{s}^{k}}}{1-{{a}_{k}^{k}}} (S_{s-1}-S_{k+1}) \\ \!&=&\! u_{k} \left( {a}_{k}^{k+1} P_{k+1} \frac{1-{a}_{k+1}^{k}}{a-{{a}_{k}^{k}}} - {{a}_{k}^{k}} P_{k}\right)\\ && \!+ u_{k} ({a}_{k+1}^{k} P_{k}-{a}_{k+1}^{k+1} P_{k+1}) \\ \!&=&\! u_{k} ({a}_{k}^{k+1} P_{k} (1-{a}_{k+1}^{k}) - {{a}_{k}^{k}} P_{k})\\ && \!+ u_{k} ({a}_{k+1}^{k} P_{k}-{a}_{k+1}^{k+1} P_{k+1}) \end{array} $$

This completes the proof for TDAM (Table 1). The proof for SAM (Table 2) follows from replacing ${a_{x}^{i}}$ with a_x.

Note that for SAM, this becomes:

$$ \begin{array}{@{}rcl@{}} \mathcal{E}\mathcal{U}(\pi_{k\leftrightarrow s}) &=& u_{k} ({a}_{k}^{k+1} P_{k} (1-{a}_{k+1}^{k}) - {{a}_{k}^{k}} P_{k})\\ && + u_{k} ({a}_{k+1}^{k} P_{k}-{a}_{k+1}^{k+1} P_{k+1}) \\ &=& u_{k} (a_{k} P_{k} (1-a_{k+1}) - a_{k} P_{k})\\ && + u_{k} (a_{k+1} P_{k}-a_{k+1} P_{k+1}) \\ &=& u_{k} (-a_{k}a_{k+1} P_{k}) + u_{k+1} (a_{k}a_{k+1} P_{k}) \\ &=& P_{k} a_{k} a_{k+1} (u_{k+1}-u_{k}) \end{array} $$

This expression is always positive if u_k+ 1 > u_k, which leads to the greedy concession lemma proven by Baarslag et al. [9].

1.4 A.4 Proof for π _ω@i

$$ \begin{array}{@{}rcl@{}} &&\mathcal{E}\mathcal{U}({\pi}_{\omega{@}{i}}) \\&=& \sum\limits_{i=1}^{k-1} u_{i}{{a}_{i}^{i}} P_{i} + a(\omega, k)u(\omega)P_{k} \end{array} $$

$$ \begin{array}{@{}rcl@{}} && + \sum\limits_{i=k}^{T+1} u_{i} {a}_{i}^{i+1} P_{i} (1-a(\omega^{k})) - \sum\limits_{i}^{T+1} u_{i}{{a}_{i}^{i}} P_{i} \\ &=& a(\omega, k)u(\omega) P_{k}+ (1-a(\omega, k)) \sum\limits_{i=k}^{T+1} u_{i} {a}_{i}^{i+1} P_{k}\\ && \times \prod\limits_{j=k+1}^{i-1} (1-a_{j}^{j+1}) - \sum\limits_{i=k}^{T} {{a}_{i}^{i}} u_{i} P_{k} \prod\limits_{j=k+1}^{i-1} (1-{{a}_{j}^{j}}) \\ &=& P_{k} \left( u_{k}a(\omega, k) + (1-a(\omega, k)) \sum\limits_{i=k}^{T+1} u_{i} ({a}_{i}^{i+1}-{{a}_{i}^{i}})\right.\\ &&\left. \times\left( \prod\limits_{j=k+1}^{i-1} (1-{a}_{j}^{j+1}) - \prod\limits_{j=k+1}^{i-1} (1-{{a}_{j}^{j}}) \right)\right) \\ &=& u(\omega)a(\omega, k) P_{k} + (1-a(\omega, k) ) ({S}_{T+1}^{+1} - {S}_{k-1}^{+1})\\ && - (S_{T+1} - S_{k-1}) \end{array} $$

where we define ${S}_{j}^{+1} = {\sum }_{i=1}^{j} u_{i} {a}_{i}^{i+1} {P}_{i}^{+1}$, and ${P}_{i}^{+1} = {\prod }_{j=1}^{i-1} 1-{a}_{j}^{j+1}$. These two new lists can be calculated incrementally the same way that S and P are calculated (See (3) and (4)).

For the special case of a SAM, ${S}_{j}^{+1}=S_{j}$ and ${P}_{i}^{+1}=P_{i}$ which simplifies the expression greatly:

$$ \begin{array}{@{}rcl@{}} &&\mathcal{E}\mathcal{U}({\pi}_{\omega{@}{k}}) \\&=& u(\omega)a(\omega, k) P_{k} + (1-a(\omega,k)) ({S}_{T+1}^{+1} - {S}_{k-1}^{+1})\\ && - (S_{T+1}-S_{k-1}) \\ &=& u(\omega)a(\omega) P_{k} - a(\omega) (S_{T+1}-S_{k-1}) \\ &=& a(\omega) (u(\omega) P_{k} - (S_{T+1}-S_{k-1})) \\ &=& a(\omega) (S_{k-1} + u(\omega) P_{k} - S_{T+1}) \end{array} $$

Appendix B: Counter example for the Greedy Concession Lemma

To prove that the Greedy Concession Lemma is false for a TDAM, we only need to find a single counter example. This is one such counter example:

Let T = 2, Ω = {ω_i|0 ≤ i ≤ 4}, and let a, u be as follows:

$$ a = \left( \begin{array}{cccccc} t & \omega_{0} & \omega_{1} & \omega_{2} & \omega_{3} & \omega_{4} \\ 0 & 0.0729 & 0.3047 & 0.2323 & 0.9274 & 0.7293 \\ 1 & 0.2280 & 0.9936 & 0.9323 & 0.2979 & 0.1223 \end{array}\right) $$

$$ u(\omega_{i}) = \left( \begin{array}{ccccccccc} \omega_{0} & \omega_{1} & \omega_{2} & \omega_{3} & \omega_{4} \\ 0.0723 & 0.1778 & 0.8086 & 0.8768 & 0.9446 \end{array}\right) $$

By substituting into (2), It is easy to verify that the optimal policies of length one and two are:

$$ {\pi}_{1}^{*} = \langle\omega_{3}\rangle $$

$$ \pi^{*}_{2} = \langle\omega_{4} , \omega_{2}\rangle $$

It is clear that the longer policy does not contain outcome ω₃ which is contained in the shorter policy. This shows that greedy construction of policies is not optimal for TDAMs.

For another example that directly contradicts the statement that the optimal policy is monotonically decreasing in terms of agent utility which is the core of the greedy concession lemma, consider the following monotonic acceptance model which is a special case of a TDAM:

$$ a = \left( \begin{array}{cccccccccc} t & \omega_{0} & \omega_{1} & \omega_{2} & \omega_{3} & \omega_{4} & \omega_{5} & \omega_{6} & \omega_{7} & \omega_{8} \\ 0 & 0.9087 & 0.9689 & 0.7115 & 0.7464 & 0.8324 & 0.8715 & 0.6600 & 0.7836 & 0.8372 \\ 1 & 0.8320 & 0.9460 & 0.6462 & 0.6810 & 0.8048 & 0.5904 & 0.5358 & 0.6562 & 0.7934 \\ 2 & 0.8061 & 0.9123 & 0.6030 & 0.4878 & 0.6479 & 0.5686 & 0.5130 & 0.6285 & 0.6937 \\ 3 & 0.6980 & 0.7857 & 0.5387 & 0.2988 & 0.4877 & 0.5280 & 0.4482 & 0.5021 & 0.5893 \\ 4 & 0.6034 & 0.6827 & 0.4343 & 0.2978 & 0.3306 & 0.5007 & 0.3056 & 0.5014 & 0.5025 \\ 5 & 0.4819 & 0.5891 & 0.3277 & 0.2167 & 0.3041 & 0.2851 & 0.2348 & 0.4322 & 0.3639 \\ 6 & 0.4026 & 0.5125 & 0.2302 & 0.1843 & 0.2877 & 0.2763 & 0.2285 & 0.3490 & 0.3554 \\ 7 & 0.3087 & 0.3689 & 0.1115 & 0.1464 & 0.2324 & 0.2715 & 0.0600 & 0.1836 & 0.2372 \end{array}\right) $$

And let the utility function be:

$$ u(\omega_{i}) = \left( \begin{array}{ccccccccc} \omega_{0} & \omega_{1} & \omega_{2} & \omega_{3} & \omega_{4} & \omega_{5} & \omega_{6} & \omega_{7} & \omega_{8} \\ 0.0065 & 0.0257 & 0.0372 & 0.0847 & 0.3885 & 0.4643 & 0.4896 & 0.6835 & 0.7590 \end{array}\right) $$

By exhaustive search over all policies of length 7, it is straightforward to show that the optimal policy is:

$$ pi^{*}: \langle \omega_{8}, \omega_{8}, \omega_{8}, \omega_{8}, \omega_{8}, \omega_{7}, \omega_{8}\rangle $$

This policy is certainly not a concession policy because the outcome ω₇ appears both before and after ω₈ and they have different utility values.

Appendix C: Proofs for time complexities

This appendix provides proofs for all time-complexity results mentioned in the paper.

1.1 C.1 Time and space complexity of GCA

An efficient implementation of GCA will keep the policy in a sorted list by utility values allowing for insertion of new outcomes in $O(\log (K))$ operations. This means that $sort_{\mathcal {E}\mathcal {U}}(\pi \circ \omega )$ takes $O(\log (K))$. For the K such policies, we also need to calculate the expected utility $\mathcal {E}\mathcal {U}$ which is an order K operation. This means that a single loop of the algorithm takes O(K²) operations. This is repeated for each output of the final policy. Thus, the whole algorithm will still require O(K²T) steps.

The space complexity of the algorithm is O(K) as the algorithm needs only to keep track of the optimal policy which cannot exceed K outcomes because of the assumption that offers are not repeated.

The algorithm needs to keep the policy which needs O(T) space. Becasue the $\arg \max \limits$ operation does not require any extra storage, the space complexity of GCA is O(T).

1.2 C.2 Time complexity of QGCA and QGCA⁺

The time complexity of QGCA can be found by following Algorithm 2. Initialization lines (Line 1–3) can be done in O(1). The inner most loop (Lines 5–10) is executed O(K) times and each iteration of this loop can be executed in O(1) steps leading to O(K) for the loop. Outcome insertion (Line 11) can be done in $O(\log (T))$ steps by keeping an ordered list of outcomes by utility value (same as in GCA). This is dominated by the inner loop though. Updating §, ¶, and ${\mathscr{L}}$ is O(K) operations. This implies that the each iteration of the outer loop (Lines 5-14c) takes O(K) steps. This is repeated T times leading to an overall time complexity of O(KT).

Other than the policy, the algorithm needs to keep track of three arrays (§, ¶, and ${\mathscr{L}}$). § and ¶ are O(T) elements, and ${\mathscr{L}}$ is O(outcomes) elements. This means that the space complexity of QGCA is $O(\max \limits (K, T))$.

As QGCA⁺ is the same as QGCA computationally, it has the same time and space complexities.

1.3 C.3 Time and space complexity of PBS

The inner loop of Algorithm 3 (Lines 4–6) runs T iterations each taking O(1) operations while the outer loop (Lines 2–8) has T iterations. This leads to a time complexity of O(T²).

The algorithm keeps no data strctures that are not kept by QGCA which leads to the same space complexity (O(max(K,T))).

1.4 C.4 Time and space complexity of PA

The inner loop of Algorithm 4 (Lines 7-8) is O(T). This is repeated R times (Lines 2-9) leading to a time complexity of O(TR).

The algorithm keeps no data strctures that are not kept by QGCA which leads to the same space complexity (O(max(K,T))).