Policy Gradient for Arabic to English Neural Machine Translation

Abstract

In this work, we present a strategy of using the policy gradient training method from deep reinforcement learning to train a Seq2Seq model. This strategy is based on the combination of the classical cross-entropy and the policy gradient in the training phase. To evaluate the effectiveness of this strategy, we compare two Seq2Seq models trained with two different training methods to translate from Arabic to English. The first method is the cross-entropy, and the second one is a combination of cross-entropy and policy gradient with different amounts. Experimental results show that the second training method leads to improve the performance of the sequence-to-sequence model by 0.71 BLEU points.

A Proof of Equation 11

Here we demonstrate the Eq. (11).

We have the Eq. (10):

$$ L_{PG} = - E_{\tau \sim \pi _{\theta }}\left[ R(\tau ) \right] $$

$$ \nabla _{\theta } L_{PG} = -\nabla _{\theta } E_{\tau \sim \pi _{\theta }}\left[ R(\tau ) \right] $$

Given a function f(x), $p(x|\theta )$ a parametrized probability distribution, and its expectation $E_{x \sim p(x|\theta )}\left[ f(x) \right]$:

$$\begin{aligned} \nabla _{\theta }E_{x \sim p(x|\theta )}\left[ f(x) \right]= & {} \nabla _{\theta } \int dx\; f(x) p(x|\theta ))\\= & {} \int dx\; \nabla _{\theta } \left( f(x)p(x|\theta ) \right) \\= & {} \int dx\; \left( \nabla _{\theta } f(x)p(x|\theta ) + f(x) \nabla _{\theta } p(x|\theta ) \right) \\= & {} \int dx\; f(x)\nabla _{\theta }p(x|\theta ) \\= & {} \int dx\; f(x)p(x|\theta ) \frac{\nabla _{\theta }p(x|\theta )}{p(x|\theta )} \\= & {} \int dx f(x)p(x|\theta )\nabla _{\theta }\log p(x|\theta ) \\= & {} E_{x \sim p(x|\theta )}\left[ f(x)\nabla _{\theta }\log p(x|\theta ) \right] \end{aligned}$$

Now, we suppose $x = \tau$, $f(x)=R(\tau )$, and $p(x|\theta )=p(\tau |\theta )$:

$$\begin{aligned} \nabla _{\theta } L_{PG} = - E_{\tau \sim \pi _{\theta }}\left[ R(\tau )\nabla _{\theta }\log p(\tau |\theta ) \right]&(*) \end{aligned}$$

The trajectory $\tau$ is a sequence of events $a_t$ and $s_{t+1}$, respectively sampled from the agent’s policy $\pi _{\theta }(a_{t}|s_{t})$ and the probability of transition $p(s_{t+1}|s_{t}, \; a_{t})$. The probability of the complete trajectory is the product of the individual probabilities:

$$p(\tau |\theta ) = \prod _{t=0}^{T}p(s_{t+1}|s_{t}, \; a_{t}) \pi _{\theta } (a_{t}|s_{t})$$

$$\log p(\tau |\theta ) = \log \prod _{t=0}^{T}p(s_{t+1}|s_{t}, \; a_{t}) \pi _{\theta } (a_{t}|s_{t})$$

$$\log p(\tau |\theta ) = \sum _{t=0}^{T} \left( \log p(s_{t+1}|s_{t}, \; a_{t})+\log \pi _{\theta } (a_{t}|s_{t}) \right) $$

$$\begin{aligned} \nabla _{\theta }\log p(\tau |\theta ) = \nabla _{\theta }\sum _{t=0}^{T} \log \pi _{\theta } (a_{t}|s_{t})&(**) \end{aligned}$$

By putting (**) in (*), we find the Eq. (11):

$$\nabla _{\theta }L_{PG} = - E_{\tau \sim \pi _{\theta }}\left[ R(\tau ) \sum _{t=0}^{T}\nabla _{\theta } \log \pi _{\theta }(a_{t} | s_{t}) \right] $$