In this letter, we study the optimal power allocation problem where the goal is to maximize the long-term end-to-end sum rate of an underwater full-duplex energy harvesting relay network. The problem is formulated as an online sequential decision-making problem where the relay adapts the transmit power in each time-slot based on past and current information of harvested energy, battery level, channel state information, and interference level. The optimal transmission policy is obtained through the reinforcement learning framework. Simulation results show that the optimal online power allocation policy achieves a higher sum rate than the computationally-efficient sub-optimal online greedy power allocation policy, especially under insufficient harvested energy.