Existing reinforcement learning (RL)-based routing protocols in underwater wireless sensor networks (UWSNs) do not consider the network topology when selecting a next-forwarder for packet forwarding. To eliminate resource waste from the forwarding in a wrong direction, this paper proposes a network topology-aware RL routing protocol for UWSNs. Taking the network topology into account, sensor nodes first find next-forwarder candidates and then select a highest-valued one of them to forward data. The simulation result shows that the proposed scheme outperforms QELAR in terms of latency and total energy consumption.