This letter presents an uncertainty-aware motion planner for an autonomous underwater vehicle (AUV) which navigates in a cluttered scenario under an uncertain flow field. Safe and efficient navigation in such environments requires a precise motion model of an AUV. However, the motion of an AUV is difficult to be accurately predicted due to its nonlinear dynamics significantly affected by its surrounding fluids. To resolve this difficulty, we propose to use a Bayes Adaptive Markov Decision Process (BA-MDP) for robust decision making, which is able to hedge against errors in the AUV’s motion model through online learning. To construct the BA-MDP accurately, a Clustering Guided Explorative Tree (CGETree) is proposed to explore all the possible connectivities within the AUV’s state space in an offline manner. The CGETree’s nodes are clustered together to provide a discretization of the state space, and the tree edges, after being combined with a discrete noise model, provide a physics-based-prior for the transition model, which will be updated online along with the robot’s movement. The simulation results demonstrate that our proposed method achieves good performance in a seabed environment with a flow field whose uncertainty may not be known beforehand.