In this paper, a neural network based controller is proposed for robot manipulators. By considering the second order term of the Taylor expansion of the robot dynamics, the weight tuning algorithm can guarantee the tracking performance of the robot with unknown dynamics. The generic structure selection problem for the neural network controller is addressed by using the structural learning with forgetting, which can automatically remove the redundancy in the structure. Simulations have been conducted on trajectory tracking for various elliptic trajectories. The result demonstrates the effectiveness of the proposed controller.