In this paper, a novel concept is proposed where exploration, which is essential in reinforcement learning, is considered to be one aspect of the motions generated by the learner's internal dynamics and is expected to develop through learning towards more purposeful higher dynamic functions such as “thinking”. To realize such a concept, a chaotic neural network is introduced for generating motions with exploratory factors that are derived from the internal chaotic dynamics without adding external random noises. Effective exploration is expected based on the dynamics called “chaotic itinerancy”, which is also expected to be the key to learning higher dynamic functions more easily that require both stable and transitive dynamics. This paper also proposes a reinforcement learning method without any external random noise, using the temporal difference (TD) error of the state value and the contribution trace of each input to the output increase in each neuron. It was confirmed in a simple learning task that by using a chaotic neural network, an agent could explore in accordance with the internal chaotic dynamics and could learn goal-directed behaviors. The proposed framework seems promising to explain the emergence of higher intelligence in real lives and also to develop human-like intelligence though there are many remaining problems to be solved.