Integrating deep neural networks with reinforcement learning has exhibited excellent performance in the literature, highlighting the ability of neural networks to extract features. This paper begins with a simple Markov decision process inspired from a cognitive task. We show that Q-learning, and approximate Q-learning using a linear function approximation fail in this task. Instead, we show that Q-learning combined with a neural network-based function approximator can learn the optimal policy. Motivated by this finding, we outline procedures that allow the use of a neural network to extract appropriate features, which can then be used in a Q-learning framework with a linear function approximation, obtaining performance similar to that observed using Q-learning with neural networks. Our work suggests that neural networks can be used as feature extractors in the context of Q-learning.