Image based localization is a key block of visual simultaneous localization and mapping (SLAM) system where image data is used to localize the camera relative to an arbitrary reference frame. Although finding the location from one image or between two images is well studied in the literature, few works study the problem of finding the pose of multiple images in videos of different frame lengths. Here, we propose two different architectures to address this problem, one using a combination of 2D convolutional neural network (CNN) and recurrent neural networks (RNN) and the other using 3D CNN. We demonstrate that 3D CNN is better for pose estimation problem than CNN-RNN by visualizing the learned features per layer of both architectures and the accuracy performance. Further, instead of using RGB images as input to the networks, we use SURF descriptors to reduce the image dimension of 480×640×3 by more than 48 folds, making the training time much faster and the learning model less complex. Both architectures show competitive performance in comparison to the state of the art on indoor localization dataset with the ability to generalize to test scenes that are completely different from the training scenes.