We propose a novel, real-time, camera-based framework for traffic light detection and recognition from a moving vehicle. For increasing safe driving, automated detection and recognition of traffic lights plays an important role in intelligent vehicles and Advanced Driver-Assistance Systems (ADAS). However, automated detection and recognition of the traffic lights is challenging because of their small size, colors (red, yellow and green) that may be similar to the background, illumination variation, environmental conditions, etc. In our proposed framework, we detect the traffic lights using Faster Region-based Convolutional Neural Network (R-CNN) and recognise the traffic lights based on Grassmann manifold learning. We use transfer learning on VGG16 to extract features from the detected traffic lights and use these features for creating subspaces for each traffic light (red, yellow and green). These subspaces lie on a Grassmann manifold and encompass the variations in different instances of the same traffic light and the uncertainties during detection. We use discriminant analysis on the manifold for recognising them in real-time. Our experiments on multiple publicly available datasets and comparison with several state-of-the-art methods show that our proposed framework has a high degree of accuracy and is robust.