Face Detection and Recognition is an active research area in surveillance, biometric security and computer vision community. Several methods have been developed for face segmentation and recognition. Many of these are computationally time taking and expensive. This paper proposes another methodology for the same. For face detection, we use a conjunction of RGB and YCbCr color space bounds to segment skin pixels in the image. After noise removal, facial regions are identified by applying the knowledge of general geometric properties of human faces and a Haar cascade for eyes. The method is tested on CMU database, and is found to give accurate results with high detection rates. In this paper, we try to recognize faces by feeding region proposals for faces into a Convolutional Neural Network for feature extraction and finally into a Softmax Classifier for classification.