Enormous volumes of data made available by the high-resolution satellite imagery enable us to use a deep framework in the field of remote sensing for image classification. Recently, deep learning has been an area of interest for the researchers in the computer vision domain due to its high efficiency toward large-scale, high-dimensional data. In this letter, we propose an unsupervised learning algorithm to cluster hybrid polarimetric SAR images, and dual-polarized SAR images using the deep framework. We use feature extraction layers of the VGG16 model with batch normalization, which is trained with small patches derived from the hybrid polarimetric SAR images. It uses an entropy-based loss function and an adaptive learning rate optimization algorithm, Adam, for training. Broadly, the patches are segmented into three classes, namely, surface, volume, and double-bounce, which are defined with reference to the SAR scattering characteristics. Furthermore, we classify volume into dense forest region and agricultural crop fields. We also observe mixed classes between volume and double-bounce, mainly covering the settlements surrounded by areas covered by tall trees. Furthermore, we use transfer learning for generating the labels for dual-polarized images by using the learned weights of a hybrid polarized image model. Such a technique renders an average accuracy of 89.70% and 86.08% for hybrid polarized SAR images and dual-polarized SAR images, respectively. Hence, this method explores the spatial characteristics of remotely sensed images to distinguish urban settlements, water bodies, agricultural, and forest areas from the underlying scene in an unsupervised fashion.