In real-world situations, the performance of vision-based systems in indoor environments has been observed to provide noticeable results. However, in the case of the outdoor environment, the proper functioning of vision-based systems is often disrupted due to the occurrence of atmospheric/weather effects. To ensure the functional operation of all vision-based devices in any weather condition, a reliable classification/recognition system is needed to identify the weather pertinent to the scenes for effective removal. Consequently, a trustworthy classification/recognition system is required to recognize the weather pertinent to the outdoor scenes for its effective removal to guarantee the efficient functioning of all vision-based systems in any weather situation. For this, this work investigated and analyzed the performance of conventional handcrafted features extracted from outdoor scenes to solve the problem of atmospheric/weather classification tasks in real-world outdoor scenes. Recently, convolutional neural network (CNN)-based architectures have been used by research communities for solving many vision-based problems. Motivated by the enormous success of CNNs, in the present scope of this article, the perception capability of deep CNN features is also investigated for effective analysis and discrimination of atmospheric/weather-degraded scenes. To achieve the precise necessities of the objectives, several experiments were conducted. The analysis and experiments were carried out using the created Extended Tripura University Video Dataset (E-TUVD), which consists of a diverse set of atmospheric/weather-degraded outdoor scenes. The experiment results reveal that effectively selecting the discriminative features, thereby describing the weather properties, can improve classification accuracy and provide an accurate weather type classification using machine learning classifiers.