Ground robots operating in outdoor environments often face rough and uneven terrains, which can significantly challenge their navigation capabilities. This paper proposes an approach for classifying roughness levels of outdoor terrains using an inertial sensor for wheeled robots. For this, a Deep Learning based approach is proposed to classify the level of irregular terrains. Our methodology consists of two main steps: (i) inertial measures representation; and (ii) roughness level classification. First, inertial measures from a sliding window are acquired and depicted as a bi-dimensional representation. After, the vibration features are learned by a Convolutional Neural Network and classified into different roughness levels. Additionally, the impact of different terrain and robot conditions is assessed to comprehend its effect during terrain analysis. Simulated and real-world experiments were carried out to validate the proposed approach, achieving accurate and reliable results, even in different surface circumstances. The proposed approach achieved an of accuracy over 96% in simulated experiments regarding different surface heights, distances, and shapes. In addition, it achieved an accuracy of over 88% in real experiments.