Diagnosis of rolling bearing damage is one of the essential maintenance tasks to improve the operational efficiency of rotating machinery. Recent studies on bearing diagnostic models have been based on deep learning. However, there has been little discussion of how the performance of deep learning-based diagnostic models is affected by changes in the train-test data split rules of the CWRU dataset used in those studies. Motivated by this question, we examined the diagnostic performance of deep learning models under multiple conditions of data split rules and a number of diagnostic classes in the CRWU dataset. The frequency-domain and time-domain characteristics of the train-test data were also compared, and their relationship to diagnostic performance was discussed. We found that differences occurred mainly in the frequency-domain for the train-test data split conditions that could be set for the CWRU dataset. The more differences exist between train and test data in the frequency-domain, the worse the diagnostic performance of the deep learning model tended to be. On the other hand, there were no split conditions with apparent differences in time-domain characteristics. Therefore, to construct a more robust diagnostic model, it is desirable to use data with various frequency and time domain characteristics for train-test, such as using more diverse datasets together.