The effectiveness of machine learning (ML) approaches for machine fault diagnosis (MFD) has been proved in previous studies. The majority of the previous studies used simulation or laboratory datasets. However, the robustness of the ML models for (MFD) in real-world applications has rarely been discussed. In this work, we conducted an empirical study on the robustness of ML approaches in case of wheel flat (WF) detection for railway freight wagons. WF is a common failure on wheel tread surface, causing large impulsive impacts on vehicles and infrastructure. We made great efforts to collect and clean up the relevant field data that was measured on different freight wagons running with different faulty conditions on different lines at different speed ranges. The collected datasets can represent the complexity of real-world railway operational conditions. We build several baseline models, incorporating deep convolutional neural network (CNN) and different signal processing techniques for WF detection, in order to study their robustness against diverse conditional variations in practice.