When a tool is used to tap onto an object or it is dragged over the object surface, vibrations are induced in the tool that can be captured using acceleration sensors. Based on these signals, this paper presents an approach for tool-mediated surface classification which is robust against varying scan-time parameters. We examine freehand recordings of 69 textures and propose a classification system that uses perception-related features such as hardness, roughness and friction as well as selected features adapted from speech recognition such as modified cepstral coefficients. We focus on mitigating the effect of varying contact force and hand speed conditions on these features as a prerequisite for a robust machine-learning-based approach for surface classification. Our system works without explicit scan force and velocity measurements. Experimental results show that our proposed approach allows for successful classification of surface textures under varying freehand movement conditions. The proposed features lead to a classification accuracy of 95% when combined with a Naive Bayes Classifier.