The surface properties of an object can be explored by sliding the tip of a tool over the surface. This induces vibrations in the tool that can be captured using acceleration sensors. Based on these signals, this paper presents a first step towards tool-mediated retrieval of perceptually similar surfaces. We conduct subjective experiments to identify ground truth data about perceptually similar surfaces in a database of 69 materials. The results of this experiment are used to evaluate our retrieval system. For a surface query, the retrieval system should return those surfaces the user perceives as similar. We test various features in our retrieval pipeline and evaluate their usefulness in such a retrieval scenario. We target towards a tradeoff between high classification accuracy and high recall of perceptually similar surfaces.