Spam over IP-telephony (SPIT) may emerge as a major threat requiring effective protection mechanisms. A number of anti-SPIT frameworks have been proposed in the last years. These are mainly based on call pattern and signaling analysis as well as caller reputation techniques resulting in black, grey or white lists of callers. Charging schemes and an extension of the call setup with challenge-response procedures have also been investigated. An analysis of the audio content is often claimed to be inappropriate since the call in question is already established when audio data are available. This contribution, however, shows that media data can nevertheless be effectively used for SPIT mitigation. A robust audio fingerprint of spectral feature vectors is computed for incoming audio data. Using a database of feature vectors, new calls are compared with previous ones and replays with identical or similar audio data are detected. Depending on the policy, future calls from the same source can then be blocked during call setup. A prototype based on this approach has been developed and first results show that the system can effectively detect and block Spam calls.