The kernel perceptron is one of the simplest and fastest kernel machines, its performance, however, is inferior to other well known kernel machines. We introduce an algorithm that combines several approaches, mainly Herbrich's large-scale Bayes point machine and the soft perceptron in order to improve the kernel perceptron. Our experiments, which were based on standard benchmark datasets, show that the performance of the perceptron can be improved significantly with similar computational effort.