In recent years the number of cores in computers has increased considerably, opening new lines of research to adapt classical techniques of machine learning to a parallel scenario. In this paper, we have developed and implemented with the multi-platform application programming interface OpenMP a method to train Semiparametric Support Vector Machines relying on Sparse Greedy Matrix Approximation (SGMA) and Iterated Re-Weighted Least Squares algorithm (IRWLS). We take advantage of the matrix formulation of SGMA and IRWLS. We recursively apply the partitioned matrix inversion lemma and other matrix decompositions to obtain a simple procedure to parallelize SVMS with good performance and computational efficiency.