This paper explores the RSA (Rivest, Adi Shamir and Leonard Adleman) encryption/decryption performance using a scalable and programmable approach with an improved processor architecture and software implementation. The methodology gives a new perspective when investigates the design and performance of custom hardware on a flexible instruction set architecture platform targeting embedded systems. The approach consists of design new hardware stages at the same time uses an open-source big number library to execute operations up to 4096-bit length. All the optimizations are based on Montgomery Modular Multiplication and Exponentiation, employing it to perform RSA operations. Our experiments show that the overall performance of the hybrid solution is comparable and sometimes outperforms dedicated solutions, therefore it can be put into perspective as an effective and viable choice between an ASIC and a software solution.