Recent progress on quantum physics shows that quantum computers may be a reality in the not too distant future. Based on new mathematical hard problems, post-quantum cryptography (PQC) has been studied to make sure the attacks from quantum computers can be resistant. The latest supersingular isogeny Diffie-Hellman (SIDH) key exchange protocol shows promising security properties among various post-quantum cryptosystems. In this paper, we propose an improved modular multiplication algorithm with special primes that can be used in SIDH key exchange protocol. Both software and hardware implementations are provided and compared with original modular multiplication algorithm. The results show that the software results of improved algorithm can be 24% faster than the original software implementation, while the hardware implementation based on the proposed hardware architecture can be 6 times faster than previous hardware implementation.