Cryptographic keys are critical assets in all cryptographic systems. However, keys stored in plain text format in memory are vulnerable to a variety of memory based attacks, such as buffer overflows, improper bound checks, privilege escalation and physical attacks such as DIMM freeze attacks. Key management in cloud and CDN services is typically performed at two extreme ends of the security spectrum: either the customer keys are shared with the provider in clear violation of security and regulatory requirements, or their keys are maintained securely in customer owned infrastructure, which is a secure solution, but sacrifices performance due to the large roundtrip latencies between the provider and the customer. Existing key management solutions based on network HSMs, PCIe card HSMs or TPMs offer excellent security properties, yet do not scale well at the cloud and CDN. In this paper, we propose a secure, scalable solution which combines the benefits of secure hardware based key management with very high throughput acceleration for private key operations. This novel solution integrates key protection with hardware acceleration and scales easily to support a distributed cloud infrastructure. Through our experiments based on Intel® KPT, we demonstrate that this solution could accelerate the throughput of TLS handshake connections based on RSA2048 by about 4 times with 20% of CPU resources saved.