How Danggeun Built a Large-Scale Signing Key Management System with AWS CloudHSM – Part 3: Signing System Implementation, Troubleshooting, and Zero-Downtime Key Migration

Danggeun built a high-availability signing system integrating AWS CloudHSM and AWS KMS to handle over 65 million JWT signing requests per day. CloudHSM uses the PKCS#11 standard interface, and in the Go environment, the crypto11 library efficiently handles session state management and key handle caching.

For system stability, Danggeun abstracted the signing backend through the TokenSigner interface and designed an Active-Standby architecture. The main components are as follows.

• HSMSigner (Active): Performs signing based on CloudHSM PKCS#11
• KMSSigner (Standby): Performs signing based on the AWS KMS API
• LocalSigner (Standby): For migration from the existing system

Using Dynamic Config, signing weights can be adjusted or backends switched in real time without restarting the service. In an actual incident where latency spiked due to an Istio configuration issue, the system automatically failed over to KMS signing, achieving the result of maintaining service with no impact on users.