Kakao Unveils Serving Optimization for Kanana-O Voice AI with OpenAI API Compatibility and Process Isolation

Kakao has unveiled the optimization technologies and architecture for the production serving of its multimodal voice AI model, Kanana-O. To handle the transfer of hidden state embeddings between the Thinker (LLM) and Talker (voice generation) components, which is difficult for existing general-purpose frameworks, they built a custom pipeline utilizing shared memory.

Bottleneck Resolution and Streaming Optimization

The server introduces a Cascaded Streaming Pipeline that connects the Thinker, Talker, and VoiceBox via asynchronous queues for parallel execution. This significantly reduces the latency for users to hear the first voice response. Additionally, CUDA IPC and process isolation are applied to prevent GPU memory conflicts and block fault propagation, ensuring operational stability.

Concurrency Control and Flexible Service Modes

In a single-worker FastAPI environment, async/await chains and thread pools are used to handle CPU-bound tasks, while semaphores are applied to the VoiceBox to prevent memory overflow. Clients can select Latency-First (real-time conversation) or Quality-First (high-quality TTS) modes on a per-request basis, allowing the same server to support both scenarios.

Ecosystem Compatibility

Designed to be 100% compatible with the OpenAI Chat Completions API specification, it allows for the direct reuse of existing OpenAI SDK code. Furthermore, watermarks are automatically inserted into all generated audio to enable AI voice detection.