AI-Generated Personal Voice (SK AI SUMMIT Presentation Case)

At the Special Session on the second day of SK AI SUMMIT 2025, held on November 4, 2025, SK Planet Manager Kim Woo-ju presented AI Voice Conversion technology. The core idea goes beyond simple Text2Speech, converting the input speaker's utterance patterns and rhythm into the target speaker's timbre and tone while preserving them.

Existing Speech Voice Conversion takes both Source and Target speech as input, preserving the source's content and rhythm while applying only the target's speaker characteristics. Adding Text on top of this moves it closer to TTS with voice conversion, and for natural synthesis at an actual service level, preprocessing such as Denoising, Speech Enhancement, and Source Separation is also needed.

Singing Voice Conversion has a similar structure, but regenerates a song in the target speaker's timbre while preserving the content, rhythm, and melody of the song. Here, a pitch condition and a Mixing step that combines vocals and accompaniment are added, ultimately producing a result that sounds like the listener singing another singer's song.

The pipeline is organized around a Feature Extractor and a Generator.

• Feature Extractor

  • Mel-Spectrogram represents the frequency characteristics and prosody of speech
  • Pitch Extractor extracts the F₀ (Fundamental Frequency) curve to organize pitch variation
  • Semantic Encoder separates content information such as lyrics, rhythm, and melody
  • Style Encoder encodes style information such as timbre, tone, and vocalization technique

• Generator

  • Timbre Extractor refines the target timbre to reduce timbre leakage
  • Length Regulator aligns semantic information with acoustic frame length
  • Diffusion Transformer Decoder (U-DiT) combines style, pitch, and semantic information to generate a high-quality Mel-Spectrogram
  • Vocoder (BigVGAN) restores this into an actual waveform to complete the final voice

The core of this structure is the Zero-Shot VC Network. It generalizes so that even new speakers not present in the training data can be converted and synthesized using just 5–10 seconds of speech, controlling speaker characteristics and speaking style by separating them.

Quality is evaluated using Speaker Cosine Similarity for speaker similarity, and F₀ Correlation and F₀ RMSE for the similarity and error of the pitch trajectory. The better the timbre match, the higher the similarity; the more naturally the pitch is maintained, the higher the correlation and the lower the RMSE.

The limitations are also clear. Real-time response places a heavy burden on GPU resources, high-quality speech and song data are scarce, and there are copyright issues. Additionally, after conversion, the original speaker's timbre may become blurred or converge toward a similar style, requiring supplements such as model lightweighting, data augmentation, refinement of speaker embeddings, and fine-tuning of the style extractor.