Webinar Recap: Building AI Agents That Sound Natural

Most AI agents give accurate answers, but they still sound mechanical because they fail to read emotion, context, and urgency. ElevenLabs' Luigi Sambuy identified rhythm and flow, emotional mirroring, context awareness, and brand-specific voice identity as the four key traits for overcoming this.

The demo showcased a call with a customer trying to change a reservation on their 5th wedding anniversary. The agent first congratulated them on the anniversary, self-corrected a mismatch in the reservation time, and proactively suggested a window seat. All of this was achieved not through a more powerful model, but purely through system prompt design.

7 levers for natural agents:

• Emotional tags: Implementing tonal shifts using the v3 conversational model's expressive modes
• Background noise: Creating familiar context with office noise, typing sounds, etc.
• System prompt tone: The element that makes the biggest difference
• Eagerness settings: Choosing between eager / normal / patient depending on the situation
• Pronunciation dictionary: Preventing mispronunciation of brand names and proper nouns
• Output settings: Intentionally applying a slightly muffled audio quality for users accustomed to traditional call centers
• Voice design / cloning: Creating brand-specific voices via instant cloning (30-second audio) or high-quality professional cloning

The core practical principles are to explicitly write tone into the system prompt, place emotional tags in line with the conversational flow, and adjust eagerness according to the situation.