AI BriefingTalking to Statues: Building a Multimodal App with ElevenLabs Agents
Key point
This introduces the architecture and implementation of a multimodal app that uses ElevenLabs' API to have real-time voice conversations with statues in photos.
Details
You can build an app that takes a photo to identify a figure, then holds a real-time conversation using a voice suited to each character's historical era. This is implemented by combining ElevenLabs' Voice Design and Agent API.
The overall pipeline works in the following 5 steps:
- Image capture
- Identifying the artwork and character via OpenAI
- Researching historical information
- Generating a unique voice for each character using the ElevenLabs API
- Starting a real-time voice conversation via ElevenAgents based on WebRTC
The OpenAI Vision model extracts the artwork's name, location, artist, character information, and a detailed voice description from the image. In particular, the detailed voice description—including the character's gender, age, accent, and personality—serves as the key data used to generate high-quality voice through the Voice Design API.
This app has such a clear structure that it can be built with just a single prompt using Cursor and Claude Opus 4.5.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Privatemode Benchmarks GLM-5.3-Flash as Vision-Capable Decision Model with Parity to Jev but Higher Cost