Building Real-Time Speech Translation

gpt-realtime-translate is a live speech-to-speech translation model for broadcasts, streaming, phone calls, and video conversations. It automatically detects the input speech and returns audio and a transcript in the target language, with developers only needing to specify the output language.

This model is optimized for interpretation. It was trained on thousands of hours of professional interpreter audio to suppress non-translation responses, and it is designed to wait for sufficient context before speaking even in languages with different sentence structures. If your goal is to build an AI voice agent, you should use gpt-realtime-2 instead of this model.

Sessions set the target language via session.audio.output.language, and gpt-realtime-whisper can be attached if input captions are needed. Instead of choosing a fixed voice, the output audio uses dynamic voice adaptation, which follows the speaker's tone, pitch, and manner of speech. It currently supports 70+ input languages and 13 output languages, and does not support custom prompting or voice selection parameters.

• Dedicated endpoint: /v1/realtime/translations
• Input: continuous 24 kHz PCM16 audio sent via session.input_audio_buffer.append, including silence between sentences
• Output: 200 ms PCM16 chunks and target-language transcript deltas
• State: there is no response.create, assistant turn, tool call, or conversation state

There are two application patterns. Broadcast-style translation is for translating a single source audio to many listeners, as in livestreams, webinars, lectures, earnings calls, and keynotes. Conversational translation is for real-time conversation between participants speaking different languages, as in call centers, phone calls, and video chats.

Implementation examples are divided into three paths.

• Browser tab translation: captures tab audio with getDisplayMedia(), sends it over WebRTC, and receives transcript deltas via the oai-events data channel.
• Twilio phone translation: the backend converts audio coming in through Twilio Media Streams and passes it to Realtime Translation, then returns the resulting audio back as Twilio media messages.
• LiveKit video translation: subscribes to a remote participant's microphone track, translates locally, and renders the translated audio and captions.

For phone calls, you typically need to open separate sessions per direction, such as A-to-B and B-to-A, and for multi-party calls, sessions must be split according to who needs to hear whose speech in which language. Also, without original-audio passthrough or mixing, speech that is already in the output language the listener chose may be treated as silence.

In the browser demo, the server only issues short-lived client secrets and never exposes the OpenAI API key. Best practices, model limitations, and evals for production deployment are also covered.