How Tolan Built Voice-First AI Using GPT-5.1

Voice-first AI startup Tolan is developing an AI companion capable of sustained, open-ended conversation, going beyond simple Q&A. The Portola team focused on solving the core challenges of voice AI: Latency and Context Management.

The adoption of GPT-5.1 became a technical turning point. Using the OpenAI Responses API, they reduced conversation start time by over 0.7 seconds, and by leveraging the model's improved Steerability, they were able to more precisely implement the tone and personality of their designed characters.

Tolan's architecture adopted a method of rebuilding context every turn, rather than relying on traditional prompt caching. This allows for seamless response even when users suddenly switch topics. The context reconstruction includes the following elements:

• Recent message summaries
• Persona cards
• Vector search-based memory
• Tone guides and real-time app signals

Additionally, for long-term conversations, they built a memory system combining the OpenAI text-embedding-3-large model with Turbopuffer, a high-speed vector database. This system stores not just plain facts but also the emotional 'vibe' of conversations, and maintains memory quality through nightly data compaction jobs.