AI Briefing#latency
The latest AI and developer news about #latency, with the original source and a short summary.
The latest AI and developer news about #latency, with the original source and a short summary.
The new API parameter reduces median time to first token by 35% for 500k to 1M token requests by sending only new messages per turn.
In conversations over two minutes, voice agents achieved a 31.7% higher success rate than chat and resolved issues in 4.6 minutes compared to 7 minutes for text.
The feature reduces time to first token by sending only new messages since the last turn, with threads held in encrypted memory and expiring after 5 minutes of inactivity.
Snapshot restoration time now counts toward creation, adding about 50 ms to p50 latency but eliminating wait times for the first operation.
Vercel reduced Sandbox domain lookup latency from 62ms to 3.4ms, an 18x improvement.
The 6G user plane will maximize latency and energy efficiency by simplifying the existing hierarchical structure and introducing intelligent processing.
As AI inference speeds increase, the additional computation capacity available within a deadline, known as the 'deadline dividend', grows.
AI Gateway has launched a unified Fast Mode in beta that provides lower latency and higher throughput on a per-model basis.
Model routing in AI agent systems isn't a model-selection problem but a system optimization problem involving caching, infrastructure, and regulations.
Covers architecture design and communication protocol comparisons for building ultra-low-latency real-time STT systems.
A technical analysis has raised concerns that MCP excessively occupies context windows and has low operational reliability.
AI Gateway has added a `sort` option that sorts providers by cost, TTFT, and TPS.
A local 35B model handled about half of daily tasks faster than the cloud.
Luke Curley criticized WebRTC for dropping audio and prompts in order to achieve low latency.
AI agent inference exposes data and storage bottlenecks first.
Per-GPU KV cache locality determined LLM serving cost and TTFT.
Anthropic disclosed three causes behind performance degradation in Claude Code, SDK, and Cowork.
Simon Willison introduced a tool that converts milliseconds from LLMs into seconds and minutes.
WebSocket mode made agentic workflows on the Responses API up to 40% faster.
On A100 40GB, MIG significantly lowered protected-tenant p95 compared to shared vLLM.
Zo Computer cut its retry rate by 20x using Vercel's AI SDK and AI Gateway.
Complaints about Ollama Cloud's response latency and low TPS have escalated to the point of subscription cancellations.
AI infrastructure must move away from Seoul metropolitan power and network bottlenecks toward coastal landing-edge sites.
ElevenLabs introduced multi-region serving, cutting global TTS API latency by up to 40%.