AI BriefingTogether AI Releases TKCC Compiler for Continual AI Inference Optimization
·2026.11.04 09:00
Key point
The compiler uses persistent engines to reduce startup time from 467 seconds to 9.0 seconds, enabling real-time adaptation to traffic and hardware changes.
Details
- Together AI introduces TKCC, a continual compilation tool that optimizes AI inference in real time, achieving up to 2.6x higher TPS on frontier open source models like Kimi K3 and DeepSeek v4.1-Flash.
- TKCC sets State-of-the-Art (SOTA) low-latency benchmarks on SemiAnalysis InferenceX, recording 1111 TPS for MiniMax M3.1-Flash and 1205 TPS for DeepSeek v4.1-Flash.
- The system utilizes persistent engines to eliminate engine restarts, cutting startup time from 467 seconds to 9.0 seconds and increasing experiment throughput by 252x compared to existing agent systems.
- A custom cost model selects parallelization strategies based on measured forward passes and hardware specs, outperforming static configurations across six model and chip pairs.
- TKCC generalizes beyond LLM inference to weather modeling, PDEs, time series, and biology, showing geometric mean speed improvements over Claude Code /goal commands in three out of four tested models.
- Traffic patterns drive optimization needs, with prompt lengths varying by 1.4x across regions and increasing by 80% two weeks after model releases due to a shift toward agent workloads.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Qualcomm Launches Cloud AI 100 Ultra Inference Card with 128GB Memory and Up to 870 TOPS
Baseten Launches Grid, Sandboxes, and Serverless LoopsFireworks Unveils Ultra-fast Inference, Nexus, and Sovereign at Forge