FuriosaAI SDK 2025.3 Brings Major Performance Boost to RNGD

Furiosa SDK 2025.3 and its 2025.3.1, 2025.3.2, 2025.3.3 updates have significantly boosted the performance and efficiency of RNGD. The improvements are especially large for large-scale models and agentic AI workloads, with average throughput on Llama 3.3 70B increasing by up to 3x and average TTFT (Time to First Token) decreasing by 35%.

The biggest change is support for inter-chip tensor parallelism, which links multiple RNGD cards together. P2P (peer-to-peer) communication based on PCIe Gen 5 raises inter-chip data transfer to up to 64 GB/s bidirectional, and path optimization along with advanced communication scheduling improve multi-chip scalability. Furiosa stated that running Llama 3.3 70B on an 8-card RNGD server consumes less than half the total power of competing NVIDIA solutions.

The compiler and runtime were also improved together.

• Global optimization in the Furiosa Compiler maximizes SRAM reuse between transformer blocks
• The runtime reduces overhead by minimizing interference between RNGD and the host
• Explicit overlapping of inter-chip DMA and computation further reduces latency

This combination showed up in actual benchmarks as well. For Llama 3.1 8B, average throughput increased by 4.5% and TTFT decreased by 55%.

Model support has also expanded. Qwen 2 and Qwen 2.5 are now additionally supported, and W8A16 quantization is also provided. Precompiled artifacts on Hugging Face Hub support context lengths of up to 32K tokens, addressing applications that need longer context.

Operational convenience has also been strengthened. Production metrics are exposed via a /metrics endpoint, and logs display average throughput, KV Cache usage, and running/waiting request information. This allows developers to more finely track performance bottlenecks and inference status.

Additionally, the SDK supports the Structured Outputs feature that OpenAI added to its API. This allows output to be forced to conform to a JSON schema, improving format stability for MCP (Model Context Protocol)-based tool calling or tasks where agents interact with APIs.

Following SDK 2025.2.0 in May, which added Hugging Face Hub integration, reasoning model support, and chunked prefill, Furiosa continues its rapid release cadence. Together with the case of LG AI Research adopting RNGD for EXAONE model inference and achieving 2.25x better performance-per-power efficiency compared to existing GPUs, these improvements are strengthening RNGD as a platform for datacenter inference.