Furiosa SDK 2026.4 boosts MoE performance and adds hierarchical caching

Furiosa SDK 2026.4, the fourth major release for the RNGD platform, focuses on optimizing AI workloads on existing air-cooled data center infrastructure. The update delivers significant efficiency gains, particularly for Mixture-of-Experts (MoE) models, with token generation per watt improvements across 18 models compared to version 2026.3.

MoE Performance Gains

The most substantial improvements are seen in MoE architectures:

GPT-OSS-120B: +138% token generation per watt
GPT-OSS-20B: +57%
Qwen3-30B-A3B: +43%

Hierarchical KV Cache Offloading

An experimental feature, disabled by default, extends prefix reuse beyond NPU DRAM limits using a unified radix tree. The system implements a three-tier hierarchy:

L1: NPU DRAM for active, low-latency access.
L2: Host memory for evicted or backup blocks, stored via DMA.
L3: External storage for cluster-wide reuse, currently supporting the Mooncake backend.

Two L2 write policies are available: Write-through (copies to L2 after a reuse threshold, defaulting to 2) and Write-back (records to L2 upon L1 eviction). These policies can be used simultaneously to support hybrid attention and pipeline parallelism.

Optimized Multimodal Serving

Vision-language models now run end-to-end in the regular FCFS scheduler, integrating text-only and image-and-text requests. Key optimizations include:

Batched vision encoding: Simultaneous encoding of images from multiple requests to boost throughput.
Overlapped image preparation: Overlapping host-side image preparation with NPU text generation to reduce idle time.
Image embedding reuse: Caching embeddings across conversation turns and data-parallel workers.

Supported models for these optimizations include Gemma 4, Qwen3-VL, and EXAONE 4.5.

Expanded Model Coverage

The release adds support for numerous new model variants:

Qwen3-VL: 2B and 4B Instruct/Thinking models.
Qwen3 Embedding/Reranker: 0.6B and 4B models.
Harrier OSS v1: 0.6B multilingual instruction-aware text embedding model.
Mistral NeMo: 12B Mistral-Nemo-Instruct-2407-FP8 text-generation model.
Llama 3.1/3.3: Dynamically quantized FP8 variants.
BGE-M3 and BGE Reranker v2 M3: Multilingual embedding and cross-encoder models.
E5-Mistral: E5-Mistral-7B-Instruct instruction-aware embedding model.
Gemma 4: 31B vision-language model.
EXAONE 4.5: 33B vision-language model.

Profile-Guided Decode Batching

Previously an opt-in feature, Profile-Guided Decode Batching is now enabled by default for generative models. It optimizes decode batch and bucket selection based on NPU execution and I/O cost profiling. Benchmarks on GPT-OSS-120B show a median 22.7% increase in ShareGPT output-token throughput and a 16.0% decrease in mean TPOT. Initial profiling takes 1-2 minutes, after which profiles are cached.

Faster Token Sampling

CPU sampling now uses a faster top-k/top-p filtering algorithm inspired by Qrita, utilizing SIMD instructions and BF16 logits distribution estimation. This results in 2.7-13.2x faster filtering and up to 16.9% higher output-token throughput. An experimental NPU sampling option moves token selection directly to the NPU, reducing host processing delays from over 600µs to approximately 100µs in microbenchmarks.

Complete Migration to FXB

The SDK has fully migrated to Furiosa Executable Bundles (FXB), removing support for the deprecated legacy artifact format. FXB packages compiled kernels and metadata separately from model weights, allowing models to start without kernel compilation on the serving host. Compatible fine-tuned or weight-updated models can reuse existing FXB bundles without recompilation.