Rust-based safe GPU inference engine cuTile unveiled

cuTile Rust, developed by NVIDIA Labs, introduces Rust's Ownership and Borrow checking mechanisms into GPU kernel writing, providing a programming model that can verify memory safety and the absence of data races at compile time.

Key features are as follows:

• Safe kernel generation: Through a Tile-based programming model, kernels are converted into CUDA Tile IR, and the compiler safely manages memory access across thread blocks.
• High-performance inference engine Grout: A Qwen3 inference engine built on cuTile Rust, recording 171 tok/s on RTX 5090 (Qwen3-4B) and 82 tok/s on B200 (Qwen3-32B), showing performance on par with vLLM and SGLang.
• Performance efficiency: Safe GEMM kernels achieve 99.7% of the performance of manually written low-level versions on B200, and element-wise operations achieve a throughput of about 7 TB/s.

This technology aims to provide a verifiable, safe target for future kernel synthesis for AI models.