AI Briefing#ai-inference
The latest AI and developer news about #ai-inference, with the original source and a short summary.
The latest AI and developer news about #ai-inference, with the original source and a short summary.
To resolve memory bottlenecks caused by increasing KV Cache, a hierarchical structure utilizing layers from HBM to NVMe SSD is required.
The new PCIe card features 64 AI cores, 576 MB on-die SRAM, and 548 GB/s DRAM bandwidth for generative AI workloads.
The compiler uses persistent engines to reduce startup time from 467 seconds to 9.0 seconds, enabling real-time adaptation to traffic and hardware changes.
Baseten launches Grid, Sandboxes, and Loops to extend its inference platform with routing, execution, and training capabilities.
New products include Ultra-fast inference on NVIDIA Blackwell, Nexus Desktop for model routing, and Fireworks Sovereign for on-premises deployment.
The company plans to produce 20,000 RNGD cards this year and more than double that in 2027, with the chip already in mass production and deployed by customers like Samsung SDS and LG Group.
Modular has open-sourced Mojo 1.0 and released the source code for the MAX framework, while completing its acquisition by Qualcomm.
The D270 features 32 Xᵉ3P cores and 256 XMX engines, designed for air-cooled, standard PCIe 5.0 data center deployments.
Meta aggregates fragmented idle inference capacity into a shared pool to enhance ad relevance without requiring new hardware.
Offers a lightweight decision model specialized for classification and routing tasks at $0.042 per 1M input tokens
The Rebel100s accelerator integrates switch functionality into I/O chiplets to support up to 256 sockets without external switches.
The new inference engine achieves 56 concurrent sessions on a single NVIDIA H100 SXM 80 GB by eliminating CPU-GPU overhead and batching continuous inference workloads.
Transitioning to a hierarchical architecture utilizing HBM, Host DRAM, and NVMe SSD is essential to resolve AI inference memory bottlenecks caused by the proliferation of long contexts and RAG.
NVIDIA Vera Rubin NVL72 took the lead in MLPerf Inference v6.1, recording up to 3.7x the throughput of the GB300.
As AI inference demand surges, new hardware architectures and quantization techniques that separate workloads into Prefill and Decode stages and resolve memory bottlenecks are emerging.
The open-source engine 'jinfer' has been released, enabling AI inference on the JVM without Python dependencies.
FuriosaAI has established a subsidiary in Singapore to accelerate the commercial deployment of its RNGD AI inference platform across the APAC region.
d-Matrix has adopted NVIDIA NVLink Fusion to accelerate the large-scale deployment of its next-generation Raptor XPU in AI factories.
Hugging Face has released 207 optimized WebGPU kernels and a loader library, improving browser-based AI inference performance by an average of 2.57x.
OpenAI's proprietary inference chip Jalapeño has demonstrated superiority over existing hardware in terms of throughput per watt and latency.
OpenAI's proprietary inference chip Jalapeño has been confirmed to significantly outperform existing systems in performance per watt and latency.
NVIDIA has begun mass production of the Groq 3 LPX, significantly boosting inference speeds on the Vera Rubin platform.
NVIDIA has begun mass production of Groq 3 LPX, significantly enhancing agent inference performance based on Vera Rubin.
Router provides a service that reduces inference costs by an average of 40% by allowing users to access all AI models with a single API key.