AI Briefing#gpu-optimization
The latest AI and developer news about #gpu-optimization, with the original source and a short summary.
The latest AI and developer news about #gpu-optimization, with the original source and a short summary.
The Reshard operation announced by NVIDIA reduces weight transfer time in disaggregated RL environments by up to 5.7x.
Pinterest solved embedding overfitting by delaying table updates until the final epoch, enabling multi-epoch fine-tuning that improved online saves by 1.09%.
Modular has released a free 57-page handbook covering LLM inference metrics, optimization, and operational costs.
The compiler uses persistent engines to reduce startup time from 467 seconds to 9.0 seconds, enabling real-time adaptation to traffic and hardware changes.
The new architecture uses a Mixture-of-Experts model and fused GPU kernels to deliver near-ranking-level performance at 300x fewer FLOPs than the 2024 system.
cuTile kernels that boost Mamba-2 forward propagation speed by up to 1.82x on B300 GPUs have been released.
PyTorch 2.15 improves Mac GPU MoE decoding by 5-6.5x and supports FP8.
The library supports NVIDIA Turing and later architectures, achieving significant speedups over vLLM Marlin and DeepGEMM across dense and MoE workloads.
Cache misses caused by harness modifications like timestamp changes or tool list updates increase energy consumption by up to 2.5x compared to fully cached requests.
The conference showcased case studies on efficient AI infrastructure operations, ranging from GPU resource optimization to fault resilience.
The new model improves conversion rates by 18.9% and click-through rates by 12% across Meta's advertising surfaces.
StreamRender-H3 achieves 218ms per streaming step on an 8 GB200 GPU setup, a 3.67x speedup over the PyTorch baseline, by separating world logic from visual generation.
Speculative decoding accelerates LLM inference by increasing total FLOPs to utilize idle compute resources, shifting the workload from memory-bound to compute-bound at low batch sizes.
The new architecture maintains within 3.4% of native token throughput on NVIDIA H100 GPUs while enabling support for out-of-tree accelerators and older hardware.
Underdog has released 'Husky', a model-specific inference engine that is faster than MLX across all measured metrics, achieving up to 4.5x performance gains particularly in editing tasks.
Hugging Face plans to release tokenizers v1, aiming to reduce GPU idle time and achieve up to 30x faster processing speeds.
Anthropic used Claude to accelerate over 30 open-source biomolecular models by an average of 4x and successfully predicted large-scale systems with over 10,000 tokens on a single GPU.
NVIDIA achieved 1,025 TFLOPs per GPU on GB200 for Dropless MoE training via Transformer Engine for JAX, improving throughput by approximately 10x.
A small Transformer trained from scratch in 1.5 hours on a 5090 GPU achieved 44% on ARC-AGI, criticizing the low sample efficiency of LLMs.
vLLM v0.28.0 adds large-scale performance optimizations and inference capabilities for the Kimi-K3 and DeepSeek V4 models.
LayerStoRm, an open-source engine that runs large MoE LLMs on consumer GPUs using PCIe streaming, has been released.
LMSys has released optimization strategies to maximize DeepSeek-V4-Pro serving performance on H20 GPUs.
It implements a block-diagonal attention kernel that is up to 3.5x faster than Flash Attention v2 on NVIDIA Blackwell GPUs.
An open-source kernel has been released that enables efficient training of MiniMax Sparse Attention (MSA) on Hopper and Blackwell GPUs.