AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
Qwen3.6-27B-INT4 achieved 256k context and 105~108 tps on a single RTX 5090.
MoE models achieve higher inference acceleration when Speculative Decoding techniques are applied.
FlashDrive optimizes the entire autonomous driving VLA inference pipeline, cutting latency from 716ms to 159ms.
Open-source reproduction code for Cartridges and STILL, KV-cache compression techniques for long-context inference, has been released.
OpenAI, in partnership with Cerebras, has unveiled GPT-5.3-Codex-Spark, a real-time coding model that generates over 1,000 tokens per second.
Overworld has released Waypoint-1, a video diffusion model that supports real-time control via text and keyboard/mouse, along with the inference library WorldEngine.
This guide explains how to optimize and run VLMs on Intel CPUs using OpenVINO and Optimum Intel.
Hugging Face's transformers library has announced a major update to support OpenAI's gpt-oss models.
This introduces a method to speed up model inference on ZeroGPU Spaces by up to 1.8x using PyTorch's AoT compilation.
A method using Diffusers and PEFT was released that improves LoRA inference speed for Flux models by about 2.3x.
NVIDIA NIM enables fast and stable deployment of over 100,000 LLMs from Hugging Face.
This analyzes the queue-blocking problem caused by long prompts in vLLM's chunked-prefill strategy and presents an optimization approach.
PipelineRL has been released, resolving the trade-off between inference throughput and data quality during LLM reinforcement learning.
This proposes introducing fair scheduling at the API server layer to prevent specific users from monopolizing requests.
NVIDIA has released KVPress, a KV cache compression toolkit for handling long context in LLMs.
A static embedding model and training methodology have been released that deliver up to 400 times faster speed than existing models in CPU environments.
XLSCOUT collaborated with Hugging Face to launch ParaEmbed 2.0, optimized for patent and intellectual property (IP) analysis.
Hugging Face has officially launched a dedicated container for efficiently deploying embedding models on Amazon SageMaker.
Intel Gaudi now supports Speculative Sampling-based Assisted Generation, boosting LLM inference speed by up to 2x.
By applying Speculative Decoding, the inference speed of Whisper was doubled while maintaining accuracy.
HuggingFace kept the base model in memory and dynamically swapped only LoRA adapters, improving inference speed by 300%.
With LCM LoRA, SDXL models can generate images quickly using just 4-8 steps without any separate retraining.
ONNX Runtime supports over 130,000 models on Hugging Face, accelerating inference performance.
AudioLDM 2's inference speed has been improved by more than 10x through Hugging Face Diffusers.